MSL 4.1.0 Regressions - MultiBody

[GallLeo]

The following models fail in result comparison. Tested revision: f9bddf86 (2024-02-16)

Changed models, need reference update after library officer check:

• [ ] Modelica.Mechanics.MultiBody.Examples.Elementary.PointGravityWithPointMasses2
  • Reason: very tiny error at the end of the simulation on freeMotion.v_rel_a[2], which is very close to zero. The problem here is that CSV compare is just too strict on variables close to zero. One option is to remove the variable until we have a proper solution for that. This is done by PR #4420 (needs to be ported to maint/v4.1.0). However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421.
  • Updated reference files?
• [ ] Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D
  • Reason: the trajectories diverge sharply for time > 0.52, most likely due to chaotic motion - the system is highly nonlinear and has more than two states, which are necessary ingredients for the path to chaos. Just changing Dymola version produces a different trajectory after that point, and also OpenModelica behaves differently. There's no point trying to reproduce chaotic trajectories, so we can limit the time span to StopTime = 0.4 only for the reference results. CSV compare should stop comparing when it no longer has data, but it needs to be fixed, see modelica-tools/csv-compare#65
  • Updated reference files? @GallLeo please generate reference files with StopTime = 0.4.
• [ ] ModelicaTest.MultiBody.Parts.FreeBodies
  • Reason: body.Q[2] is very close to zero, same story as the first example. Interim solution: remove it from the reference trajectories. This could be done by PR #4420 (needs to be ported to maint/v4.1.0), however, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421.
  • Updated reference files?
• [ ] ModelicaTest.MultiBody.Sensors.AbsoluteSensor
  • Reason: Test model changed, see https://github.com/modelica/ModelicaStandardLibrary/pull/4148 (and esp. https://github.com/modelica/ModelicaStandardLibrary/commit/82d1b167e20e0decf632a8c9a2ea58f3c92415cb). The new results are correct.
  • Updated reference files? @GallLeo please update reference files.

No signals to compare, define more comparison signals:

• [X] ModelicaTest.MultiBody.Visualizers.Arrow
  • Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
  • Updated reference files? No need to
• [X] ModelicaTest.MultiBody.Visualizers.ColorMaps
  • Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
  • Updated reference files? No need to
• [X] ModelicaTest.MultiBody.Visualizers.Shapes
  • Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
  • Updated reference files? No need to
• [X] ModelicaTest.MultiBody.Visualizers.Torus
  • Updated comparisonSignals.txt? This model is basically meant to test 3D visualization capabilities, there's really no point checking numerical results, which are trivial
  • Updated reference files? No need to

Change due to new Dymola version

• [ ] ModelicaTest.MultiBody.Joints.Universal

  • Reason: Numerical differences on accelerations (second derivative). Martin Otter (during MAP-LIB meeting): Please update reference. Francesco: I further analyzed the issue after the meeting with @tobolar. universal.w_a is zero until about time = 0.3, then it starts showing some oscillations which are growing over time, but are clearly triggered by numerical errors and have a negligible amplitude (2e-8 rad/s). This is also confirmed by looking at the regression in OpenModelica. There is no point trying to reproduce this signal, which clearly only depends on numerical approximations, so until CSV compare can handle that properly, we can remove it in PR #4420 (needs to be ported to maint/v4.1.0). However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421.
  • Updated reference files?

• [ ] ModelicaTest.MultiBody.Joints.UniversalSpherical

  • Reason: Numerical differences on accelerations (second derivative). Martin Otter (during MAP-LIB meeting): Please update reference. We made further analysis, in this case there are several variables which are very close to zero, similarly to PointGravityWithPointMasses2. We should remove all variables that are close to zero, because there is currently no way to test them reliably with CSV-compare, and on the other hand it is unlikely that these are wrong while the other ones are OK, since they are components of some physical vectors. This can be done by PR #4420 (needs to be ported to maint/v4.1.0) However, as discussed in the MAP-Lib meeting 02-07-2024, maybe the best solution is to flag this regression as a false positive and fix the CSV tool to include some absolute tolerance for near-zero signals, see #4421.
  • Updated reference files?

• [ ] ModelicaTest.MultiBody.Parts.Rotor1D.GearConstraint3

  • Reason: chaotic motion with bifurcation after time = 0.30
  • Updated reference files? @GallLeo please generate reference files with StopTime = 0.30. Also needs fixing modelica-tools/csv-compare/#65 to avoid a false positive

• [X] ModelicaTest.MultiBody.PlanarLoopWithMove

  • Reason: No more regression, probably due to resolved table generator issues.
  • Updated reference files? No.

• [X] ModelicaTest.MultiBody.Sensors.Distance2

  • Reason: No more regression, probably due to resolved table generator issues.
  • Updated reference files? No

• [X] ModelicaTest.MultiBody.Sensors.RelativeSensor

  • Reason: No more regression, probably due to resolved table generator issues.
  • Updated reference files? No.

• [ ] Release notes check: Are all classes mentioned which could lead to result changes in user models?

---

Useful Links

_Current comparison report: https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/comparison_report_overview.html -> Reference result test -> Comparison_

Comparison signal definitions: https://github.com/modelica/ModelicaStandardLibrary/tree/master/Modelica/Resources/Reference/Modelica https://github.com/modelica/ModelicaStandardLibrary/tree/master/ModelicaTest/Resources/Reference/ModelicaTest

_Reference results: https://github.com/modelica/MAP-LIB_ReferenceResults_

[tobolar]

Library officer check:

• Modelica.Mechanics.MultiBody.Examples.Elementary.PointGravityWithPointMasses2 Checking https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/Modelica/20240227232653/Modelica.Mechanics.MultiBody.Examples.Elementary.PointGravityWithPointMasses2/CSVCompare/PointGravityWithPointMasses2_report.html, the error message is: 6 (relative error is 1,81520807176844E-16) So I really wonder why csv-compare catch this. Anyway - this example is fine to me.

• Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D I simulated both MSL400 and MSL4100 in Dymola2023x Refresh1, and then MSL400 and MSL4100 in Dymola2024x. Comparing results of the two MSL versions calculated in ONE of Dymola yields the same results. So the problem seems to be the comparison across the Dymola versions. I tried to increase the numerical tolerance but this alone doesn't help. What seems could help is to a) introduce a damping and b) increase the tolerance (to=1e-6). So maybe better place this under "Change due to new Dymola version" below?

• ModelicaTest.MultiBody.Parts.FreeBodies Checking https://www.ltx.de/download/MA/Compare_MSL_v4.1.0/Compare/ModelicaTest/20240228011806/ModelicaTest.MultiBody.Parts.FreeBodies/CSVCompare/FreeBodies_report.html, the error message is: 433 (relative error is 3,22845694576259E-13) So I really wonder why csv-compare catch this. Anyway - this example is fine to me.

• ModelicaTest.MultiBody.Sensors.AbsoluteSensor Test model changed, see https://github.com/modelica/ModelicaStandardLibrary/pull/4148 (and esp. 82d1b167e20e0decf632a8c9a2ea58f3c92415cb) So the reference files shall be updated.

No signals to compare, define more comparison signals:

• ModelicaTest.MultiBody.Visualizers The tests are intended to test only 3d visualization of the objects. So not sure how to proceed here.

Change due to new Dymola version (reference update after library officer check?):

• ModelicaTest.MultiBody.Parts.Rotor1D.GearConstraint3 This seems to be similar to Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D above. So a) introduce a damping and b) increase the tolerance (to=1e-6).

Not classified, yet (needs library officer check):

• Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D This item is doubled, see the same class above.

• ModelicaTest.MultiBody.PlanarLoopWithMove I can't simulate - maybe missing binaries? https://github.com/modelica/ModelicaStandardLibrary/pull/4250

• ModelicaTest.MultiBody.Sensors.Distance2 There is no error reported.

• ModelicaTest.MultiBody.Sensors.RelativeSensor There is no error reported.

[casella]

On 2024-06-18 I reviewed all issues with @tobolar, took decisions, and made actions, as reported in the top ticket description

[GallLeo]

Modelica.Mechanics.MultiBody.Examples.Rotational3DEffects.BevelGear1D

I tried to reduce the reference CSV to 0.4 s.

It nearly works in csv-compare. Unfortunately, it reports errors for each variable, because the tube extrapolates a bit in the future.

Proposal:

• update reference file to stop at 0.4 s
• accept regression failure, for now.
• improve CSV-compare or agree on test-stop-time-annotation for future MSL development (CI)

[casella]

Agreed, as discussed during the 2024-07-02 MAP-Lib meeting