search

modelica / ModelicaStandardLibrary

Free (standard conforming) library to model mechanical (1D/3D), electrical (analog, digital, machines), magnetic, thermal, fluid, control systems and hierarchical state machines. Also numerical functions and functions for strings, files and streams are included.
https://doc.modelica.org
BSD 3-Clause "New" or "Revised" License
452 stars 165 forks source link

Release plan for MSL 3.2.2 #1867

Closed modelica-trac-importer closed 7 years ago

modelica-trac-importer commented 7 years ago

Reported by ahaumer on 18 Dec 2015 11:44 UTC In a web-/phone-conference today a smaller working group developed the release plan for MSl 3.2.2:

  1. Dec. 23, 2015: Feature Freeze. Trunk is tagged as MSL 3.2.2 Alpha.1

  2. All tool vendors should check MSL

  3. Fix remaining tickets for Milestone 3.2.2 (also documentation can still be improved)

  4. If new models of MSL 3.2.2 Alpha.1 have no meaningful continuous-time states, then the authors of these models have to provide meaningful variables as outputs on the top level, provided there are no or no meaningful continuous-time states (for example: “output Real noise_a = noise.a”). This change should be performed before the “freeze” (on Dec. 23, 2015) for: Magnetic.QuasiStatic.FundamentalWave (Anton Haumer, Christian Kral) Blocks.Examples.NoiseExamples (Martin Otter) Tables in ModelicaTest (Thomas Beutlich) If there are other authors of such new models, they should check this.

  5. Jan. 12, 2016: Trunk is tagged as MSL 3.2.2 Beta.1

  6. Contract: Generate reference results for MSL 3.2.1 with Dymola 2014 FD01 (the regression testing of MSL 3.2.1 was performed with this Dymola version).

  7. Contract: Regression testing of MSL 3.2.2 Beta.1 with Dymola 2016 FD01 with respect to reference results of MSL 3.2.1. This tests whether there is a regression of MSL 3.2.2 Beta.1 with respect to MSL 3.2.1 and at the same time, whether Dymola 2016 FD01 still produces the same results as Dymola 2014 FD01.

  8. Jan. 29, 2016: Contract: The simulations results of MSL 3.2.2 Beta.1 are provided as new reference results (including newly added models).

  9. Contract: A list of models that do not fulfill the regression tests are provided in a new ticket.

  10. The authors of these models have to fix the models or provide an explanation for the regression.

  11. All tool vendors should check MSL

  12. Feb. 19, 2016: A release candidate is provided (MSL 3.2.2 RC.1)

  13. Contract: Regression testing of MSL 3.2.2 RC.1 with Dymola 2016 FD01 with respect to reference results of MSL 3.2.1. If there are regression errors, it is checked whether all of them are explained in the ticket that this is correct. The simulation results are provided as new reference results of MSL 3.2.2.

  14. March 4, 2016: Electronic voting starts for this release by the MAP “Modelica Libraries” members.

  15. March 11, 2016: MSL 3.2.2 is released.

Leo Gall is proposed to provide a quote for the items marked as “contract”. The MA Board checks and decides whether this contract is given to Leo Gall. (Leo Gall made the last regression tests for MSL 3.2.1 as a contract; if an open call would be made, the release of MSL would be delayed by at least 2 months. How to organize this in the future, should be discussed at the next Modelica Design Meeting, March 4-6).

If there are issues, please comment in this ticket.

Migrated-From: https://trac.modelica.org/Modelica/ticket/1867

modelica-trac-importer commented 7 years ago

Comment by hansolsson on 19 Feb 2016 09:28 UTC Replying to [comment:50 jmattsson]:

Replying to [comment:46 beutlich]:

I do not see an RC tomorrow if there are 4 open issues (#1886, #1890, #1899 and #1911) that still require discussion. It looks to me like we need another beta once the issues are resolved. There have been 30 revisions since beta 2 was tagged, 12 with commit messages that seem like bug fixes.

What were the other 18 commits? Just documentation? I certainly hope we aren't discussing new functionality.

Since we are trying to move towards a more structured release process, we should stick to that process. If a release candidate has code changes compared to the preceding beta, then what is the distinction between a beta and a release candidate?

You just described the difference https://en.wikipedia.org/wiki/Software_release_life_cycle

Obviously everyone uses their own definitions instead - but that gives a general flavor of the difference.

So, a release candidate will often have bug fixes compared to previous betas, but shouldn't add new functionality.

If a release contain changes compared to the release candidate (except for version number and date), then there are major problems.

Also, what is the main goal of the process changes - more confidence in the quality of released versions, or a more predictable release schedule?

Ideally both. By saying "no" to late additions we both avoid delays and improve quality.

modelica-trac-importer commented 7 years ago

Comment by jmattsson on 19 Feb 2016 12:07 UTC Replying to [comment:51 hansolsson]:

Replying to [comment:50 jmattsson]:

Replying to [comment:46 beutlich]:

I do not see an RC tomorrow if there are 4 open issues (#1886, #1890, #1899 and #1911) that still require discussion. It looks to me like we need another beta once the issues are resolved. There have been 30 revisions since beta 2 was tagged, 12 with commit messages that seem like bug fixes.

What were the other 18 commits? Just documentation? I certainly hope we aren't discussing new functionality.

Ah, sorry, I should have been clearer. The others were documentation, release notes, etc.

By saying "no" to late additions we both avoid delays and improve quality.

Seems my argument was unclear as well. It just seems to me like we are treating the schedule that Anton sent as set in stone, and rushing in bug fixes to get them in before the release candidate. Shouldn't the move to a release candidate be a decision that we are in state where that is the proper step rather than an automatic occurrence?

modelica-trac-importer commented 7 years ago

Comment by otter on 19 Feb 2016 13:09 UTC Replying to [comment:33 msasena]:

It looks like there are still 3 models that include Tolerance in their experiment annotations, but not StopTime:

  • ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.Plate.kc_overall
  • ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.StraightPipe.kc_laminar_KC
  • ModelicaTest.MultiBody.Forces.SpringDamperParallel

We could really use the StopTime values in order to automate some testing, or at the very least explicitly state how long the expected simulation test should run.

Added the missing StopTime annotation in 22be976c2fe55629f7d35b31083467ebfeaefa09

modelica-trac-importer commented 7 years ago

Comment by otter on 19 Feb 2016 13:32 UTC Replying to [comment:41 msasena]:

The LMS Amesim compiler found several "warnings" related to equality comparison of Real variables. For example, there are lines like if (d_vap <> d_liq) or if abs(m) <= tol or fb == 0.0. I suspect most other tools just ignore the warnings, like we do. However, it might be nice to clean up as many of them as we can before the v3.2.2 release. This is most likely just a partial list...

Functions that use == comparisons on Real variables:

  • Modelica.Media.Water.IF97_Utilities.BaseIF97.Basic.f3
  • Modelica.Media.Common.OneNonLinearEquation.solve

Functions that use <> comparisons on Real variables:

  • Modelica.Media.Water.IF97_Utilities.waterBaseProp_ph
  • Modelica.Media.Water.IF97_Utilities.waterBaseProp_ps
  • Modelica.Media.Water.IF97_Utilities.waterBaseProp_pT
  • Modelica.Media.Water.IF97_Utilities.BaseIF97.Isentropic.hofps4
  • Modelica.Media.Water.IF97_Utilities.BaseIF97.TwoPhase.waterSat_ph
  • Modelica.Media.Water.IF97_Utilities.BaseIF97.TwoPhase.waterR4_ph
  • Modelica.Media.Water.IF97_Utilities.BaseIF97.TwoPhase.waterR4_dT

Opened a separate ticket for this (#1915). Anyway, this seems uncritical for MSL 3.2.2 and might be analyzed for the follow-up MSL version.

modelica-trac-importer commented 7 years ago

Comment by otter on 19 Feb 2016 13:43 UTC Replying to [comment:38 hubertus]:

Regarding ModelicaTest.Fluid.TestExamplesVariants.BranchingDynamicPipes_MomentumSteadyState: * Dymola seems to work with tighter tolerances and * JModelica handles it fine

My suspicion is that the model is numerically very sensitive, and should maybe be cleaned up/changed for the future, but it should not block the release, we could simply update the tolerance for Dymola. I opened a ticket with Milestone MSL next minor release and assigned it to Francesco for investigating this test. (#1900)

Added Tolerance=1e-5 annotation in 563e24c54b18f9a8cf2991effbd41233bd6b560a, in order that this model simulates in Dymola 2017 Alpha (and hopefully still in the other tools)

modelica-trac-importer commented 7 years ago

Comment by otter on 19 Feb 2016 14:20 UTC Here is a summary of the changes/commits since Beta.2:

Made models to work in some tools

Minor improvements that have no influence on the usage of MSL (same simulation results)

The only open item is:

Leo Gall told me that he will finalize the first regression testing soon but that he did not yet detect any critical issue (one current issue: the comparison program crashed for some models and he has to manually check them).

My conclusion is that we should soon fix #1899 (say latest on Monday), wait for the regression testing results of Leo Gall and then release RC1 (say on Tuesday or Wednesday next week).

modelica-trac-importer commented 7 years ago

Comment by jmattsson on 19 Feb 2016 14:46 UTC Replying to [comment:56 otter]:

My conclusion is that we should soon fix #1899 (say latest on Monday), wait for the regression testing results of Leo Gall and then release RC1 (say on Tuesday or Wednesday next week).

That is, given that the regression tests checks out, right?

modelica-trac-importer commented 7 years ago

Comment by otter on 19 Feb 2016 14:47 UTC Replying to [comment:57 jmattsson]:

Replying to [comment:56 otter]:

My conclusion is that we should soon fix #1899 (say latest on Monday), wait for the regression testing results of Leo Gall and then release RC1 (say on Tuesday or Wednesday next week). That is, given that the regression tests checks out, right?

Yes, under the assumption that there are no critical issues.

modelica-trac-importer commented 7 years ago

Comment by otter on 19 Feb 2016 21:32 UTC

Replying to [comment:28 otter]:

There are also the following uncritical failures (due to overdetermined initial equations): One of our developers have previously examined the ones that have names ending with "InitialInconsistent" and came to the conclusion that they have inconsistently overspecified initial conditions, as the names suggest, and should thus fail to translate. I'd say we should move them to ModelicaCompliance - see #1591.

Fixed #1591 in 9e763c47be2f6b25b9b7b22955281df97ae88abd by moving all test models with overdetermined initial equations from ModelicaTest to the new library trunk/ModelicaTestOverdetermined (these models test advanced features of Modelica tools, but do not test MSL for "correctness". Since ModelicaTest is used to test whether the MSL models are correct, it is better to remove them from ModelicaTest in order to avoid always "wrong" error messages where tool vendors have to take care off.)

Making "Check" with Dymola 2017 Alpha with pedantic flag on Modelica gives 7 warning messages (that are uncritical, and are a Dymola issue) and no errors, and on ModelicaTest there are no warning and no error messages.

Making "Check with Simulation" with Dymola 2017 Alpha with pedantic flag on Modelica and ModelicaTest is successfull (all example models translate and simulate).

modelica-trac-importer commented 7 years ago

Comment by hubertus on 19 Feb 2016 22:38 UTC Replying to [comment:56 otter]:

Here is a summary of the changes/commits since Beta.2:

Made models to work in some tools

  • 1886: Introduced impure vendor annotations

  • 1900: One ModelicaTest model failed in Dymola (and run in JModelica). Stricter tolerances added (the model now simulates in Dymola).

  • 1890: Replaced M*v of Complex matrix/vector by function call, since some tools do not yet support this part of operator overloading.

Minor improvements that have no influence on the usage of MSL (same simulation results)

  • 1884 Removed unqualified import in Spice3 library

  • 1903 Made initialization of switches in Analog.Ideal uniform

  • 1856 Introduced verboseRead flag for readRealMatrix

  • Added missing StopTime annotations in ModelicaTest
  • Some Compiler-specific improvements in ModelicaUtilities.h (noreturn attribute)
  • Minor issues in documentation fixed (spelling errors, HTML tags)
  • Removed dummy and default annotations

The only open item is:

  • 1899 (can be seen as bug): Tables should be modified so that restarting a model that contains tables is possible (which is currently not the case).

Leo Gall told me that he will finalize the first regression testing soon but that he did not yet detect any critical issue (one current issue: the comparison program crashed for some models and he has to manually check them).

My conclusion is that we should soon fix #1899 (say latest on Monday), wait for the regression testing results of Leo Gall and then release RC1 (say on Tuesday or Wednesday next week).

Martin, given on where we are with changes to MSL, commits and testing, I can suggest two variants on how to proceed with the release:

  1. We continue to use Dymola only to determine that MSL is actually ok and
    • Tag a beta-3 after all things are fixed
    • Run a complete regression test with Dymola
    • and re-tag as RC-1 if there are no errors.
  2. We check with several tools before calling something an RC, so
    • Tag a beta-3 after all things are fixed
    • Run a complete regression test with Dymola, SimulationX, OpenModelica, JModelica and maybe more platforms
    • and re-tag as RC-1 if there are no errors in any platform

With errors I mean regressions compared to those parts of MSL and ModelicaTest that worked for these platforms with MSL 3.2.1, latest official build. That should not be as hard and time-consuming as it sounds, since both JModelica and OpenModelica run the their regression tests nightly, or even after every single build. I am not sure if they run against MSL-trunk though, but I think yes. I also assume SimulationX does run an internal regression test. Thomas? I would feel that we would get a much more vendor-neutral release process if we go to option 2.

And no matter what, going right to an RC after all changes that were done without tagging a beta and running a complete (and internal) regression test seems not good process to me.

modelica-trac-importer commented 7 years ago

Comment by otter on 22 Feb 2016 12:53 UTC Replying to [comment:60 hubertus]:

Replying to [comment:56 otter]:

Here is a summary of the changes/commits since Beta.2:

Made models to work in some tools

  • 1886: Introduced impure vendor annotations

  • 1900: One ModelicaTest model failed in Dymola (and run in JModelica). Stricter tolerances added (the model now simulates in Dymola).

  • 1890: Replaced M*v of Complex matrix/vector by function call, since some tools do not yet support this part of operator overloading.

Minor improvements that have no influence on the usage of MSL (same simulation results)

  • 1884 Removed unqualified import in Spice3 library

  • 1903 Made initialization of switches in Analog.Ideal uniform

  • 1856 Introduced verboseRead flag for readRealMatrix

  • Added missing StopTime annotations in ModelicaTest
  • Some Compiler-specific improvements in ModelicaUtilities.h (noreturn attribute)
  • Minor issues in documentation fixed (spelling errors, HTML tags)
  • Removed dummy and default annotations

The only open item is:

  • 1899 (can be seen as bug): Tables should be modified so that restarting a model that contains tables is possible (which is currently not the case).

Leo Gall told me that he will finalize the first regression testing soon but that he did not yet detect any critical issue (one current issue: the comparison program crashed for some models and he has to manually check them).

My conclusion is that we should soon fix #1899 (say latest on Monday), wait for the regression testing results of Leo Gall and then release RC1 (say on Tuesday or Wednesday next week).

Martin, given on where we are with changes to MSL, commits and testing, I can suggest two variants on how to proceed with the release:

  1. We continue to use Dymola only to determine that MSL is actually ok and * Tag a beta-3 after all things are fixed * Run a complete regression test with Dymola * and re-tag as RC-1 if there are no errors.
  2. We check with several tools before calling something an RC, so * Tag a beta-3 after all things are fixed * Run a complete regression test with Dymola, SimulationX, OpenModelica, JModelica and maybe more platforms * and re-tag as RC-1 if there are no errors in any platform

With errors I mean regressions compared to those parts of MSL and ModelicaTest that worked for these platforms with MSL 3.2.1, latest official build. That should not be as hard and time-consuming as it sounds, since both JModelica and OpenModelica run the their regression tests nightly, or even after every single build. I am not sure if they run against MSL-trunk though, but I think yes. I also assume SimulationX does run an internal regression test. Thomas? I would feel that we would get a much more vendor-neutral release process if we go to option 2.

And no matter what, going right to an RC after all changes that were done without tagging a beta and running a complete (and internal) regression test seems not good process to me.

The plan for testing and releasing MSL 3.2.2 was published in mid-December (see beginning of this ticket). Substantial changes to this plan are much too late now and should have been brought up in December.

The basic idea of this plan is that MA pays two regression tests with Dymola for Beta.2 and RC.1 and that tool vendors have enough time to perform their own tests with MSL trunk. MA has no influence on the tool vendors and cannot force such tests. Also, MA cannot wait with a new MSL version until all tool vendors fully support Modelica 3.2. Note, all new models included in MSL 3.2.2 trunk have been tested at least with two tools (Dymola, and (OpenModelica or SimulationX)) before including this models in the trunk.

The regression test for Beta.2 will soon be reported (probably tomorrow; previous information indicates that no essential issue is to be expected). During Alpha and Beta testing people working with different tools (Dymola, SimulationX, OpenModelica, JModelica, AMEsim) have reported issues and these have been all fixed (only #1899 is pending). After Beta.2 only tool-specific fixes have been performed (MSL is correct, but in order that some models work better in some tools, some models have been adapted), as well as fixes that does not have an influence on using the MSL (e.g. tiny corrections of the documentation). This justifies to directly build an RC.1 and no Beta.3 (The current information indicates that the MSL trunk version is successfully running at least in Dymola and in OpenModelica and therefore it is to be expected that RC.1 will be successful for these two tool). RC.1 has also the advantage that the paid work with regression testing with Dymola can start at once. In parallel, tool vendors can make their own regression tests with RC.1. If they still detect issues, an RC.2 will be build after fixing them.

Note, in Dymola 2017 Alpha, all models with the StopTime annotation in Modelica and ModelicaTest translate and simulate successfully. OpenModelica publishes simulation results of Modelica and ModelicaTest (from the trunk) every night. The latest results are here:

Modelica (13 failures) https://test.openmodelica.org/hudson/job/MSL_trunk_Simulation/854/#showFailuresLink

ModelicaTest (24 failures) https://test.openmodelica.org/hudson/job/ModelicaTest_trunk_Simulation/518/#showFailuresLink

Note, none of the failures is with respect to a model newly introduced in MSL 3.2.2 trunk. To my understanding, all these failures are also present for MSL 3.2.1.

modelica-trac-importer commented 7 years ago

Comment by beutlich on 22 Feb 2016 19:46 UTC Replying to [comment:61 otter]:

The regression test for Beta.2 will soon be reported (probably tomorrow; previous information indicates that no essential issue is to be expected).

Are those noisy variables of Modelica.Blocks.Examples.NoiseExamples that rely on a non-fixed seed removed from the regression results?

modelica-trac-importer commented 7 years ago

Comment by hubertus on 22 Feb 2016 20:54 UTC Martin, a couple of comments here.

The release process is mostly volunteer work, and I don't propose to change anything of what was contracted by the MA. One important observation though: there would be almost no issue with whether to call this another beta or an RC if the regression testing were automatic (and it could be that for several tools)! That would really get us to the next level on Quality.

For OpenModelica, I'd be much happier with their testing if they would run the test suite that they would call MSL_trunk_Verify. Their MSL_trunk_Simulate produces a result, but it is not clear that the result is correct. They can probably only verify that their results did not change (much) from the comparable tests they did for 3.2.1, but that would be ok. One of our common problems is how to determine the "golden" reference result for a test. In our internal testing with JModelica we discovered that a few results that we considered reference results form Dymola were actually not good enough, i.e. needed tighter tolerance and sometime more result points to become reference quality.

If we would get, in this ticket, the actual written feedback from OpenModelica, SimulationX and JModelica.org that they are happy with their internal regression testing, I'd feel much happier and the MA does not need to do that testing now. So Martin S, Thomas and Jesper: what do you think from your tests?

There is an open ticket/process for better regression test automation. That would make the release process much better, in particular if it works for several tools.

So I think we have a minor disagreement on criteria for an RC or a beta-version, but proper test automation would hopefully make that almost pointless.

modelica-trac-importer commented 7 years ago

Comment by otter on 23 Feb 2016 08:04 UTC Replying to [comment:62 beutlich]:

Replying to [comment:61 otter]:

The regression test for Beta.2 will soon be reported (probably tomorrow; previous information indicates that no essential issue is to be expected). Are those noisy variables of Modelica.Blocks.Examples.NoiseExamples that rely on a non-fixed seed removed from the regression results?

Modelica.Blocks.Examples.NoiseExamples only contains examples with a fixed (given) global seed. This means that all examples produce always the same result.

modelica-trac-importer commented 7 years ago

Comment by otter on 23 Feb 2016 08:23 UTC Replying to [comment:63 hubertus]:

Martin, a couple of comments here.

The release process is mostly volunteer work, and I don't propose to change anything of what was contracted by the MA. One important observation though: there would be almost no issue with whether to call this another beta or an RC if the regression testing were automatic (and it could be that for several tools)! That would really get us to the next level on Quality.

For OpenModelica, I'd be much happier with their testing if they would run the test suite that they would call MSL_trunk_Verify. Their MSL_trunk_Simulate produces a result, but it is not clear that the result is correct. They can probably only verify that their results did not change (much) from the comparable tests they did for 3.2.1, but that would be ok. One of our common problems is how to determine the "golden" reference result for a test. In our internal testing with JModelica we discovered that a few results that we considered reference results form Dymola were actually not good enough, i.e. needed tighter tolerance and sometime more result points to become reference quality.

If we would get, in this ticket, the actual written feedback from OpenModelica, SimulationX and JModelica.org that they are happy with their internal regression testing, I'd feel much happier and the MA does not need to do that testing now. So Martin S, Thomas and Jesper: what do you think from your tests?

There is an open ticket/process for better regression test automation. That would make the release process much better, in particular if it works for several tools.

So I think we have a minor disagreement on criteria for an RC or a beta-version, but proper test automation would hopefully make that almost pointless.

I agree that the testing should be improved along your discussion lines above. However, this takes time and should be decoupled from the release of MSL 3.2.2. The past experience shows that making regression tests (including generation of reference results) cannot be fully automatic. Reasons are that for MSL all regressions that are claimed to fail must be inspected and either decided this is o.k., the underlying model corrected, or the compare tool improved. Furthermore, the compare tool seems to crash sometimes, a test model may have non-appropriate output variables (e.g. no state states and no outputs), etc. So, the only way seems to be, also in the future, that MA gives a contract to someone who makes this work.

You are right that the part that can be automated can still be improved. Since the regression testing with Dymola (on MSL 3.2.2) seems still not be fully automatic (performing the test, the comparison, producing a summary log), this needs to be fixed first. Afterwards, one can think of to include OpenModelica and JModelica.org in these automatic tests as well. However, most likely this requires again an MA contract.

modelica-trac-importer commented 7 years ago

Comment by jmattsson on 24 Feb 2016 12:24 UTC Replying to [comment:63 hubertus]:

If we would get, in this ticket, the actual written feedback from OpenModelica, SimulationX and JModelica.org that they are happy with their internal regression testing, I'd feel much happier and the MA does not need to do that testing now. So Martin S, Thomas and Jesper: what do you think from your tests?

I set up a manual test run with MSL 3.2.1 and the latest JModelica.org trunk to compare against our regular regression tests, and found the following regressions:

I haven't had time to do a thorough investigation of the problems.

modelica-trac-importer commented 7 years ago

Comment by jmattsson on 24 Feb 2016 12:48 UTC I also discovered a model error that I had earlier missed or mistaken for an issue in the compiler:

Error at line 51, column 60, in file 'Modelica\Math\Random.mo':
Calling function Utilities.initializeImpureRandom(): too many positional arguments
modelica-trac-importer commented 7 years ago

Comment by hubertus on 24 Feb 2016 13:05 UTC This is an error in Dymola as well, both for check and translate. Why has this not been discovered? We don't have sufficient test coverage it seems, and certainly should run an error-free check of the complete Library. I repeat my conclusion fro a few days ago: the release testing needs to be done in a fully automated way, otherwise there is a risk that important steps are missed.

modelica-trac-importer commented 7 years ago

Comment by hubertus on 24 Feb 2016 13:08 UTC

I set up a manual test run with MSL 3.2.1 and the latest JModelica.org trunk to compare against our regular regression tests, and found the following regressions: * Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonQuasiStationary * Significant result changes * Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonPullInStroke * Simulation failed * Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMPM_Inverter_MultiPhase * Significant result changes * Modelica.Fluid.Examples.Tanks.TanksWithOverflow * Initialization failed * Modelica.Fluid.Examples.BranchingDynamicPipes * Minor result changes

I haven't had time to do a thorough investigation of the problems.

Toni, Martin S. and Francesco, are there also significant (minor) result changes between 3.2.1 and 3.2.2 in Dymola and/or OpenModelica

modelica-trac-importer commented 7 years ago

Comment by sjoelund.se on 24 Feb 2016 13:14 UTC We have been busy preparing for OM 1.9.4 release which is not focused on 3.2.2. But the matrixVectorMultiply broke some things in the backend and the given magnetic models I think started failing. We have not compared MSL trunk with reference files from 3.2.1.

modelica-trac-importer commented 7 years ago

Comment by beutlich on 24 Feb 2016 13:34 UTC Replying to [comment:60 hubertus]:

I also assume SimulationX does run an internal regression test. Thomas?

SimulationX runs daily and nightly regression test against MSL and ModelicaTest with different solvers. However, we still run against MSL 3.2.1 to validate our own code changes after internal development migration and restructuring. The MSL 3.2.2 trunk tests were not automated so far, thus the issues we found during early beta testing are results when manually testing the newly introduced example models.

modelica-trac-importer commented 7 years ago

Comment by otter on 24 Feb 2016 14:27 UTC Replying to [comment:67 jmattsson]:

I also discovered a model error that I had earlier missed or mistaken for an issue in the compiler:

Error at line 51, column 60, in file 'Modelica\Math\Random.mo':
Calling function Utilities.initializeImpureRandom(): too many positional arguments

This has been fixed in 3d14ff3673dc8e9f6b99dce6d6cc99ce14c521b3 (#1917).

modelica-trac-importer commented 7 years ago

Comment by otter on 24 Feb 2016 14:33 UTC Replying to [comment:68 hubertus]:

This is an error in Dymola as well, both for check and translate. Why has this not been discovered? We don't have sufficient test coverage it seems, and certainly should run an error-free check of the complete Library. I repeat my conclusion fro a few days ago: the release testing needs to be done in a fully automated way, otherwise there is a risk that important steps are missed.

I had forgotten that initializeImpureRandom(..) was also called in Modelica.Math.Random.Examples.GenerateRandomNumbers (I only changed it at the other places).

When making changes to the trunk, I usually only make checks for the models/examples that might been influenced by it (and have overlooked one model in this case). From time to time, I make "Check" and "Check with Simulation" for Modelica and ModelicaTest. However, this test takes > 1 hour and therefore it is not practical to do it before every commit. Of course, such a test will be performed before tagging a release, and this release will be additionally checked by regression testing by someone else. Therefore, the test is fine.

modelica-trac-importer commented 7 years ago

Comment by otter on 24 Feb 2016 14:37 UTC Replying to [comment:69 hubertus]:

I set up a manual test run with MSL 3.2.1 and the latest JModelica.org trunk to compare against our regular regression tests, and found the following regressions: * Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonQuasiStationary * Significant result changes * Modelica.Magnetic.FluxTubes.Examples.SolenoidActuator.ComparisonPullInStroke * Simulation failed * Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMPM_Inverter_MultiPhase * Significant result changes * Modelica.Fluid.Examples.Tanks.TanksWithOverflow * Initialization failed * Modelica.Fluid.Examples.BranchingDynamicPipes * Minor result changes

I haven't had time to do a thorough investigation of the problems.

Toni, Martin S. and Francesco, are there also significant (minor) result changes between 3.2.1 and 3.2.2 in Dymola and/or OpenModelica

We are waiting for the regression results performed by Leo (via a contract), including generation of reference results.

modelica-trac-importer commented 7 years ago

Comment by beutlich on 24 Feb 2016 15:14 UTC In #1899 (for MSL 3.2.2) it is proposed to add new API functions for ModelicaStandardTables. With that being the case, I propose to have another look at #1839 which also requires adapted/new API functions which I did not want to introduce for MSL 3.2.2. But working on #1839 certainly requires some more development time.

I also consider #1922 as a strong reason to put the RC1 on hold for now.

modelica-trac-importer commented 7 years ago

Comment by otter on 24 Feb 2016 15:58 UTC Replying to [comment:75 beutlich]:

In #1899 (for MSL 3.2.2) it is proposed to add new API functions for ModelicaStandardTables. With that being the case, I propose to have another look at #1839 which also requires adapted/new API functions which I did not want to introduce for MSL 3.2.2. But working on #1839 certainly requires some more development time.

No, we cannot just start yet another discussion about a not-so-easy to solve problem. Note, this is not a bug, but just a feature that could be improved. It is an inconvenience, but a user can still define exactly what he/she wants.

RC.1 should have been released last Friday according to the plan, and the above topic is one of 100 other topics that would be "nice to improve". Feature freeze was with Beta.1 and this is a new feature. Therefore, neither this nor another new feature should be introduced now.

Note, #1899 proposes two good solutions to fix a BUG:

  1. Use new interface as constructors (you just have to apply the patch and re-generate the Object library).
  2. Keep everything, but add a one-liner in the C-Code to reload a table, if it is not yet loaded (proposal by Hans).

Please, select one of the two options. If you do not have time, I will do it tomorrow (using option (1)).

I also consider #1922 as a strong reason to put the RC1 on hold for now.

If this bug can be fixed soon (within a few days), we can include it in the MSL 3.2.2 release, otherwise the fix needs to wait for the next maintenance release.

modelica-trac-importer commented 7 years ago

Comment by leo.gall on 25 Feb 2016 09:24 UTC Regression testing of v3.2.2-beta.2 showed a few minor result deviations. See #1924 for details.

All simulation results of v3.2.2-beta.2 with Dymola 2017 Dev 4 are available on MA file server: [​sftp://modelica.org:20022/files/RegressionTesting/ReferenceResults/MSL/tags/v3.2.2%2Bbuild.0-beta.2/]

modelica-trac-importer commented 7 years ago

Comment by beutlich on 25 Feb 2016 12:21 UTC Given that

modelica-trac-importer commented 7 years ago

Comment by otter on 25 Feb 2016 13:57 UTC Replying to [comment:78 beutlich]:

Given that * the Dymola regression results of MSL 3.2.2 beta arrived almost 4 weeks behind release schedule

This does not change anything (even if they would have arrived 4 weeks ago, we would stay exactly at the same point, because the regression result just tells us that we have to inspect some models and we can just do it now, best in the next 1-2 days. However, the other improvements performed since Beta.2 are not at all influenced by the result of the regression testing).

* these Dymola regression results of MSL 3.2.2 beta did not consider the proposed comparison signals from all lib officers (comment 1 of #1924)

This does not matter much. Will be included in the next regression testing of RC.1. Since the tables have been correctly used in Dymola before, it is unlikely that an issue is to be expected here. (This is also the only thing that MA can do now; of course MA will avoid to pay for a third regression testing and we there is good confidence that this is not necessary).

* issues #1856 and #1886 (aka #1478) were added late after freeze and heavily modified the library

The last committ of #1856 (read/write matrices) was done 5 weeks ago before Beta.2 was released and it was tested to work in Dymola, OpenModelica, JModelica and SimulationX.

1886 was the introduction of the impure annotation. Again, this change was made before Beta.2 was released. This change was not necessary for Dymola. Due to an intelligent heuristics,

Dymola can cope with impure functions. These changes have been only made in order that impure functions work better in OpenModelica, JModelica, and SimulationX. These tools had 4 weeks time to figure out, whether these changes work for them.

So, to summarize, I do not understand why these tickets should be an argument to not release MSL 3.2.2 RC.1.

* #1899 needs more time for a proper fix (and not just a quick hack)

Will be fixed tomorrow by me (including object library generation). This is not a hack, but a typcial approach for functions that have a cache.

* #1839 should be reconsidered

This is just an inconvenience of the tables, it is not a bug. As previously said there are many other improvement suggestions of this type in the ticket system and why should this one suddenly be treated for RC.1`?

* MSL 3.2.2 trunk is not yet in automatic regression at JModelica.org, OpenModelica or ITI

This was never planned and agreed. You cannot suddenly change the rules. All tool vendors, including the vendors of SimulationX, JModelica.org, OpenModelica had gotten the time schedule and the alpha release in Dec. 16, 2015, more as 2 months ago. If these tool vendors have other priorities (I assume for good reasons), its up to them. MA gave them enough time (2 months).

By experience, there is always panic before an MA deadline, because only if the deadline is approaching the priority to perform work is raising. If a new Beta would be made and scheduled in 2 months, exactly the same would appear again: one day before the next deadline, new issues will come up and the pressure to again shift the deadline for 2 months.

I really would like to ask you to handle MA deadlines in the same way as for your tool. You will not shift a SimulationX deadline for any of the reasons above.

modelica-trac-importer commented 7 years ago

Comment by jmattsson on 25 Feb 2016 14:47 UTC Replying to [comment:79 otter]:

The last committ of #1856 (read/write matrices) was done 5 weeks ago before Beta.2 was released and it was tested to work in Dymola, OpenModelica, JModelica and SimulationX.

On what do you base your statement that this was tested to work in JModelica.org? The list I posted above was regressions, i.e. models that worked for us with 3.2.1 and no longer work.

I had not seen this ticket before (it is cc to map-lib, where I am not a member), so I have not updated the build scripts for JModelica.org to match the changes in Resources/BuildProjects/_readme.txt yet. Modelica.Utilities.Examples.WriteRealMatrixToFile/ReadRealMatrixFromFile fails in JModelica.org.

We decided to accept that some of the new models in this release does not yet work in JModelica.org for the time being, and I'm not arguing for delaying based on that. However, please do not claim that specific library changes were tested to work in JModelica.org without grounds.

modelica-trac-importer commented 7 years ago

Comment by dietmarw on 25 Feb 2016 14:57 UTC Just a small comment regarding feature freeze. And I quote from the release plan above:

Dec. 23, 2015: Feature Freeze. Trunk is tagged as MSL 3.2.2 Alpha.1

So Thomas Beutlich is correct in pointing out that all the new features which are now causing the headaches were implemented long after the freeze. Now if you set up a time triggered release plan then you can only keep the dates unless something occurs that demands more time. And exactly this has happened. New features were added which can be also implemented in a nice and clean way but that simply needs more time and can simply not fit the original time based release schedule.

In general you have two choices in order to provide a proper quality assurance (and this is what we really want, isn't it?). Either:

  1. Time triggered release: Here you will simply have to decide what features you can include. And new once added long after feature freeze won't simply do. Then you will have to make the decision to postpone those features to the release after. So candidates that would need to be taken out again for that release would be #1856, #1886, #1899 since they caused this "disruption" late in the release cycle.

  2. Feature triggered release: Here you pick the features you want to implement. And once they are fixed you can make a release. In this case it is harder to decide on a release plan unless you have a very tight integration of the developers in the release cycle. So when Martin Otter decided that the three tickets from 1. above ought be in MSL 3.2.2 we basically switched from the time triggered release plan to a feature triggered release plan.

Finally, in the past we have put lots of resources and time into making the Modelica Standard Library a proper standard library again. Removing vendor annotations, specifying tool dependent solutions etc. Now creating a release plan which does not consider any problems that came up along the way. Not taking delays of deliverables into account, and at all costs sticking to release date that is forced by Dymola's deadline for libraries is simply not a good way to ensure proper quality. It just smells of Dymola Standard Library all over again and I really hoped we were beyond that.

Remember that we will live with MSL 3.2.2 for a while including other tools. So from a QA point of view I would not be willing to put my signature under a MSL 3.2.2 that is forcedly released ignoring any quality concerns raised and containing hacks because "we don't have time for this".

modelica-trac-importer commented 7 years ago

Comment by leo.gall on 25 Feb 2016 14:59 UTC Replying to [comment:63 hubertus]:

Martin, a couple of comments here.

One of our common problems is how to determine the "golden" reference result for a test. In our internal testing with JModelica we discovered that a few results that we considered reference results form Dymola were actually not good enough, i.e. needed tighter tolerance and sometime more result points to become reference quality.

Please let me know, if the current MA reference results need tighter tolerance or more result points. We can easily change this during generation.

Most of the paid work goes into generating reference results, which are available to all tool vendors. We use Dymola, because most library officers use this tool during development. If models have been developed in a different tool than Dymola, it should be natural to include their native results in the reference suite! We just need the CSV files.

There is an open ticket/process for better regression test automation. That would make the release process much better, in particular if it works for several tools.

Could you please point to this current ticket? I wasn't able to find it. We shouldn't use this ticket here for discussing the process. Probably, we need a follow-up to #1392. Back then, we stopped working on an automated, tool-neutral way of supplying results.

modelica-trac-importer commented 7 years ago

Comment by hubertus on 26 Feb 2016 15:12 UTC Replying to [comment:82 leo.gall]:

Replying to [comment:63 hubertus]:

Please let me know, if the current MA reference results need tighter tolerance or more result points. We can easily change this during generation.

The answer is probably yes, but we will only fully find out if we start to implement multiple tool testing. I consider this necessary from now on, for practical reasons from the next release on.

Most of the paid work goes into generating reference results, which are available to all tool vendors. We use Dymola, because most library officers use this tool during development. If models have been developed in a different tool than Dymola, it should be natural to include their native results in the reference suite! We just need the CSV files.

There is an open ticket/process for better regression test automation. That would make the release process much better, in particular if it works for several tools.

Could you please point to this current ticket? I wasn't able to find it. We shouldn't use this ticket here for discussing the process. Probably, we need a follow-up to #1392. Back then, we stopped working on an automated, tool-neutral way of supplying results.

This ticket should probably be reopened to continue the discussion where we left of. There has been some discussion on this in previous Design meetings, and I wrongly thought it has been captured in a ticket.

modelica-trac-importer commented 7 years ago

Comment by fcasella on 26 Feb 2016 18:54 UTC Replying to [comment:79 otter]:

1886 was the introduction of the impure annotation. Again, this change was made before Beta.2 was released. This change was not necessary for Dymola. Due to an intelligent heuristics, Dymola can cope with impure functions. These changes have been only made in order that impure functions work better in OpenModelica, JModelica, and SimulationX.

Martin, for the record and as the person who opened this ticket, I cannot really agree on this argument. I think we should all agree that the correct functioning of the MSL cannot be based on undocumented and unpublished intelligent heuristics.

Unfortunately, the language spec 3.2r2 didn't help much here, because it based the behaviour of the function on recognizing "impure external functions", which are not defined anywhere. I guess determining whether and external function is pure or impure is an undecidable problem in general, in particular if the external code is not available in source code form...

Therefore, this issue had to be solved properly, taking whatever time it took.

modelica-trac-importer commented 7 years ago

Comment by hubertus on 26 Feb 2016 22:35 UTC Replying to [comment:81 dietmarw]:

Just a small comment regarding feature freeze. And I quote from the release plan above:

Dec. 23, 2015: Feature Freeze. Trunk is tagged as MSL 3.2.2 Alpha.1

So Thomas Beutlich is correct in pointing out that all the new features which are now causing the headaches were implemented long after the freeze. Now if you set up a time triggered release plan then you can only keep the dates unless something occurs that demands more time. And exactly this has happened. New features were added which can be also implemented in a nice and clean way but that simply needs more time and can simply not fit the original time based release schedule.

In general you have two choices in order to provide a proper quality assurance (and this is what we really want, isn't it?). Either:

  1. Time triggered release: Here you will simply have to decide what features you can include. And new once added long after feature freeze won't simply do. Then you will have to make the decision to postpone those features to the release after. So candidates that would need to be taken out again for that release would be #1856, #1886, #1899 since they caused this "disruption" late in the release cycle.

  2. Feature triggered release: Here you pick the features you want to implement. And once they are fixed you can make a release. In this case it is harder to decide on a release plan unless you have a very tight integration of the developers in the release cycle. So when Martin Otter decided that the three tickets from 1. above ought be in MSL 3.2.2 we basically switched from the time triggered release plan to a feature triggered release plan.

Finally, in the past we have put lots of resources and time into making the Modelica Standard Library a proper standard library again. Removing vendor annotations, specifying tool dependent solutions etc. Now creating a release plan which does not consider any problems that came up along the way. Not taking delays of deliverables into account, and at all costs sticking to release date that is forced by Dymola's deadline for libraries is simply not a good way to ensure proper quality. It just smells of Dymola Standard Library all over again and I really hoped we were beyond that.

Remember that we will live with MSL 3.2.2 for a while including other tools. So from a QA point of view I would not be willing to put my signature under a MSL 3.2.2 that is forcedly released ignoring any quality concerns raised and containing hacks because "we don't have time for this".

With some help I gathered some data on the status of a number of the tickets referenced above, and here is the summary:

1856: New features were added very late during the development process to the MSL. As Martin S. has remarked on MD the modularization of the ModelicaStandardTables in three modules ModelicaStandardTables + ModeliaMatIO + zlib actually breaks the backwards-compatibility of the Modelica tools, i.e., a tool supporting MSL 3.2.2 fails to also support MSL 3.2.1. A new ticket will be filed where we need to find a proper solution for it. To me this looks like a candidate for a blocker-ticket.

1886: Vendor-specific annotations (of __Dymola_pure, __OpenModelica_Impure and __Modelon_Impure) of 3 different vendors are introduced for the first time in MSL. The same issue was raised in #1478 but not really considered there and actually left undecided to any later MSL where the impure keyword of Modelica 3.3 can be used. Discussion was closed on Feb. 18 though there not all corner cases covered. Actually I am not happy with this solution at all: We have the impure keyword in Modelica 3.3 but are not allowed to use it. Instead we created a workaround once again. Why didn't we simply do so when #1478 was filed?

1890: Different tools have different problems on the complex matrix-vector-product m*v: There are different ways how to write it but

1899: In #1103 we agreed on reading external table data during an initial algorithm in the blocks of the ModelicaStandardTables. That was fine and was according to the Modelica specification. Now, that it turns out to be fragile - only for the specific scenario of a restarted simulation from a saved initialization state - it needs to be fixed as quick as possible (to meet some deadline).

However, the detailed discussion revealed more open problems and resulted in a required clarification of the Modelica specification (#1907). The discussion also showed new ideas of extending the external object API to save/restore functions. Well, this really should be discussed and a new ticket should be filed such that the ideas do not get lost. Furthermore the fix (or workaround or hack) that was added today by f85205a60371a2ebd01d47d2232808e9b814c85b added new C-API functions to the ModelicaStandardTables. When working on another tables ticket (#1839) Thomas also noticed that new C-API functions would be needed which he did not want to introduce late in the MSL development process. For this reason he rescheduled #1839 to the next milestone. Now it feels strange that there are suddenly other new C-API functions that could be improved further if #1839 was considered again. Additionally, the new functions added by f85205a60371a2ebd01d47d2232808e9b814c85b introduce memory leaks if reading a table from a file fails (for whatever reason). Thomas stated this problem in the ticket. In the previous implementation - where reading was not part of the constructor - there was no memory leak, if the tool called the object destructor after the ModelicaError raised by the read function. Thus it happens by bad luck/design that the new fix of f85205a60371a2ebd01d47d2232808e9b814c85b actually creates new problems.

In summary: to me this does not look like RC-quality, and I think we should have a vote on this on map-lib. There are many voices in this thread that make clear that they are not happy with the quality, so we should really do a project-level opinion gathering. After all, this is a community project with many volunteers putting in time. It is not ok to simply ignore these voices.

modelica-trac-importer commented 7 years ago

Comment by otter on 28 Feb 2016 09:09 UTC Mid december 2015, the release/testing plan for MSL 3.2.2 was proposed in this ticket. As a few assumptions might be not fully clear in this ticket description, let me try to state this in a better way (I wanted to post this on Friday, but did not have etime to do it):

Quality measures/testing for MSL 3.2.2:

  1. All tickets with milestone 3.2.2 are either fixed or shifted to a future milestone.
  2. Issues reported by tool vendors are fixed (even if the original MSL code is Modelica 3.2 compliant) provided this can be performed with reasonable effort in MSL.
  3. The newest Dymola version is successful with "Check" and with "Check with Simulation" using the pedantic flag.
  4. The newest OpenModelica version (latest nightly build) is succesful in the sense that the (simulation) models that fail simulation, are present in MSL 3.2.1 and also fail there (this measurement is evaluated by comparing the statistics of MSL 3.2.1 and of the trunk in the latest nightly build of OpenModelica).
  5. Regression testing is performed with the latest Dymola version with respect to the latest generated (and accepted) reference files and there is either no regression or an explanation why there is a difference.
  6. New reference files are generated (with exception of the table reference files that have been provided by ITI).

For future MSL releases, the above procedure could be further improved (but not for the MSL 3.2.2 release).

modelica-trac-importer commented 7 years ago

Comment by otter on 28 Feb 2016 09:20 UTC Replying to [comment:86 otter]:

Quality measures/testing for MSL 3.2.2:

  1. All tickets with milestone 3.2.2 are either fixed or shifted to a future milestone.
  2. Issues reported by tool vendors are fixed (even if the original MSL code is Modelica 3.2 compliant) provided this can be performed with reasonable effort in MSL.
  3. The newest Dymola version is successful with "Check" and with "Check with Simulation" using the pedantic flag.
  4. The newest OpenModelica version (latest nightly build) is succesful in the sense that the (simulation) models that fail simulation, are present in MSL 3.2.1 and also fail there (this measurement is evaluated by comparing the statistics of MSL 3.2.1 and of the trunk in the latest nightly build of OpenModelica).
  5. Regression testing is performed with the latest Dymola version with respect to the latest generated (and accepted) reference files and there is either no regression or an explanation why there is a difference.
  6. New reference files are generated (with exception of the table reference files that have been provided by ITI).

For future MSL releases, the above procedure could be further improved (but not for the MSL 3.2.2 release).

Status of MSL 3.2.2 testing as of Feb. 28

On Friday (Feb. 26), the last open ticket (at this time instant) with milestone 3.2.2 was fixed and (1)-(3) of the quality measure were fulfilled. The expectation was that (4) will be successful with the next nightly build of OpenModelica and therefore Toni announced the plan to tag the trunk on Sunday or Monday.

However, it turned out that the expectation was wrong (see new ticket #1928).

Since OpenModelica makes tests with respect to trunk it does not make sense to make a new tag of the trunk, before the open issues are fixed. The next step is therefore to fix #1928.

In parallel, I evaluated (4) more carefully for trunk/Modelica (neglecting #1928 for the moment). The result is (see https://trac.openmodelica.org/OpenModelica/ticket/3729 for details), that 6 (simulation) models of the trunk are not present in MSL 3.2.1 and they fail simulation in the latest OpenModelica nightly build. In all cases, the reason is that OpenModelica fails to solve nonlinear or linear algebraic equation system. In Dymola, all these models simulate without a particular message. It is not yet clear to me how to judge these cases.

If quality measures (1)-(4) are fulfilled for the trunk, a new MSL release should be tagged. It does not matter much whether this release is called Beta.3 or RC.1. The only essential practical difference is that for an RC.1 version everything needs to be finalized (e.g. release notes, code comparison file between 3.2.2 and 3.2.1), whereas this is not necessary for a beta release. Independently how this next tagged version is called, quality measures (5) and (6) should be applied on it, especially to generate the latest reference files (so that other tools can make checks with respect to these reference files).

modelica-trac-importer commented 7 years ago

Comment by otter on 28 Feb 2016 10:10 UTC Replying to [comment:85 hubertus]:

With some help I gathered some data on the status of a number of the tickets referenced above, and here is the summary:

1856: New features were added very late during the development process to the MSL. As Martin S. has remarked on MD the modularization of the ModelicaStandardTables in three modules ModelicaStandardTables + ModeliaMatIO + zlib actually breaks the backwards-compatibility of the Modelica tools, i.e., a tool supporting MSL 3.2.2 fails to also support MSL 3.2.1. A new ticket will be filed where we need to find a proper solution for it. To me this looks like a candidate for a blocker-ticket.

You are right, this needs to be fixed (see #1928).

1886: Vendor-specific annotations (of __Dymola_pure, __OpenModelica_Impure and __Modelon_Impure) of 3 different vendors are introduced for the first time in MSL. The same issue was raised in #1478 but not really considered there and actually left undecided to any later MSL where the impure keyword of Modelica 3.3 can be used. Discussion was closed on Feb. 18 though there not all corner cases covered. Actually I am not happy with this solution at all: We have the impure keyword in Modelica 3.3 but are not allowed to use it. Instead we created a workaround once again. Why didn't we simply do so when #1478 was filed?

During analysis of #1886 it was detected that MSL 3.2.1 contains about 30 impure functions (the tables are only part of it, there are others). In the trunk a few new impure functions have been added (random number generators). Since all these functions could give trouble in tools (both in MSL 3.2.1 and the planned 3.2.2), we discussed to improve the situation now. The best compromize seem to be to use impure vendor annotations (because this does not require changes in existing tools, and existing tools will just work better afterwards, without any change).

1890: Different tools have different problems on the complex matrix-vector-product m*v: There are different ways how to write it but

  • SimulationX fails on m*v
  • OpenModelica fails on function matrixVectorProduct(m, v)
  • Dymola also fails on function matrixVectorProduct(m, v) if not completely inlined Now the least elegant solution by array comprehension is implemented again in three places. Is this really the kind of Modelica code we want to see here?

The original code used operator overloading of Complex (Modelica 3.2 compliant code). Since it turned out that some tools do not yet support these features, Toni evaluated possible solutions and finally implemented one that seem to run in Dymola, OpenModelica and SimulationX. We do this all the time in MSL, to sligthly rewrite models in order that they are better supported in tools. This is a standard approach also used in other areas (see e.g. jquery Javascript library that tries to support all web browser and for this introduces much much more ugly code as we do it in MSL).

1899: In #1103 we agreed on reading external table data during an initial algorithm in the blocks of the ModelicaStandardTables. That was fine and was according to the Modelica specification. Now, that it turns out to be fragile - only for the specific scenario of a restarted simulation from a saved initialization state - it needs to be fixed as quick as possible (to meet some deadline).

When implementing the new tables, this scenario was overlooked (in the "old" Dymola-specific table implementation, the Modelica implementation was different and the problem of #1899 did not occur). This is an important application case in all tools (and all tools have the same problem): The goal is to start an analysis in steady-state (or periodic steady-state). However, it is hard or not possible to compute this state in advance. A common approach is therefore to simulate until steady-state and then store this state on file. In practical cases it often takes a LONG time to simulate until this state and therefore it is of practical importance to store this state once and restart from it later. With the new (open source) table implementation, this feature was gone and since practically all industry relevant models have tables, it was no longer possible to restart simulation.

However, the detailed discussion revealed more open problems and resulted in a required clarification of the Modelica specification (#1907).

As Hans pointed out in the ticket, the table case is a simpler problem because the external object is only used for caching. Therefore, the only issue is that the caching works properly for a restart (theoretically, the table could be read in every model evaluation and then no caching would be needed). For other external objects, especially FMUs, this case cannot be solved in current Modelica and needs an API change.

The discussion also showed new ideas of extending the external object API to save/restore functions. Well, this really should be discussed and a new ticket should be filed such that the ideas do not get lost.

Yes, I agree.

Furthermore the fix (or workaround or hack) that was added today by f85205a60371a2ebd01d47d2232808e9b814c85b added new C-API functions to the ModelicaStandardTables. When working on another tables ticket (#1839) Thomas also noticed that new C-API functions would be needed which he did not want to introduce late in the MSL development process.

And because he wants to use "getInstanceName(.)" that is not available in Modelica 3.2.

For this reason he rescheduled #1839 to the next milestone. Now it feels strange that there are suddenly other new C-API functions that could be improved further if #1839 was considered again.

You are right, whenever new functionality is added in the future (instance name logging, options for extrapolation, ...) we need to provide a new function call.

Additionally, the new functions added by f85205a60371a2ebd01d47d2232808e9b814c85b introduce memory leaks if reading a table from a file fails (for whatever reason). Thomas stated this problem in the ticket. In the previous implementation - where reading was not part of the constructor - there was no memory leak, if the tool called the object destructor after the ModelicaError raised by the read function. Thus it happens by bad luck/design that the new fix of f85205a60371a2ebd01d47d2232808e9b814c85b actually creates new problems.

The comment from Thomas was with respect to the initial patch proposal. In the code committed to svn, this problem is resolved (there is no memory leak).

In summary: to me this does not look like RC-quality, and I think we should have a vote on this on map-lib. There are many voices in this thread that make clear that they are not happy with the quality, so we should really do a project-level opinion gathering. After all, this is a community project with many volunteers putting in time. It is not ok to simply ignore these voices.

Hm. As I wrote in https://trac.modelica.org/Modelica/ticket/1867#comment:8/ MSL cannot be tagged now (as originally planned). So I agree with you (but for other reasons). As I also wrote in this comment, it does not matter much whether to call the next version Beta.3 or RC.1 (and both is fine for me). The important point is now to fix the open issue and fulfill the quality measures of https://trac.modelica.org/Modelica/ticket/1867#comment:86.

modelica-trac-importer commented 7 years ago

Comment by otter on 28 Feb 2016 10:27 UTC Replying to [comment:84 fcasella]:

Replying to [comment:79 otter]:

1886 was the introduction of the impure annotation. Again, this change was made before Beta.2 was released. This change was not necessary for Dymola. Due to an intelligent heuristics, Dymola can cope with impure functions. These changes have been only made in order that impure functions work better in OpenModelica, JModelica, and SimulationX.

Martin, for the record and as the person who opened this ticket, I cannot really agree on this argument. I think we should all agree that the correct functioning of the MSL cannot be based on undocumented and unpublished intelligent heuristics.

Unfortunately, the language spec 3.2r2 didn't help much here, because it based the behaviour of the function on recognizing "impure external functions", which are not defined anywhere. I guess determining whether and external function is pure or impure is an undecidable problem in general, in particular if the external code is not available in source code form...

Therefore, this issue had to be solved properly, taking whatever time it took.

For the minutes: I partially disagree (we can discuss this topic at another time more thoroughly).

It is impossible to define all details necessary for implementation of a programming/moodel language in a specification. For every programming/modeling language one needs to expect some basic know-how and pre-knowledge that is utilized. Of course, one can improve the specification more and more, but the final goal of a complete description is not possible.

For Modelica, this is more difficult as for a programming language because the user wants to have a simulation result, and this simulation result depends on the applied symbolic and numeric methods. Since it is not clear what are the best methods, and also the best methods might fail, this cannot be described in a specification.

The "impure function" stuff is on the border: From practical applications one knows that one has to use impure functions in Modelica for some applications. Examples are network communication via TCP/IP or in general all type of hardware drivers. A simple strategy to handle this case in a tool is that all external C-funtions called in when-clauses are treated as impure (meaning, that no simplification is applied on them). The reasoning is: Functions called outside of a when-clause must be "pure", because otherwise the integrator might fail. An impure marking of the function helps to get diagnostics if the user makes an error (but a correct program is not influenced by it). On the other hand, functions in when-clauses can be impure, because this does not influence the numerical integration algorithm. Since by experience it is known that users are forced to call impure, external C-code in when-clauses, a tool should switch off any simplifications of external C-functions in when-clauses (e.g. no common sub-expression elemination).

Note, I do not know the heuristics of Dymola for handling impure external C-functions, but the above sketch is obvious.

modelica-trac-importer commented 7 years ago

Comment by fcasella on 1 Mar 2016 08:58 UTC Apparently, the heuristics you are suggesting cannot be implemented in OMC, see OM:#3645.

modelica-trac-importer commented 7 years ago

Comment by otter on 2 Mar 2016 10:38 UTC Current status:

I plan to inspect the latest OpenModelica nightly check this afternoon (and will report the result in this ticket).

I plan to tag the trunk tomorrow (March 3) in the morning as Beta.3. Please, if any possible, do no longer fill tickets with milestone 3.2.2 (especially not trivial ones with typos, documentation improvements etc.).

Tomorrow (March 3), Leo Gall will make regression testing with respect to the previous reference files and will provide new reference files. For this, please take into account:

On Mo. March 7, there will be a MAP-LIB Meeting/Web-Meeting at the Modelica Design Meeting next week and the further release plan discussed and decided (current planning is to have this meeting 15:30 - 16:30; but this may change).

Proposal for release plan:

modelica-trac-importer commented 7 years ago

Comment by otter on 2 Mar 2016 17:32 UTC Here is an evaluation of the latest nightly check of OpenModelica from March 2, 2016, 7:39 AM comparing MSL 3.2.1 and MSL 3.2.2 on the trunk. Trunk version from Feb. 29 (revision 20160229-152644~git~master-om1) OpenModelica 1.9.4-dev.beta2-120 (2016-03-02 03:29):

An overview of the statistics is given here (including regression with respect to the latest reference files generated by Leo Gall; I checked a few and the regression here seemed to be only at the first point, such as Modelica.Blocks.Examples.BooleanNetwork1):

MSL 3.2.2 (trunk): https://test.openmodelica.org/libraries/MSL_trunk/BuildModelRecursive.html successfully compiled models: 358 from 366 successfully simulated models: 345 from 366 successfull regression test: 235 from 366

MSL 3.2.1: https://test.openmodelica.org/libraries/MSL_3.2.1/BuildModelRecursive.html successfully compiled models:278 from 278 successfully simulated models: 271 from 278 successfull regression test: 222 from 278

I checked the models individually in OpenModelica that did not compile or did not simulate and posed corresponding tickets for OpenModelica (for some reason, with the newest trunk and the same OpenModelica version as above, some of the models compiled and simulated, although in the statistics they are marked as failed).

7 Models of MSL 3.2.2 trunk do not compile: https://trac.openmodelica.org/OpenModelica/ticket/3739: (for MSL 3.2.1, all models compile)

Modelica.Mechanics.Rotational.Examples.GenerationOfFMUs The given system is mixed-determined. [index > 3] Please checkout the option "+maxMixedDeterminedIndex".

Modelica.Mechanics.MultiBody.Examples.Systems.RobotR3.fullRobot [Modelica.Mechanics.MultiBody.Visualizers.FixedShape: 1979:8-1980:83]: PartialModelicaServices is partial, name lookup is not allowed in partial classes.

Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMEE_Generator Too many equations

Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMEE_Generator_MultiPhase Too many equations

Modelica.Magnetic.FluxTubes.Examples.Hysteresis.ThreePhaseTransformerWithRectifier Initialization problem is structurally singular, error found sorting equations

Modelica.Fluid.Examples.HeatingSystem Warnings and then OpenModelica hangs

Modelica.Utilities.Examples.ReadRealMatrixFromFile OMEdit crashes

8 Models of MSL 3.2.2 trunk fail during simulation: https://trac.openmodelica.org/OpenModelica/ticket/3740: (for MSL 3.2.1, 7 models fail during simulation; for MSL 3.2.2 trunk, 12 models fail during simulation; below the models of MSL 3.2.2 trunk are listed that do not fail in MSL 3.2.1 or are not present in MSL 3.2.1):

Modelica.Electrical.Analog.Examples.OpAmps.SignalGenerator Nonlinear solver failed

Modelica.Electrical.Analog.Examples.OpAmps.Multivibrator Nonlinear solver failed

Modelica.Electrical.Spice3.Examples.Spice3BenchmarkFourBitBinaryAdder Simulation successful, but fatal error in GC: Too many heap sections; OMEdit crashes

Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.AIMC_DOL_MultiPhase Underdetermined linear system

Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMPM_Inverter_MultiPhase Underdetermined linear system

Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMR_Inverter_MultiPhase Underdetermined linear system

Modelica.Magnetic.QuasiStatic.FundamentalWave.Examples.BasicMachines.InductionMachines.IMC_DOL Underdetermined linear system

Modelica.Magnetic.QuasiStatic.FundamentalWave.Examples.BasicMachines.SynchronousMachines.SMEE_Generator Nonlinear solver failed

Probably, most of these errors are OpenModelica issues and not an issue of MSL trunk.

modelica-trac-importer commented 7 years ago

Comment by beutlich on 2 Mar 2016 17:41 UTC Replying to [comment:91 otter]:

* Use the table reference files provided by Thomas Beutlich for the comparison.

Just to state it right: The reference files are still generated by Leo Gall. In comment:2 I pointed Leo Gall to the wanted signals for these reference files.

modelica-trac-importer commented 7 years ago

Comment by otter on 3 Mar 2016 05:06 UTC I have tagged the trunk as 3.2.2 Beta.3 (https://svn.modelica.org/projects/Modelica/tags/v3.2.2+build.0-beta.3). Please, test this version.

modelica-trac-importer commented 7 years ago

Comment by leo.gall on 3 Mar 2016 15:16 UTC I ran tests on v3.2.2-beta.3. Please refer to #1949 for details. There is one new deviation for TanksWithOverflow, but I think this was expected. Otherwise I couldn't find new deviations.

modelica-trac-importer commented 7 years ago

Comment by otter on 3 Mar 2016 15:44 UTC Replying to [comment:95 leo.gall]:

I ran tests on v3.2.2-beta.3. Please refer to #1949 for details. There is one new deviation for TanksWithOverflow, but I think this was expected. Otherwise I couldn't find new deviations.

All regressions are explained and fine (so regression test successful)

modelica-trac-importer commented 7 years ago

Comment by leo.gall on 3 Mar 2016 16:59 UTC Reference results of 3.2.2-beta.3 are now available: ​sftp://modelica.org:20022/files/RegressionTesting/ReferenceResults/MSL/tags/v3.2.2%2Bbuild.0-beta.3/

modelica-trac-importer commented 7 years ago

Comment by hubertus on 4 Mar 2016 19:37 UTC A bug was discovered and ticket #1627 reopened. This looks like an important issue lacking a test.

modelica-trac-importer commented 7 years ago

Comment by otter on 6 Mar 2016 23:30 UTC Here is a summary of the status of MSL 3.2.2:

My summary is that the trunk is ready for a release, that testing is fine and MSL 3.2.1 Beta.3 has a reasonable quality. My propsal

  1. Release MSL 3.2.2 RC.1 on Mo. March 7 (after the web meeting)
  2. Start voting on 3.2.2 release on Mo. March 14.
  3. Release MSL 3.2.2 on Mo. March 21.
  4. After the release, regenerate the reference files (new contract to Leo Gall).
modelica-trac-importer commented 7 years ago

Comment by mwetter on 7 Mar 2016 16:48 UTC The library "Buildings 2.1.0 (LBNL)" has been tested successfully with rc.1 and can be added to the list of tested tools.