search

OpenModelica / OpenModelicaLibraryTesting

Test script for OMCompiler+OpenModelicaLibraries
Other
9 stars 13 forks source link

Make sure the master-fmi job uses the latest development version of OMSimulator on master #173

Open casella opened 1 week ago

casella commented 1 week ago

@lochel is restarting work on OMSimulator, planning to implement event detection properly. We should set up the master-fmi job so that the master-fmi test results use the latest development version on the master branch. This allows @lochel to check the impact of his changes to the code on a large base of test cases.

@lochel please confirm that you are going to do the development on master, not on some other branch, or alternatively let us know what this other branch name is, so we can use it for testing.

We are currently mostly relying on the master-fmi-fmpy job to check our FMUs, so it is no problem if some commits by @lochel temporarily break tests in the master-fmi job.

lochel commented 1 week ago

Yes, development for OMSimulator 3 will happen on the master branch, but I will also port the event detection changes to maintenance/v2.1.

casella commented 1 week ago

During development, we can test with master, then we'll decide what to do when you are done.

casella commented 1 week ago

I think it is also interesting to compare the performance of OMSimulator and FMPy.

lochel commented 3 days ago

@adrpo @casella I've now merged the changes for event handling to master. Since the OMSimulator test suite has barely relevant test cases for this feature, I'm curious to see the results from the master-fmi report. This should give us a better picture of the impact of these changes across a broader set of test cases.

adrpo commented 1 day ago

Should be fixed by #178 together with other Linux "Zombi" issues.

adrpo commented 17 hours ago

Seems OMSimulator is about 2 times or more slower than FMPy currently:

OMSimulator 3
[Parallel(n_jobs=16)]: Using backend LokyBackend with 16 concurrent workers.
[Parallel(n_jobs=16)]: Done  40 tasks      | elapsed:  2.8min
[Parallel(n_jobs=16)]: Done 130 tasks      | elapsed: 15.0min
[Parallel(n_jobs=16)]: Done 256 tasks      | elapsed: 30.4min
[Parallel(n_jobs=16)]: Done 418 tasks      | elapsed: 42.6min
[Parallel(n_jobs=16)]: Done 616 tasks      | elapsed: 51.4min
[Parallel(n_jobs=16)]: Done 850 tasks      | elapsed: 56.4min
[Parallel(n_jobs=16)]: Done 1120 tasks      | elapsed: 70.2min
[Parallel(n_jobs=16)]: Done 1426 tasks      | elapsed: 84.8min
[Parallel(n_jobs=16)]: Done 1768 tasks      | elapsed: 108.2min
[Parallel(n_jobs=16)]: Done 2146 tasks      | elapsed: 132.5min
[Parallel(n_jobs=16)]: Done 2560 tasks      | elapsed: 143.6min
[Parallel(n_jobs=16)]: Done 3010 tasks      | elapsed: 161.2min
[Parallel(n_jobs=16)]: Done 3496 tasks      | elapsed: 193.4min
[Parallel(n_jobs=16)]: Done 4018 tasks      | elapsed: 219.6min
[Parallel(n_jobs=16)]: Done 4576 tasks      | elapsed: 245.4min
[Parallel(n_jobs=16)]: Done 5170 tasks      | elapsed: 281.9min
[Parallel(n_jobs=16)]: Done 5800 tasks      | elapsed: 310.2min
[Parallel(n_jobs=16)]: Done 6466 tasks      | elapsed: 345.2min
[Parallel(n_jobs=16)]: Done 7168 tasks      | elapsed: 387.1min
[Parallel(n_jobs=16)]: Done 7906 tasks      | elapsed: 416.0min
[Parallel(n_jobs=16)]: Done 8680 tasks      | elapsed: 448.5min
[Parallel(n_jobs=16)]: Done 9490 tasks      | elapsed: 480.3min
[Parallel(n_jobs=16)]: Done 10336 tasks      | elapsed: 510.6min
[Parallel(n_jobs=16)]: Done 11218 tasks      | elapsed: 534.8min
[Parallel(n_jobs=16)]: Done 12136 tasks      | elapsed: 561.2min
[Parallel(n_jobs=16)]: Done 13090 tasks      | elapsed: 585.5min
[Parallel(n_jobs=16)]: Done 14080 tasks      | elapsed: 608.9min
[Parallel(n_jobs=16)]: Done 15106 tasks      | elapsed: 629.7min
[Parallel(n_jobs=16)]: Done 16168 tasks      | elapsed: 688.0min
[Parallel(n_jobs=16)]: Done 17266 tasks      | elapsed: 714.7min
[Parallel(n_jobs=16)]: Done 18400 tasks      | elapsed: 755.2min
[Parallel(n_jobs=16)]: Done 18590 out of 18590 | elapsed: 762.7min finished

FMPy
[Parallel(n_jobs=16)]: Using backend LokyBackend with 16 concurrent workers.
[Parallel(n_jobs=16)]: Done  40 tasks      | elapsed:  2.1min
[Parallel(n_jobs=16)]: Done 130 tasks      | elapsed:  6.4min
[Parallel(n_jobs=16)]: Done 256 tasks      | elapsed:  9.1min
[Parallel(n_jobs=16)]: Done 418 tasks      | elapsed: 11.9min
[Parallel(n_jobs=16)]: Done 616 tasks      | elapsed: 22.4min
[Parallel(n_jobs=16)]: Done 850 tasks      | elapsed: 31.5min
[Parallel(n_jobs=16)]: Done 1120 tasks      | elapsed: 42.5min
[Parallel(n_jobs=16)]: Done 1426 tasks      | elapsed: 65.1min
[Parallel(n_jobs=16)]: Done 1768 tasks      | elapsed: 88.3min
[Parallel(n_jobs=16)]: Done 2146 tasks      | elapsed: 119.8min
[Parallel(n_jobs=16)]: Done 2560 tasks      | elapsed: 130.6min
[Parallel(n_jobs=16)]: Done 3010 tasks      | elapsed: 150.0min
[Parallel(n_jobs=16)]: Done 3496 tasks      | elapsed: 179.2min
[Parallel(n_jobs=16)]: Done 4018 tasks      | elapsed: 197.7min
[Parallel(n_jobs=16)]: Done 4576 tasks      | elapsed: 215.7min
[Parallel(n_jobs=16)]: Done 5170 tasks      | elapsed: 248.8min
lochel commented 16 hours ago

The code currently performs unnecessary copying and memory allocations at each step. I will optimize this later once the simulation results are correct.

Looking forward to show off by implementing magic performance improvements. However, for now my focus remains on ensuring correct results.

casella commented 15 hours ago

Looking forward to show off by implementing magic performance improvements. However, for now my focus remains on ensuring correct results.

I agree.

@lochel one essential thing to get the correct results is that the StartTime, StopTime, and Interval annotations of the Modelica models are actually used to run the OMSimulator simulations. In case they are not set, appropriate defaults should be used. These issues caused a lot of problems with the fmpy simulations, including very slow simulations due to completly inappropriate default values of the communication interval, or failing simulations due to too long and incorrectly set communication intervals.

It would be best to avoid them entirely with the OMSimulator job.

Please coordinate with @adrpo to this effect.

adrpo commented 13 hours ago

@casella, I think we already are giving the same command line options to FMPy and OMSimulator but I will check to make sure.

adrpo commented 7 hours ago

Fun stuff, fmpy was so fast because it was not found! See #181 and: https://libraries.openmodelica.org/branches/history/master-fmi-fmpy/2024-11-08%2001:05:34..2024-11-22%2010:16:02.html