Adjust SimResList.plot() to account for continued simulations

[arnoutaertgeerts]

The SimResList class does an excellent job of plotting simulation results of different simulations with the same start and stop time.

For continued simulations, it is unnecessary to divide the plot in different segments and give each segment a new name and color. For example:

I see two solutions:

• Adjust the plot() method to take this into account
• Create a new method (e.g. merge()) which merges the different simulation results in a new single result file

[kdavies4]

I see your point. This is tricky. We could handle that in SimResList.plot(), but it makes that method more complicated and SimResList isn't really intended to concatenate results anyway. We could add a merge() to SimRes or as a standalone function, but the details to truely merge the data depend on the format it was loaded from. I wasn't planning to add a method to write or rewrite simulation result files (although SimRes.to_pandas() can export to some standard formats).

The easiest way might be to create a dedicated class that extends from SimRes. There are two abstract methods in SimRes: values and times. As it stands these are populated by the Dymola file loader (_modelicares.io.dymola.readsim). We could create a SimResSequence class that extends from SimRes but has an __init__ method similar to that of SimResList. It would define the values() and times() methods to pull from the SimRes instances stored within it. In theory, everything else (its plot(), min, max, etc.) would fall into place.

What do you think?

[arnoutaertgeerts]

Good thinking, that could be an easy solution!

[kdavies4]

Would you consider taking the first shot at it?

[arnoutaertgeerts]

Sure, I was planning to ;)

[arnoutaertgeerts]

I followed your approach and everything works :) see https://github.com/kdavies4/ModelicaRes/commit/75b565b892196b2b845244424cf53dfffb1d3b27. However, I did had to change values in NamedTuple instances which is something you should normally not do.

If you are okay with my implementation, we can close this.

PS: I think the continue_run() method is a great addition to the module. It enables developers to use external control methods on Dymola models (as is also possible in Matlab). I think the FMU approach works best but I did not yet succeed in running FMUs with the Dymola solvers.

[arnoutaertgeerts]

Some considerations after I tried this with a large simulations:

• Each new simulation result of a continued run holds redundant data of the parameter names etc. When you do a lot of iterations of the continue_run() method, this creates a lot of overhead. The PyFMI module does provide the ability to create a custom result handler which might be useful here.
• If there are too many files, SimResSequence is unable to process these files the way it is programmed now due to Memory issues. We might need to merge the result files first and delete the redundant data for the files to be able to be read by a SimRes instance.
• When exporting a FMU from Dymola, you have the ability to store the results in a .matfile. This .mat file does work for continued simulations by simply adding the new results to the file. If the FMU is able to create such a file, the simulation results should be stored in memory instead of creating extra .txt files. However, it is impossible to know from the fmu object if it has this option. I added a method `move_mat_file() to the fmi simulator to move the .mat file to the result directory but currently I see no generic way to automate this.
• I uploaded different types of FMUs to the directory and they all work for me except for the CS-2.0/1.0-Dymola FMU which is a Co-Simulation FMU with internal Dymola solvers.

Note that when working with continued simulations, the file writing is the bottleneck, especially when the iterations are short. This can cause the following overhead:

• Dymosim: When you have large models, dsin.txt can get big (~50 MB) and it can take a long time to write these files because Python needs to rewrite the entire file everytime you change even a single value.
• PyFMI: Uses .txt files for output which are alot heavier than the compact .mat files of dymosim. Storing the results in memory or using a .mat output file can considerably speed up the simulation. I noticed a significant speed increase of FMUs with this configuration: `fmu('model.fmu', log_level=0, result_handling='memory'

[kdavies4]

I changed the approach to generate new NamedTuple instances for each variable (179571d). I'm not sure if this will reduce the memory load but it could.

[kdavies4]

We could speed up the dsin.txt writing, but it would take some work. Options:

1. Update the file in place. This is probably not safe because the length of the new values may be longer than the old ones and the existing whitespace.
2. Use binary files (dsin.mat) generated from dymosim with the -ib option. We'd need to create functions to read and write to those (using scipy's loadmat and savemat).
3. Write only the modified parameters to a new dsin file. Dymosim apparently has an option to use two files where one overrides the other. Edit: Looking at the documentation now, I think I was wrong on this.

[kdavies4]

I think I'll close this because we've solved the original issue. I'll open a new issue (#27) related to file size and processing time.