Update initial conditions when restarting

[tadhg-moore]

The restart functionality is really useful but for our forecasting project we would like to assimilate in up-to-date initial conditions, initially for water temperature but potentially for other biogeochemical variables in the future.

We would like the ability to edit the restart file or have the file use the T_in.dat file when restarting. Currently it is not possible to edit the restart file.

[f-baerenbold]

One possibility is to use the existing initial condition file. You can use the output files and create your own initial conditions. As far as I know, the only missing ingredient is then the Seiche energy. I'd like to add the possibility to read Seiche energy from the config file in the future. I guess that this would give you the possibility to restart a simulation and manipulating the initial conditions. I'll try this soon and let you know (or you do it yourself, I can assist if you want).

[f-baerenbold]

The current master branch allows reading the initial seiche energy from the config file (by setting the switch "InitializeSeicheEnergy" to true and by giving a value to seiche_ini). I tested that if you output all state variables directly on the internal grid (i.e. the input file "grid" is the same as the output_depth) and if you use them to restart the simulation (together with seiche energy, i.e. add the E_seiche output value as seiche_ini in the config file), you get very similar results (temperature differences ~ 0.0001°C in my test) for a continuous simulation and for one with a restart in the middle. What is causing the numerical differences is probably the fact that I use the output files which have limited precision. I can develop a feature soon where the code produces a file with the state variables at simulation end at high precision (similar to the initial condition file) and a file with seiche energy and a few other potentially important variables (water level and ice and snow thickness). With this, it should be possible to have a highly accurate state of the system for a restart. It would probably not be a finished restart functionality like the snapshot, but it would allow to do it if you like. Would that help you?

[tadhg-moore]

That sounds like a great fix. And I had tried it using the outputs of U, V, eps & k_min but I was obviously missing the seiche_energy. Can you share a Windows executable with me so I can also run tests with this feature?

[f-baerenbold]

On of the issues is, that the variables in Simstrat are defined on a staggered grid, i.e. U, V, T and S are defined in the center of each layer and k, eps and other turbulence related variables are defined at the faces of layers. Thus, to have the correct model state, the output needs to be on a staggered grid as well. A second issue is that these state variables are not enough. One also needs the seiche energy and the diffusion coefficients of momentum and matter/temperature (num and nuh). If the diffusivities are neglected (i.e. initialized with 0), there will be slight differences in the model results (~1°C after a few days and < 0.1°C after a few weeks at the surface). The option of restarting the model with a defined seiche energy is already implemented (can be given in the config file), but I am not so sure whether to do it with num and nuh because these variables are not read as initial conditions until now.

I can either send you an executable where an output file is generated which can directly be reused as initial conditions and which (together with the seiche energy in the config file) gives you a pretty good restart option but with a certain error during the first.

Or I include num and nuh into the mix, but then you have to change your initial conditions to have two more columns at the end (because the output file for the restart needs to have the same format as the initial conditions). This option creates almost perfect reproducibility of the simulation results without interruption (error < 0.001°C)

Pitfalls: This restart does not yet work in the presence of changing water levels and ice cover. Tell me if this is a problem for you.

[tadhg-moore]

Ok that issue is a bit more complicated in terms of inclusion of the diffusion coefficients.

For our forecasting application, changing water levels & ice cover are not as big of a problem for us. So I would opt for the inclusion of the num and nuh if with the almost perfect reproducibility.

For your test case, are you using the test case that is within this repo? Just so I can make sure to copy a similar file structure.

[f-baerenbold]

There is now a new branch called dev/text_restart where you can check out the config file and the new init cond file. I'll send you the windows executable. The executable generates a file called "save_end_conditions.dat" which can be reused as initial conditions and a file called "save_end_conditions2.dat" where the seiche energy is saved (which can be used in the config file to restart a simulation).

[tadhg-moore]

Thanks very much for that. I have ran some local tests and it seems to be working. This is much better as a text file as it allows for them to be stored easier when running multiple ensembles. I am currently working on implementing this with our framework and will let you know if there are any problems/suggestions there.

[f-baerenbold]

Good to know. I am planning to integrate this restart possibility as a feature to the next release. So I will be happy to receive further comments and suggestions. You mean the text file is easier to work with than the binary file?

[tadhg-moore]

Yes the text file is much easier for accessing and storing the variables which then makes it easier to re-run forecasts when new data becomes available.

I have set the number of points in Input/Grid, but the depths of the interfaces change slightly (~0.05m) on each run. This is likely negligible, but just curious as to why this is?

[f-baerenbold]

Where do you see that the interfaces change? Can you give an example?

[f-baerenbold]

Do you have a changing lake level? For everything else than changing lake levels I can't see any changes in the grid.

[f-baerenbold]

@tadhg-moore There are some changes on the dev/text_restart branch. It is also possible to include ice/snow to the restart. The only thing which does not work perfectly is changing lake level (balanced in-/outflow is ok however). I also created 2 switches, one to save the text file for restart and one to use this text file. If the text file is used for restart, you have to give 8 data columns in the initial condition file; if you don't restart from text, only 6 data columns are required (without num and nuh). I would be happy if you could test the newest version.

Did you also check the AED2 variables? I don't have created a special output for them. To be perfectly accurate, you have to choose the output depth resolution equal to the internal resolution (defined by the grid file).

[tadhg-moore]

Yes I have inflows = outflows for my simulation which was probably leading to the small changes in the grid.

I would be happy to test the new version if you want to send me an executable. No I haven't looked at the AED2 variables yet as I am currently just focused on the physics. But I know that with GLM and restarting with AED2 can get complicated if you have multiple modules switched on.

[rqthomas]

@f-baerenbold I am continuing @tadhg-moore work on forecasting with Simstrat. What is the status of pulling in the capacity for the text restart into the main branch? It seems there is discussion here about restarting water levels. Has there been any updates to get the changing lake level working? Thanks for all these great updates! We have our proof-of-concept forecasts with simstrat that use the ensemble Kalman filter for data assimilation working for Mac and Windows (see other #59 about the Ubuntu binaries)

[f-baerenbold]

simstrat_linux_text_restart.zip

[f-baerenbold]

Hello, I attached the ubuntu executable here. Seems to work. I was busy with other things, so the restart functionality is not integrated into the master yet. But I should have time to do this in Fall. Making it work well for changing water levels is not so easy as far as I remember. It needs some additional work. Maybe I have time to look into it. A fast solution could be to use the existing functionality of saving the state of the simulation as a binary and then overwrite some variables with the text restart (I assume you don't want to change the water level when restarting?).

Good to hear that you have forecasts working with Simstrat.

[rqthomas]

Thank you so much! I will test on our linux system

[f-baerenbold]

This feature is now integrated into the newest version Simstrat 3.03 (see releases)