Double boundary data after restarting simulation

[C-Rogge]

So i tried to read some boundary data with the fdsreader, which worked quite well. In my last simulation however i noticed some strange behavior: When plotting the boundary data (wall temperature in this case) against the time steps, two "jumps" in the data could be seen (image below). Those discrepancies cannot be seen visually in smokeview nor in DEVC-data, but also in other boundary quantities (like AST etc,).

Also i discovered the total number of timesteps to be higher than they should be - the simulation should have had some 840-ish timesteps calculated, but the data-array included 970.

I also think i know where those two anomalies originated: While simulating, fds crashed two times and had to be restarted again. According to the cluster-logs the first "jump" in the data appeared exactly on the time step of the first crash. The second one matches with the time step of the second crash PLUS the number of timesteps which were calculated twice (e. g. the simulation crashed on 690 seconds, but the restart-backups are only saved every 100 s, so 90 s were simulated twice).

My theory from this observations: When restarting fds from an earlier moment than the last timestep calculated, fds overwrites most of the old data with the data of the newer simulation run. That is expected. However some data (at least boundary data as it seems) seems to be saved in another place while leaving the old data untouched - otherwise the fdsreader could not include this old data in the array. But while smokeview seems to get this different data structures managed in a way, that only one (probably the newer data) is shown, the reader reads in both data sources and attaches both to the same list / array. This leads to the double calculated timeframes also to be shown twice in the data and the plot below; First the data of the first time calculating (yellow), then the data of the second time (red). Thats also the reason why there are around 130 time steps more in the array than there should be - the double calculated timeframes included around 60 and 70 timesteps, respectively.

I don't know enough about the specific structure of the reader or the datasaving of fds and smokeview to pinpoint the exact issue, but i hope my explanations help in doing just that.

[JanVogelsang]

FDS.. fix your code please..

Ok, if SmokeView manages to clean up the mess produced by FDS, I am pretty sure we will be able to do the same. Could you send me your code you used to produce the image above?
I can probably just check the timesteps at which data was output and compare these every time I add new boundary data to remove duplicates. Not entirely sure why the problem occurs though, so I have to check the files generated by FDS first before I can make any assumptions.

[C-Rogge]

So my python code is literally just a _get_global_boundary_dataarrays-Command and a simple plot with matplotlib (and some power point art for the funny triangles).

My fds-file wont help either, id guess, as it is a output data issue. I mean i could send the whole case data, but those are around 4 GB, so... Ideas?

[JanVogelsang]

Fair points, I think I will manage to reproduce the issue myself. Will try it out tomorrow.

[C-Rogge]

I think the following could work:

• Simulating something with a set DT_RESTART
• Manually cancelling the simulation in between two backup points
• If a new .restart-file was produced while cancelling, deleting this file
• Restarting the simulation (which should now start from the last regular backup point set by DT_RESTART)

That should probably emulate the problem.

[JanVogelsang]

I managed to reproduce the issue and will now start investigating the cause of it.

[JanVogelsang]

The data is actually completely duplicated in the output. I fixed the issue using the solution proposed above. Should work now in 1.9.7. Please try it out and tell me if it worked for you and if you encounter any other issues induced by restarting the simulation run in between.

[C-Rogge]

Works like a charm, thank you!

[lu-kas]

Sorry for my late involvement. @JanVogelsang thank you very much for fixing this.

Background: FDS always appends data, disregard if a restart is based on 'old' data. Thus, the overlap leads to duplicated data. I prefer to prevent this by stopping the simulation at the desired restart point, so that there is no overlap (and waste of computing time). This can be achieved by creating a CHID.stop file: it creates a restart point (disregard of the DT_RESTART value, which is not even needed here), and stops the simulation.

[C-Rogge]

@lu-kas Normally i handle it just that way, but our cluster has the fine habit of crashing simulations which take too long in its opinion, so...