Creating safe_write_mode for DDH5Writter

[marcosfrenkel]

The DDH5Writter now has a new safe_write_mode that saves each data line in a separate ddh5 file. This makes sure that there are never any lock file issues. It works in the following way:

• When the DDH5Writer.add_data function is run, it will create a .tmp directory in the same place as it would write the normal .ddh5 file.
• A tree structure is created as need that goes day/hour/minute/second_#{number of second folders}. This is done to not make file interactions with the file system slow.
• Inside the specific second folder the individual data files are created with the names "yyyy-mm-dd-hh_mmss{number of second folders}_#{number of data file}
• periodically it will try and reconstruct all of the files. This is done by checking if it is the first reconstruction. If not it will look for a new metadata value stored indicating what was the last file reconstructed, and the reconstruction will only add files after that one. This is done so it does not have to load all of the data every time. The reconstruction happens by default every 60 seconds or 1000 written lines, whichever one comes first. Those values can be changed. If the reconstruction fails here, the logger throws a warning message but the data is safely saved in the .tmp folder
• Once the measurement is finished, all of the data is loaded back, compared to what is in memory and if they match the data.ddh5 gets updated, and the .tmp folder gets deleted. If anything goes wrong while the experiment is happening the operation gets aborted and a __not_reconstructed__.tag file gets created.

A new CLI command has also been added 'reconstruct-data' that takjes a path where a .tmp folder is and reconstructs it into a ddh5 file. This is meant as a backup in case something goes wrong with the measurement and the data was not backed up.

[wpfff]

Just from the description: how often does data get stored in the main file? Is that configurable?

And why the complicated folder structure?

[marcosfrenkel]

The main data file gets written every 1000 lines or 60 seconds, whichever comes first. Both of these are class level variables that can be configured.

The complicated folder structure is to not get any penalties for multiple files. I am scared that after a few thousands or so items in a directory (either other directories or files) the file system will slow down. This complicated folder structure is to reduce the chance of getting to that scenario.

[wpfff]

Hm, I'm wondering if the 1000 lines / 60secs doesn't defeat the entire purpose.

We're only getting into problems if we live plot, for example. With that cadence live plotting isn't even useful anymore in most cases.

[marcosfrenkel]

I can cut in half those numbers or even lower, it works fine with any combination of them

[wpfff]

10 secs? (And easily adjustable)

[marcosfrenkel]

Tested in the testbed, works fine all the data is being stored and reconstructed correctly. Merging