search

rubendegroote / Networked-Data-Acquisition

Python code for network-based data acquisition for laser spectroscopy
0 stars 0 forks source link

Auto-backup system #12

Closed rubendegroote closed 7 years ago

rubendegroote commented 9 years ago

Since all data goes into a single file, it may be a good idea to have a regular backup going.

Idea: controller keeps track of last time there was a backup. At the end of a running scan, and after a certain time (an hour?), a message is bounced to the Controller, asking the user if he wants a backup. If yes, the Data Server is instructed to do so. This will require briefly pauzing the saving process, making the backup, and unpauzing it. This could perhaps also be done without user permission.

woutergins commented 9 years ago

Idea for the actual backup: communicate the intention to backup to the save-process in the DataServer, and then it can do one of two things:

  1. Use the stdlib shutil to use the os to copy the hdf5 file.
  2. Create a new store, or append to a backup store, and see which data is in the original and not in the backup.Then, fill up the backup.

I would prefer 2, since this means you have manual control of the process, i.e. you are sure it will only start saving again when the backup is complete, and it is possible to do small, incremental backups. Option 1 might be easier to implement. I'll start by implementing an incremental backup system, but let me know what your opinion is.

rubendegroote commented 9 years ago

I would prefer the second as well.

woutergins commented 9 years ago

At the moment, it works like this: save_continuously_dataserver has gotten a backupFlag multiprocessing Event. If this event is set, instead of emptying the pipe and saving the data, the existing file is copied to a new h5 file, after which the pipe is emptied and data is saved in the original file. Currently, this flag is automatically set after 900s (15 minutes). A beginning method has been implemented in DataServer.py to capture the instruction to backup, and set the event from the DataServer process.