twVolc / PyCamPermanent

Permanent PiCam (SO2) installation project software
GNU General Public License v3.0
1 stars 2 forks source link

Change how often output data is written to files #123

Open ubdbra001 opened 2 months ago

ubdbra001 commented 2 months ago

Currently the output data files are split into hourly chunks. This should be reassessed, maybe files should represent days.

If changing this then it would also likely be beneficial to increase how often the data is written to a file (so if there are issues the data already processed is not lost). Either write as the data is produced (but this may be too much I/O) or write hourly chunks to a single file per day?

ubdbra001 commented 2 months ago

Code for splitting data into hourly chunks found in save_emission_rates_as_txt() within io_py.py

ubdbra001 commented 1 month ago

Maybe saving should be incremental, but into a single file for each day? Perhaps saving every hour, or half hour, but also saving if it hits an error.

ubdbra001 commented 1 month ago

To-do:

ubdbra001 commented 1 month ago

Filename structure:

[date]_emissionrates[flowtype].csv

twVolc commented 1 month ago

Just to note that this links directly to #46, so once data are written iteratively I think both issues can be closed (just trying to declutter so that the number of issues isn't totally overwhelming).