fermi-ad / datalogger-to-ml

Scripts for requesting AD Controls data logger data and transforming those to the desired ML output format and destination.
3 stars 0 forks source link

Script run time and file time have fallen out of sync #19

Closed beauremus closed 3 years ago

beauremus commented 3 years ago

If the script fails to generate a data file. The next run of the script will generate a data file from the previous hour. This has happened enough times that we are currently 9 hours behind.

I propose having nanny.py generate files from the calculated start time, the end time of the last found file, until now.

Consider:

    if path.exists(output_file):
        os.remove(output_file)

    with pd.HDFStore(output_file) as hdf:

We remove an existing file because otherwise, we append it to the existing file. If an existing process holds the output_file open, and a new process attempts to remove it, it should fail. Consider failing gracefully.

beauremus commented 3 years ago

Time estimates: