If the script fails to generate a data file. The next run of the script will generate a data file from the previous hour. This has happened enough times that we are currently 9 hours behind.
I propose having nanny.py generate files from the calculated start time, the end time of the last found file, until now.
Consider:
if path.exists(output_file):
os.remove(output_file)
with pd.HDFStore(output_file) as hdf:
We remove an existing file because otherwise, we append it to the existing file.
If an existing process holds the output_file open, and a new process attempts to remove it, it should fail. Consider failing gracefully.
If the script fails to generate a data file. The next run of the script will generate a data file from the previous hour. This has happened enough times that we are currently 9 hours behind.
I propose having
nanny.py
generate files from the calculated start time, the end time of the last found file, until now.Consider:
We remove an existing file because otherwise, we append it to the existing file. If an existing process holds the
output_file
open, and a new process attempts to remove it, it should fail. Consider failing gracefully.