reformat needs to be robust to failures

[dwr-psandhu]

• dms_datastore version: 29e5396
• Python version: 3.7
• Operating System: Linux

Description

reformat was run against the raw directory. A file not found for one of the data files causes the entire process to abort

What I Did

reformat --inpath raw --outpath formatted
in_dir=raw,out_dir=formatted,agencies=[],pattern=None
0/429 input files processed for ncro*.csv
Reformatting complete for cdec*.csv. Reformatting failed on these files:
Data None
Reformatting complete for des*.csv. Reformatting failed on these files:
Data None
0/52 input files processed for noaa*.csv
0/765 input files processed for usgs*.csv
noaa generated an exception: [Errno 2] No such file or directory: 'formatted/noaa_aamc1_9414750_elev_1995.csv' with traceback:
Traceback (most recent call last):
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/reformat.py", line 425, in reformat_main
    data = future.result()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/reformat.py", line 395, in reformat
    write_ts_csv(df, newfname, content, chunk_years=True)
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/write_ts.py", line 126, in write_ts_csv
    with open(newfname,'w',newline="\n") as outfile:
FileNotFoundError: [Errno 2] No such file or directory: 'formatted/noaa_aamc1_9414750_elev_1995.csv'

usgs generated an exception: [Errno 2] No such file or directory: 'formatted/usgs_afo_11446500_elev_2007.csv' with traceback:
Traceback (most recent call last):
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/reformat.py", line 425, in reformat_main
    data = future.result()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/reformat.py", line 395, in reformat
    write_ts_csv(df, newfname, content, chunk_years=True)
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/write_ts.py", line 126, in write_ts_csv
    with open(newfname,'w',newline="\n") as outfile:
FileNotFoundError: [Errno 2] No such file or directory: 'formatted/usgs_afo_11446500_elev_2007.csv'

ncro generated an exception: [Errno 2] No such file or directory: 'formatted/ncro_anh2_b95020_elev_1982.csv' with traceback:
Traceback (most recent call last):
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/reformat.py", line 425, in reformat_main
    data = future.result()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/reformat.py", line 395, in reformat
    write_ts_csv(df, newfname, content, chunk_years=True)
  File "/scratch/dms/psandhu/dms_datastore/dms_datastore/write_ts.py", line 126, in write_ts_csv
    with open(newfname,'w',newline="\n") as outfile:
FileNotFoundError: [Errno 2] No such file or directory: 'formatted/ncro_anh2_b95020_elev_1982.csv'

Exiting reformat_main

[dwr-psandhu]

issue seems to be related to reformatted not existing. This should either be the message to the user or the command should just create the directory if it doesn't exist. I prefer the latter

One more minor issue: the help description of inpath is incorrect.

[dwr-psandhu]

It is fairly robust to being re-run, though not sure the logic of how it figures out what has already be done correctly. Closing for now