acep-uaf / camio-meter-streams

0 stars 0 forks source link

Check script call for download_missing_file.sh #64

Closed nicolemah99 closed 6 months ago

nicolemah99 commented 7 months ago

Every time data_pipeline.sh is executed it checks for updated and missing files as expected. Running the script multiple times in a row seems to always seems to be missing "random" files from "random" events that should have previously been downloaded.

The following event_ids seem to be missing files frequently (as of 03/06/24):

nicolemah99 commented 7 months ago

If the script is stopped midway through download an event folder, the metadata.txt doesn't seem to be getting written to the directory, so when checking and downloading missing files, we are missing logic to handle missing meta_data files.

nicolemah99 commented 7 months ago

Update: When the script is stopped mid download of an event_id, that event folder does not get all the .md5 files. We should either add more logic to identify which files (including .md5) that need to be downloaded or if the directory is partially full just redownload the directory for event_id