search

OpenWaterFoundation / cdss-app-snodas-tools

Colorado's Decision Support Systems (CDSS) Snow Data Assimilation System (SNODAS) Tools
8 stars 4 forks source link

Sometimes GIS processing fails - might need a retry #8

Open smalers opened 7 years ago

smalers commented 7 years ago

When running the interactive script to process the full historical period, sometimes an error occurs and processing stops. This may be due to the SNODAS server undergoing regular restart or maintenance. This may not be an issue for SNODAS tools operationally because in most cases the processing will occur at a normal hour (8AM, for example, after SNODAS are published for the day). However, it may be appropriate to add retry loop in processing with a wait, to allow the SNODAS server to have a reasonable amount of time to restart.

This is an enhancement for the future, as long as it is possible to manually process skipped days once they are discovered. Will that require running the subsequent days in order that the 1week change in volume be correct?

egiles16 commented 7 years ago

It is true that if an error occurs with a date of SNODAS data, the 1 week change in volume will be 'NULL' for the date 1 week later.

For example, if 04/04/04 did not correctly download, the 1 week change in volume will be 'NULL' for 04/11/04. If a user manually reran the SNODAS Tools to re-download and calculate the 04/04/04 date, the 1 week change in volume for 04/11/04 would still be 'NULL'. That is until the 04/11/04 date is reprocessed.

To help SNODAS Tools users, I have incorporated a new feature in the SNODASDaily_Interactive.py script. After ALL dates of a range have processed, the script will print a list of dates that unsuccessfully downloaded. The script will ALSO print a list of dates that are affected (by assigning the 'SNODAS_SWE_Volume_1WeekChange_acft' attribute 'NULL') due to the unsuccessful downloads. If all dates are successfully downloaded, a 'All dates successfully downloaded!' message will be printed.

All above messages are printed both to the console and to the log file.

A future enhancement will be to add a try and except block of code to the download function to make sure that the download tries a second (or third) time if the download fails. This would be helpful if the SNODAS FTP site does not allow download because of high traffic. Waiting and then rerunning could fix this issue.