tjturnage / cloud-radar-server

Radar server in the cloud for NOAA computing project
MIT License
2 stars 2 forks source link

Update logic for NSE placefile generation with partial data availability #11

Closed lcarlaw closed 2 months ago

lcarlaw commented 6 months ago

Currently, if a user requests data spanning, for example, a 3 hour window but only 1 or 2 of the 3 data files are downloaded successfully (a single file doesn't exist, or it's corrupted, or something else), process.py will exit with "No model data found in foo directory."

Process.py should proceed to generate NSE placefiles for the available data, with an alert sent to the UI indicating partial data availability. The best way to handle this is probably to edit the logic within the query_files function within process.py to allow the scripts to proceed.

lcarlaw commented 6 months ago

This commit starts to address this by outputting model download status tracking information to a simple text file. Should be readable by an dcc.Interval component within a callback to read the text file. This probably won't work if multiple users have concurrent requests, however.

Removed the original multiprocessing functionality. Simpler to track with a serial process.