Prototype Service to Create Kerchunk Files for NWM Short Range Predictions

[rajadain]

Following up on the work done in #128, create a prototype service using the K8S infrastructure that creates the Kerchunk Combined metadata file for NWM Short Range Predictions every hour, that includes the most recent predictions available.

The service should run roughly every hour. It should:

• [ ] Inspect the s3://noaa-nwm-pds/nwm.*/short_range/ bucket for any new nwm.t*z.short_range.channel_rt.*.conus.nc files
• [ ] For any new files found, create a Kerchunk JSON file for it in the s3://azavea-noaa-hydro-data/noaa-nwm-pds/ bucket
  • See https://github.com/azavea/noaa-hydro-data/blob/tt/evaluate-nwm-streamflow-predictions/src/nwm-predictions/4_convert_nwm_predictions_to_kerchunk_single.ipynb for an example of how this is done
• [ ] Delete any JSON files in s3://azavea-noaa-hydro-data/noaa-nwm-pds/ bucket that no longer exist in s3://noaa-nwm-pds/nwm.*/short_range/
• [ ] Recreate a combined Kerchunk JSON file for all the single JSONs made so far, and save that in the s3://azavea-noaa-hydro-data/noaa-nwm-pds/ bucket
  • See https://github.com/azavea/noaa-hydro-data/blob/tt/evaluate-nwm-streamflow-predictions/src/nwm-predictions/6_convert_nwm_predictions_to_kerchunk_combined.ipynb for an example of how this is done

This service does not need to be production grade. Just a prototype that shows how it can be done.