Data to include (and where it's coming from)

[limnoliver]

1. Spatial

   • [x] stream reaches on which we're forecasting (local build from delaware-model-prep but filtered based on model predictions)
   • [x] reservoirs modeled with GLM (local build from delaware-model-prep)

2. Model drivers

   • [x] GEFS archive for forecasting - @jzwart will point us to Caldera location - pattern here
   • [x] NLDAS files for GLM; files here, pattern for data release here
   • [x] modeled flow and temperature at ins/outs of reservoirs for GLM; data, then filtered further for release
   • [x] observed flow and temperature at ins/outs of reservoirs for GLM; data, then filtered further for release

3. Model Config

   • [x] GLM config, see pattern and location of files here Built locally and provided by Hayley.

4. Predictions

   • [x] reservoir downstream predictions GLM, see pattern and location of files here; built locally and provided by Hayley.
   • [x] DWALLIN - which is GLM + SNTemp weighted predictions of temperature along DRB network , see pattern and location of files here; built locally in delaware-model-prep and provided by Sam
   • [x] PGDL-DA forecasts of water temperature (+ metadata) - @jzwart will point us to Caldera location

5. Evaluation

   • [x] paired predictions and observations - @jzwart will point us to Caldera location
   • [x] code to recreate figures/model error?

[jzwart]

GEFS archive for forecasting - @jzwart will point us to Caldera location

Will be here /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/2_forecast_prep/in/met_drivers. There are files there right now, but those are the files from last year (only 5 priority segments). Will ping when they're updated.

PGDL-DA forecasts of water temperature (+ metadata) - @jzwart will point us to Caldera location

Will be here /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/4_model_forecast/out. There are files there right now, but those are from last year. Will ping when they're updated. There will be .nc files for the data and .xml files for the metadata associated with each .nc file

paired predictions and observations - @jzwart will point us to Caldera location

Will be here /caldera/projects/usgs/water/iidd/datasci/water-prediction/evaluate-forecast/out/all_mods_with_obs.rds. There is a file there right now, but it is from last year. Will ping when it's updated.

[jzwart]

GEFS archive for forecasting is ready to go into data release - files are here /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/2_forecast_prep/in/met_drivers . There should 372 .nc files total

[msleckman]

GEFS archive dir /met_drivers added to forecast-data-release-v2/in_data/03_driver/. There are indeed 372 .nc files now there in met_drivers. Will be zipped up in the sb data release repo (forecasting).

[msleckman]

Will be here /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/4_model_forecast/out. There are files there right now, but those are from last year. Will ping when they're updated. There will be .nc files for the data and .xml files for the metadata associated with each .nc file

@jzwart will all the files (.nc + .xml) need to be transferred over and zipped in data release in_data folder?

[jzwart]

Yes, all will need to be transferred - there will likely be >600 files in total (.nc & .xml) so zipping up will probably be easiest?

[msleckman]

yup should the .xml and .nc be zipped up separately or together?

[limnoliver]

Last year, we zipped them up together. See metadata that described how they are stored here: https://github.com/limnoliver/forecast_data_release/blob/main/in_text/text_data_release.yml#L275-L279

[jzwart]

We'll have the output from the forecast pipeline will zip up all the forecast files for each day (so 1 zip file / day), and then we can zip all of those zip files together. The .xml and .nc files will be zipped together in the daily zip files, mirroring what we did last year

[msleckman]

stream reaches on which we're forecasting (local build from delaware-model-prep but filtered based on model predictions)

@limnoliver I can incorporate this into the pipeline, but what do we filter to?

[msleckman]

Will be here /caldera/projects/usgs/water/iidd/datasci/water-prediction/evaluate-forecast/out/all_mods_with_obs.rds. There is a file there right now, but it is from last year. Will ping when it's updated.

I've copied over all_mods_with_obs.rds to forecast-data-release-v2/in_folder in caldera but renamed it all_mods_with_obs_lst_yr.rds. @jzwart is the file now updated?

PGDL-DA forecasts of water temperature (+ metadata) - @jzwart will point us to Caldera location

Will be here /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/4_model_forecast/out. There are files there right now, but those are from last year. Will ping when they're updated. There will be .nc files for the data and .xml files for the metadata associated with each .nc file

I create a PGDL-DA folder in the in_data folder. @jzwart just checking, can I copy over the .nc and .xml files in the /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/4_model_forecast/out

[jzwart]

I've copied over all_mods_with_obs.rds to forecast-data-release-v2/in_folder in caldera but renamed it all_mods_with_obs_lst_yr.rds. @jzwart is the file now updated?

It should be updated next Monday / Tuesday

I create a PGDL-DA folder in the in_data folder. @jzwart just checking, can I copy over the .nc and .xml files in the /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da/4_model_forecast/out

Final model .nc and .xml files will be ready next Monday / Tuesday. @jdiaz4302 has some placeholder files if you need something sooner (but based on check-in meeting, sounds like next Monday / Tuesday will be ok).

[jdiaz4302]

In case they're helpful, the placeholder files can be found in /caldera/projects/usgs/water/iidd/datasci/water-prediction/forecast-pgdl-da-jdiaz/4_model_forecast/out (the -jdiaz being the only different part of the filepath)

[msleckman]

• [ ] reservoirs modeled with GLM (local build from delaware-model-prep)

@limnoliver Is the source of this canonical_lakes_sf.rds as well?

[msleckman]

Model Config GLM config, see pattern and location of files here Built locally and provided by Hayley.

In the caldera repo we only have last years two res glm files in_data/glm3_cal_nhdhr_120022743.nml and in_data/glm3_cal_nhdhr_151957878.nml. Do we have this for neversink ?

We are also missing outlet_depth_preds.csv and I can't see it in the delaware model prep /5_extract/ or 9_collaboration/res/

[limnoliver]

In the caldera repo we only have last years two res glm files in_data/glm3_cal_nhdhr_120022743.nml and in_data/glm3_cal_nhdhr_151957878.nml. Do we have this for neversink ?

I believe those are the current files for this year, and no Neversink files (we haven't modeled Neversink with GLM).

We are also missing outlet_depth_preds.csv and I can't see it in the delaware model prep /5_extract/ or 9_collaboration/res/

Sorry I don't think I updated the reservoir files (just updated the link and text above now). We have everything we need, which is the reservoir_downstream_preds.csv - we won't publish outlet_depth_preds.csv this year.

[msleckman]

@limnoliver

Predictions reservoir downstream predictions GLM, see pattern and location of files here; built locally and provided by Hayley.

Model drivers modeled flow and temperature at ins/outs of reservoirs for GLM; data, then filtered further for release

Does res_io_sntemp.feather belong in driver or predictions?