search

CIROH-UA / ngen-datastream

Tools for running the NextGen National Water Model
GNU General Public License v3.0
7 stars 9 forks source link

s3 urls for community datastream #130

Open JordanLaserGit opened 1 month ago

JordanLaserGit commented 1 month ago

What convention should the urls follow for datastream forcings and outputs in the ciroh community ngen datastream bucket?

Current ngen forcings url example: https://ciroh-community-ngen-datastream.s3.amazonaws.com/20241007/forcing_medium_range/forcings/VPU_01_forcings.nc

nwm convention for short range conus forcings files is https://noaa-nwm-pds.s3.amazonaws.com/nwm.20241003/forcing_short_range/nwm.t00z.short_range.forcing.f001.conus.nc ... https://noaa-nwm-pds.s3.amazonaws.com/nwm.20241003/forcing_short_range/nwm.t00z.short_range.forcing.f018.conus.nc

so ngen forcings created from the above nwm forcings (split by vpu) can be: https://ciroh-community-ngen-datastream.s3.amazonaws.com/ngen.20241003/forcing_short_range/ngen.t00z.short_range.forcing.f001_f018.VPU_01.nc

The same pattern would be used for other time ranges like medium_range by swapping out short_range for the appropriate name.

Thoughts? @jameshalgren @hellkite500 @ZacharyWills

JordanLaserGit commented 3 weeks ago

Anticipating retrospective output, planning on adding a stream subfolder to these files.

For forcings generated from the operational NWM https://ciroh-community-ngen-datastream.s3.amazonaws.com/stream/ngen.20241003/forcing_short_range/VPU_01/ngen.t00z.short_range.forcing.f001_f018.conus.nc

For forcings generated from the retrospective NWM https://ciroh-community-ngen-datastream.s3.amazonaws.com/retrospective/ngen.20181003/VPU_01/ngen.t00z.retrospective.forcing.f001_f018.conus.nc

JordanLaserGit commented 2 weeks ago

The time axis for reanalysis (AnA) forcings moves backwards through time (tm00 > tm01). Should we store these files under a prefix like reanalysis to denote forcings with a backwards time axis?

Something like this: https://ciroh-community-ngen-datastream.s3.amazonaws.com/reanalysis/ngen.20241029/VPU_01/ngen.t00z.analysis_assim_extend.forcing.tm00_tm27.conus.nc

EDIT Something like this: https://ciroh-community-ngen-datastream.s3.amazonaws.com/reanalysis/ngen.20241029/VPU_01/ngen.t00z.analysis_assim_extend.forcing.tm27_tm00.conus.nc

Flipping tm27 and t00 communicates that the time axis is moving forward in time.