Missing Data

[julienchastang]

@lesserwhirls found a lot missing data products at http://thredds-jetstream.unidata.ucar.edu/thredds/catalog.html. Coordinate to clean up missing data.

[lesserwhirls]

It looks like this pqact isn't linked to the LDM, as all of the datasets in the Forecast Products and Analysis 404.

I would also recommend using these configs, as they make the GSD HRRR datasets local. That means you'll also need this pqact linked to the LDM.

[julienchastang]

Alrighty, thanks. I'll study this issue further to see what's going on.

FYI

https://github.com/Unidata/xsede-jetstream/tree/master/vms/idd-relay/etc https://github.com/Unidata/xsede-jetstream/tree/master/vms/idd-archiver/etc

Incidentally, does newThredds require a certain version of the TDS?

[julienchastang]

Indeed, I think we have a problem here with the missing reference to pqact.forecastProdsAndAna:

EXEC    "pqact -f NGRID|CONDUIT etc/TDS/pqact.forecastProdsAndAna"

We already have on the idd-archiver VM the etc/TDS/pqact.forecastProdsAndAna file so this should be pretty easy.

[lesserwhirls]

Cool!

The config newThredds is for a 4.6.x server, but is configured to store more data locally given that our new hardware can support the increase in storage needed. v4.6.11 (currently used on jetstream) should work just fine.

The pqact.conf entries we use on newThredds machine looks like this:

https://github.com/Unidata/TdsConfig/blob/6a8842ce86b6b65d2e46763bb0b9a51adc04d364/newThredds/pqacts/README.txt#L43-L51

I would not bother with pqact.obsData, as they rely on the netcdf-perl package and perl-decoders, and are being replaced with python versions in the future.

[julienchastang]

In addition, we will have to ask for

REQUEST FSL2 "^GRIB2.FSL.HRRR" hrrr.unidata.ucar.edu

on idd-relay.

Out of curiosty, how come this is not coming from on idd(b).unidata.ucar.edu?

[lesserwhirls]

I think, but not 100% sure, it is because we are getting a special feed from NOAA GSD for these, and not everyone wants the experimental HRRR grids.

[julienchastang]

forecastProdsAndAna is looking better: http://thredds-jetstream.unidata.ucar.edu/thredds/idd/forecastProdsAndAna.html

Onto GSD HRRR.

[julienchastang]

Alrighty, I think we GSD HRRR as well:

http://thredds-jetstream.unidata.ucar.edu/thredds/catalog/grib/HRRR/CONUS_3km/surface/catalog.html

@lesserwhirls is that everything in terms of missing data?

[lesserwhirls]

The case study datasets are missing, but I don't know if we want to duplicate that up there or not:

http://thredds-jetstream.unidata.ucar.edu/thredds/casestudies/catalog.html

Looks like we are also missing "NWS NEXRAD Composite VIP 10 km/" (under Radar)

http://thredds-jetstream.unidata.ucar.edu/thredds/catalog/grib/nexrad/composite/nws/catalog.html

but we are missing that on all of our machines.

[julienchastang]

For the case study data, would it be a simple matter of scping a directory tree? Do you have a host and path for me? Thxs.

[lesserwhirls]

Unfortunately, it's not as easy as an scp. The casestudy data are 1+ TiB, and it can take quite a bit of time to transfer. I would talk with @m1schmidt if you really want caststudy data on jetstream.

[julienchastang]

Sounds good. Jetstream and I2 should be able to handle the date volumes.

[julienchastang]

@m1schmidt is going to rsync the data. Should take a couple of days :-)

[julienchastang]

FYI:

rsync -e ssh -axq /data/ldm/pub/casestudies xxxxx@xxx.xxx.xxx.xx:/data/ldm/pub

[julienchastang]

@m1schmidt has copied over the case study data:

http://thredds-jetstream.unidata.ucar.edu/thredds/casestudies/catalog.html

[julienchastang]

I also update things WRT https://github.com/Unidata/TdsConfig.

@lesserwhirls does that cover everything?

[lesserwhirls]

Looks good! @jrleeman - would it be possible to run the python gallery against thredds-jetstream.unidata.ucar.edu, or does that require changing up the travis matrix?

[julienchastang]

Thanks @lesserwhirls for all your help on this one!

[lesserwhirls]

No worries - thanks for keeping thing up and going on jetstream!

[jrleeman]

Should be able to add it to the build matrix.

[julienchastang]

I am going to close this one out. Can always re-open if we missed anything.

[lesserwhirls]

So the python-gallery runs against the jetstream TDS without issue:

https://github.com/Unidata/python-gallery/pull/78

Thanks @jrleeman!