search

ioos / catalog

IOOS Catalog general repo for documentation and issues
https://ioos.github.io/catalog/
MIT License
2 stars 6 forks source link

Add Unidata Motherlode forecast datasets #10

Closed mwengren closed 8 years ago

mwengren commented 8 years ago

Unidata Motherlode was in the previous IOOS Catalog.

Need to investigate metadata sources.

@rsignell-usgs

mwengren commented 8 years ago

@rsignell-usgs WAF is empty at the moment.

http://thredds.axiomdatascience.com/iso/unidata/

I believe this was the source for the Unidata forecasts. Seems that Axiom would have to be consulted on the content.

rsignell-usgs commented 8 years ago

Yes, that's where they used to be. Also there was this helpful WAF of NGDC met datasets also, now also empty: http://thredds.axiomdatascience.com/iso/ndbc_stdmet/

@kwilcox, can we get the unidata and ndbc_stdmet harvesters working again? (or have the WAF folders just moved?)

rsignell-usgs commented 8 years ago

I was going to show Unidata that we were harvesting motherlode, but then was not able to do so. :disappointed:

kwilcox commented 8 years ago

Re: ndbc_stdmet ISO links stopped working: http://dods.ndbc.noaa.gov/thredds/catalog/data/stdmet/tshf1/catalog.html?dataset=data/stdmet/tshf1/tshf1h2010.nc

Re: unidata A few datasets started throwing errors. I'll push a fix to thredds_crawler that will bypass these. https://github.com/Unidata/thredds/issues/649

* Could not process http://thredds-jumbo.unidata.ucar.edu/thredds/catalog/grib/HRRR/CONUS_3km/surface/catalog.xml?dataset=grib/HRRR/CONUS_3km/surface/Best. No serviceName definition found!.
* Could not process http://thredds-jumbo.unidata.ucar.edu/thredds/catalog/grib/HRRR/CONUS_3km/wrfprs/catalog.xml?dataset=grib/HRRR/CONUS_3km/wrfprs/Best. No serviceName definition found!.
* Error procesing http://thredds.ucar.edu/thredds/catalog/grib/NCEP/NAM/Firewxnest/catalog.xml?dataset=grib/NCEP/NAM/Firewxnest/Best/LambertConformal_379X578-34p52N-118p2W, invalid XML
mwengren commented 8 years ago

Resolved via #87

kwilcox commented 8 years ago

What is #87?

mwengren commented 8 years ago

Sorry, wrote that from here, forgot the different repo issue # issue, should have been ioos/catalog-ckan#87.

kwilcox commented 8 years ago

Thanks. Can I remove the WAF at http://thredds.axiomdatascience.com/iso/unidata/?

mwengren commented 8 years ago

Well, that depends. If that's the same as http://thredds.axiomalaska.com/iso/unidata/, then no, because we're depending on that for the metadata source. Otherwise, no problem. At this point we're relying on you guys to harvest that server for metadata... https://registry.ioos.us/harvests/xXy1M2joVL8QQRoZq.

kwilcox commented 8 years ago

It is the same thing. I mistakenly didn't click https://dev-catalog.ioos.us/harvest/unidata-waf and assumed it was a WAF that the catalog started crawling and generating. We'll keep the Unidata WAF up indefinitely.

mwengren commented 8 years ago

Yeah, that's just a mirror of yours. We could move it to data.ioos.us if necessary I imagine, I don't think it would be a problem, but stealing your script would help with that. If you guys would rather not keep that dependency on your side, let's do that.

Actually, if you don't mind sharing, I'm trying to collect example THREDDS/ncISO WAF scripts here in case people need inspiration to create their own.