Add US Navy WAFs/datasets

[mwengren]

Currently, we're missing any relevant US Navy datasets.

[benjwadams]

Which endpoint would we want to populate the Navy harvest WAF from?

[mwengren]

Info dump from email thread. May need to implement below on data.ioos.us if NCEI can't handle WAF maintenance themselves:

I think the NAVY datasets in the old IOOS catalog were actually from OceanNomads: https://ecowatch.ncddc.noaa.gov/ and I think only aggregations from this THREDDS catalog: https://ecowatch.ncddc.noaa.gov/thredds/oceanNomads/catalog_aggs.html

I just tried harvesting the ISO records from that OceanNomads catalog (using the nifty Docker crawler -- don't even need to install python!) and got 19 datasets:

(IOOS3) rsignell@gamone:~> ls iso/onomads
alaska_aggregation_Alaska_best.ncd.iso.xml
hycom_hycom_reg17_agg_HYCOM_Region_17_Aggregation_best.ncd.iso.xml
hycom_hycom_reg1_agg_HYCOM_Region_1_Aggregation_best.ncd.iso.xml
hycom_hycom_reg6_agg_HYCOM_Region_6_Aggregation_best.ncd.iso.xml
hycom_hycom_reg7_agg_HYCOM_Region_7_Aggregation_best.ncd.iso.xml
hycom_hycom_sfc_agg_HYCOM_Surface_Aggregation_best.ncd.iso.xml
ncom_amseas_agg_20091119_20130404_Amseas_May_2010_to_Apr_04_2013_best.ncd.iso.xml
ncom_amseas_agg_AmSeas_Apr_05_2013_to_Current_best.ncd.iso.xml
ncom_ncom_reg10_agg_NCOM_Region_10_Aggregation_best.ncd.iso.xml
ncom_ncom_reg1_agg_NCOM_Region_1_Aggregation_best.ncd.iso.xml
ncom_ncom_reg2_agg_NCOM_Region_2_Aggregation_best.ncd.iso.xml
ncom_ncom_reg5_agg_NCOM_Region_5_Aggregation_best.ncd.iso.xml
ncom_ncom_reg6_agg_NCOM_Region_6_Aggregation_best.ncd.iso.xml
ncom_ncom_reg7_agg_NCOM_Region_7_Aggregation_best.ncd.iso.xml
ncom_ncom_sfc8_agg_NCOM_SFC_8_Aggregation_best.ncd.iso.xml
ncom_ncom_sfc8_hind_agg_NCOM_SFC8_Hindcast_Aggregation_best.ncd.iso.xml
ncom_useast_agg_20091119_20130404_US_East_Nov_19_2009_to_Apr_04_2013_best.ncd.iso.xml
ncom_us_east_agg_US_East_Apr_05_2013_to_Current_best.ncd.iso.xml
ncom_us_east_before_depth_change_agg_US_East_Feb_2009_to_November_18_2009_best.ncd.iso.xml

I think these datasets are mutually exclusive of the NAVY models at http://thredds.ucar.edu/thredds/idd/forecastModels.html which are from FNMOC

P.S. If you want to run the crawler, here's what I did:

./do_harvest.sh onomads.py

where "do_harvest.sh" is:

https://github.com/rsignell-usgs/thredds_iso_harvester/blob/master/examples/do_harvest.sh

and "onomads.py" is

https://github.com/rsignell-usgs/thredds_iso_harvester/blob/master/examples/onomads.py

[mwengren]

For now, let's just use the WAF that Kyle deployed alongside existing Unidata WAF: http://thredds.axiomdatascience.com/iso/oceannomads/.

We still need to create a CKAN organization for US Navy: https://data.ioos.us/organization, and connect the Registry to a CKAN harvest. We don't add new Organizations too often, but I do recall some issues in a new WAF being created automatically for the new org: https://registry.ioos.us/waf/.

[mwengren]

@benjwadams I tried re-creating a US Navy org in CKAN but got the error below. It once existed in the DB, but was deleted via the CKAN UI. Seems like it didn't really disappear from the DB. Can you manually remove?

[mwengren]

@benjwadams Can you look into the database to see why there's a 'us-navy' org when it is not appearing in the UI (https://data.ioos.us/organization)? This prevents creating a new one and harvesting the WAF Kyle created.

[mwengren]

I set this up using a workaround by creating a 'us-navy-org' organization, rather than just 'us-navy', but you can see there's still an orphan org in the CKAN database. If you go to a harvest edit screen and try to change the Org association, you'll still see 'us-navy' in the pulldown: https://data.ioos.us/harvest/edit/us-navy-waf. Ideally we should drop the old one to avoid confusion.

[mwengren]

@benjwadams nevermind what we discussed yesterday about cleaning up the US Navy Org in CKAN. I figured out the problem, I think. I may have just missed the US Navy org on the second page of organizations by accident. Anyway, all records are being harvested properly: https://data.ioos.us/organization/us-navy. Going to close this issue.