NYHOPS is missing from catalog

[rsignell-usgs]

I was running my wave height comparison notebook here: https://github.com/reproducible-notebooks/coastal_wave_heights and I noticed that NYHOPS wave heights should have been there, but were not.

I investigated and found they are missing from the IOOS catalog.

Probably missing since the move to catalog.ioos.us.

@Bobfrat , the THREDDS server at Stevens (http://colossus.dl.stevens-tech.edu:8080/thredds/catalog.html) doesn't have the ncISO service enabled. In fact it's so old (4.1.2, circa 2009) it probably doesn't even support the ncISO service.

So either one could:

1. Talk to Dave Runnels or Nickitas at Stevens and get them to upgrade THREDDS, hopefully using the Unidata THREDDS Docker container and ask them to enable the ncISO service.

2. Just use the stand-alone version of ncISO and harvest the ISO that way.

Here's the plot from the wave height notebook, with no NYHOPS. :cry:

[ericmbernier]

@rsignell-usgs Hi Rich. I spoke with Nickitas and he is hesitant to invest time into upgrading their THREDDS instance. When attempting to run the stand-alone version of ncISO, the command typically hangs indefinitely. When it doesn't hang I receive back a single .xml file, with 0 bytes worth of data.

I've read both https://github.com/ioos/registry/wiki/Hosting-Your-Own-WAF and https://github.com/ioos/registry/issues/94, hoping to find something that steers me in the right direction. If it is an issue with the xls file, unfortunately the xls link in the "Hosting Your Own WAF" URL currently 404s. If you have any advice, or can point us in the right direction we'd really appreciate it. Thanks!

[mwengren]

@ericmbernier Don't forget to take a look at the github.io docs here: https://ioos.github.io/catalog/pages/registry/waf_scripting/ and here: https://ioos.github.io/catalog/pages/registry/waf_creation/. This may have a lot of the same information, but I think the plan as far as I know it is the Wiki and other information in the ioos/registry repo should be replaced by the github.io site for Catalog. At least, that's the most recent documentation I've been working on. I'm sure it could use an update even so.

@rsignell-usgs If you think there's info in that registry project wiki we should add to the github.io site please let me know or submit a PR.

All the files and examples referenced on github.io should be here: https://github.com/ioos/catalog/tree/master/waf_generation_scripts.

May be the THREDDS server is just too old anyway.

[rsignell-usgs]

@mwengren , yes, the Stevens THREDDS server is too old to run ncISO, so those scripts to gather metadata from nciso services don't work.

I just tried to run my old stand-alone ncISO java code scripts and they failed with:

 Error reported by XML parser processing
  http://www.ngdc.noaa.gov/metadata/published/xsl/nciso2.0/UnidataDD2MI.xsl: Premature end of file.

I downloaded that XSL file, opened it up in Oxygen, and about 30 errors are reported. I don't know anything about XSL, so I don't know if these are important or not.

So I guess we have even more questions:

• @amilan17, are the errors reported by Oxygen on the XSL anything to worry about?
• @kevin-obrien, @noaaroland, is an updated (e.g. working) version of the standalone ncISO java code available somewhere?

[noaaroland]

Error reported by XML parser processing http://www.ngdc.noaa.gov/metadata/published/xsl/nciso2.0/UnidataDD2MI.xsl: Premature end of file.

Why are you using that transform? The code I've released has not used that transform in many many months? What software are you using?

@kevin-obrien, @noaaroland, is an updated (e.g. working) version of the standalone ncISO java code available somewhere?

I downloaded the latest release and ran it on the catalog which I believe you are using with the following command line:

java -Xms1024m -Xmx1024m -jar ncISO-2.3.4.jar -ts http://colossus.dl.stevens-tech.edu:8080/thredds/catalog.xml -num 1 -depth 20 -iso true

and it worked for me producing the attached output.

output.tar.gz

[rsignell-usgs]

@noaaroland, can you provide the link to ncISO-2.3.4?
I didn't find it here: https://geo-ide.noaa.gov/wiki/index.php?title=NcISO#Command_Line_Utility or here: https://github.com/noaaroland/threddsIso/releases

[noaaroland]

Oops. Meant to link the text.

https://github.com/NOAA-PMEL/uafnciso/releases

[rsignell-usgs]

Okay, @ericmbernier , here the recipe then:

1. Get the lastest ncISO that @noaaroland developed:

   $wget https://github.com/NOAA-PMEL/uafnciso/releases/download/2.3.4/ncISO-2.3.4.jar

2. Run this script daily to grab the Stevens forecasts, hopefully before the IOOS catalog harvests from your WAF!

   #!/bin/bash
   for full_cat in \
   http://colossus.dl.stevens-tech.edu:8080/thredds/bight_latest.xml \
   http://colossus.dl.stevens-tech.edu:8080/thredds/apex_latest.xml \
   http://colossus.dl.stevens-tech.edu:8080/thredds/complete_latest.xml
   do
   echo $full_cat
   java -Xms1024m -Xmx1024m -jar ncISO-2.3.4.jar \
   -ts ${full_cat} -num 100 -depth 20 -iso true \
   -waf /var/www/waf/stevens
   done

[ericmbernier]

@rsignell-usgs @mwengren The three harvested ISOs can be seen here: https://data.ioos.us/dataset?q=nyhops

Unfortunately, the naming convention is identical for all three. I can easily rename them during the WAF generation, if desired. If so, please let me know what the titles should be for each.

[benjwadams]

Related: https://github.com/ioos/catalog/pull/61

[benjwadams]

@ericmbernier, is this set now?

[mwengren]

@ericmbernier @benjwadams We need to move the WAF output from ncISO from https://registry.ioos.us/waf/Stevens/ to https://data.ioos.us/ and then harvest that in the Registry under the MARACOOS organization, so the metadata eventually ends up alongside all the other MARACOOS metadata here: https://registry.ioos.us/waf/MARACOOS/ and gets into the Catalog that way.

[ericmbernier]

WAF has been moved, and added to the Harvest Registry.

[rsignell-usgs]

:champagne: