search

ioos / registry

Getting data services registered in the IOOS Service Registry
http://ioos.github.io/registry/
2 stars 7 forks source link

Make sure that WAFs from THREDDS use the latest UnidataDD2MI.xsl #64

Open rsignell-usgs opened 9 years ago

rsignell-usgs commented 9 years ago

To get proper service identification in CSW responses we need to get folks running ncISO in stand-alone or via THREDDS servers to use the latest version of the UnidataDD2MI.xsl file. See the issues here:

https://github.com/geopython/pycsw/issues/269 https://github.com/Unidata/thredds/issues/75

The new XSL should be in the next release of the TDS and ncISO, but providers can just update their THREDDS server (or ncISO) by just replacing one XSL file and restarting:

Instructions:

On your TDS, please replace:

<tomcat>/webapps/thredds/WEB-INF/classes/resources/xsl/nciso/UnidataDD2MI.xsl

with the file here:

wget  https://raw.githubusercontent.com/Unidata/threddsIso/master/src/main/resources/xsl/nciso/UnidataDD2MI.xsl

and then reload thredds (or restart tomcat)

ebridger commented 9 years ago

Just wanted to clarify for those running ncISO.jar from the command line w/o the custom -xslt option, as we do, that it picks up the latest UnidataDD2MI.xsl (version 2.3.1). E.g. the NERACOOS WAF has correct gmd:protocol elements. http://www.neracoos.org/WAF/UMaine/iso/B01_Met_HistoricRealtime_Agg.xml

After implementing the updates above the NERACOOS / TDS ncISO now matches the WAF.

dpsnowden commented 9 years ago

While we want to have every TDS running the latest STABLE software, I think our preference for metadata management is to create WAFs using the command line version of ncISO as @ebridger suggests. This gives us better configuration control and more flexibility in the long run.

In the near future I hope that every region (and the functional DACS like the glider DAC) will create their own WAF that is populated by them with metadata they are responsible for. I think this opens up the door to having hand authored or manually augmented metadata in addition to the automatically generated ncISO metadata.

rsignell-usgs commented 9 years ago

Agreed! Thanks @ebridger for confirming that the latest version of the stand-alone ncISO is producing the desired results. Indeed, searching http://www.neracoos.org/WAF/UMaine/iso/B01_Met_HistoricRealtime_Agg.xml for "OPeNDAP:OPeNDAP" was successful!

dpsnowden commented 9 years ago

Since this was posted on the registry repo and the registry is up to ate I think this can be closed. It would be great if we had a better page describing the tools that can be used to create a a WAF. I know we've started a few pages that contain some part of the answer but I don't know if they are complete, or clear or redundant. See https://github.com/ioos/registry/wiki/Hosting-Your-Own-WAF https://github.com/ioos/registry/wiki/List-of-RAs-that-maintain-WAFs https://github.com/ioos/registry/wiki/Python-Scripts-for-creating-WAFs

Can someone who understands the topic review the first link and see if it makes sense? @lukecampbell has done this for the glider DAC, @kwilcox has done it for CeNCOOS and AOOS (and I think GLOS).

Also, @amilan17 has created another repo at the request of @shane-axiom and @emiliom to host the XSLTs used in various workflows including ncSOS. Please review them and comment...They are only just being populated now so check back often.

rsignell-usgs commented 7 years ago

I just noticed that the IOOS modeling testbed (COMT) ISO records all have references that says scheme: None for all the DAP, WMS, NCSS, etc endpoints. That causes problems: when we search for all records with OPeNDAP endpoints using CSW, the COMT records don't show up.

@brianmckenna, can you please update the XSL on the comt thredds server (if that's how you are generating the ISO records) as described here and reharvest?

rsignell-usgs commented 7 years ago

@brianmckenna, the file to be replaced is: comt.sura.org:/var/www/thredds_instance/webapps/thredds/WEB-INF/classes/resources/xsl/nciso/UnidataDD2MI.xsl

This file was last modified Aug 26, 2014.

We figured out this problem in November, 2014. ;-(

rsignell-usgs commented 7 years ago

@brianmckenna, I went ahead and updated the .xsl file and restarted the thredds server. The ISO records now correctly identify the serviceType. For example search this ISO record for "OPeNDAP:OPeNDAP":

http://comt.sura.org/thredds/iso/data/comt_1_archive/inundation_tropical/UND_ADCIRC/Hurricane_Ike_2D_final_run_with_waves?catalog=http%3A%2F%2Fcomt.sura.org%2Fthredds%2Fcomt_1_archive_summary.html&dataset=inundation_tropical.UND_ADCIRC.Hurricane_Ike_2D_final_run_with_waves

Does the Testbed WAF get updated automatically, or do you have to manually kick off a harvest?

rsignell-usgs commented 7 years ago

@brianmckenna, I found your WAF generation script in ~bmckenna/WAF, backed up the existing ~bmckenna/WAF/WAF folder, and ran the script to generate new ISOs in the ~bmckenna/WAF/WAF directory

I then backed up the ISO metadata in the /var/www/html/WAF directory (made WAF_20170314.tgz to accompany the WAF_20161111.tgz you had before) and copied over the new ISO records to: /var/www/html/WAF

I don't have permission to reharvest the COMT records on https://registry.ioos.us/harvests and unfortunately I missed the nightly harvest by 38 minutes. :cry:

@brianmckenna or @lukecampbell , can you do the honors of reloading the COMT WAF?

mwengren commented 7 years ago

Just ran it

On Tue, Mar 14, 2017 at 9:10 PM, Rich Signell notifications@github.com wrote:

@brianmckenna https://github.com/brianmckenna, I found your WAF generation script in ~bmckenna/WAF, backed up the existing ~bmckenna/WAF/WAF folder, and ran the script to generate new ISOs in the ~bmckenna/WAF/WAF directory

I then backed up the ISO metadata in the /var/www/html/WAF directory (made WAF_20170314.tgz to accompany the WAF_20161111.tgz you had before) and copied over the new ISO records to: /var/www/html/WAF

I don't have permission to reharvest the COMT records on https://registry.ioos.us/harvests and unfortunately I missed the nightly harvest by 38 minutes. 😢

@brianmckenna https://github.com/brianmckenna or @lukecampbell https://github.com/lukecampbell , can you do the honors of reloading the COMT WAF?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ioos/registry/issues/64#issuecomment-286610783, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZt4QKqskW2tQF07a7Eg_pYt5LSWQnaks5rlzqIgaJpZM4DBGSV .

rsignell-usgs commented 7 years ago

@mwengren, I can see the improved serviceTypes (e.g. OPeNDAP:OPeNDAP) here: https://registry.ioos.us/waf/COMT/inundation_tropical.USF_FVCOM.Hurricane_Ike_3D_final_run_without_waves.xml but the CSW query is still returning Scheme=None.

Is there some additional step that needs to be done to sync the metadata to the CSW side of the house?

lukecampbell commented 7 years ago

@rsignell-usgs I looked at the XML, and the paths that get dropped into "scheme" for services defined under <srv:SV_ServiceIdentification> tags is:

./gmd:identificationInfo/srv:SV_ServiceIdentification/srv:containsOperations/srv:SV_OperationMetadata/srv:connectPoint/gmd:CI_OnlineResource/gmd:protocol/gco:CharacterString/text()

I looked at the XML you pointed at, and it has these tags properly filled in. I looked at another dataset https://registry.ioos.us/waf/AOOS/NOAA_CSDL_ROMS.iso.xml which is filled out the exact same way but the scheme isn't None.

I think the issue is with PyCSW.

lukecampbell commented 7 years ago

I'll try to force a CSW update by clearing the db and reharvesting, should be a few minutes.

lukecampbell commented 7 years ago

Even after clearing and reharvesting CSW, it's still that way. I'll have to ask @benjwadams to dive into the CSW code to find out what's going on.