NANOOS 52N Service is not in Geoportal

[robragsdale]

@amilan17 the NANOOS; 52 N service (http://data.nanoos.org/52nsos/sos/kvp?service=SOS&request=GetCapabilities&AcceptVersions=1.0.0) is being harvested into the production WAF. This service has been in Catalog most recently, but it was not in there this morning. Could you please look into this?

@emiliom has anything changed recently with your 52N service?

Right now NANOOS does not have any services listed in the Catalog. I would like to resolve this quickly because we just launched the new version.

[fgayanilo]

@amilan17 the GCOOS SOS 52N SOS service is also not listed -- used to be there (http://data.gcoos.org:8080/52nSOS/sos/kvp?service=SOS&request=GetCapabilities&AcceptVersions=1.0.0) -- and nothing was changed on our end.

[amilan17]

@emiliom @robragsdale The ISO metadata record in NANOOS has a validation error: http://www.ngdc.noaa.gov/docucomp/page?xml=NOAA/IOOS/NANOOS/iso/reports/IsoValidationReport.xml&view=isoValidationErrorsReport&custom=default&title=NOAA/IOOS/NANOOS%20Invalid%20Records

@fgayanilo The GCOOS record looks fine - I see it in the EMMA WAF and Geoportal....

[emiliom]

Actually, I don't think the IOOS 52N SOS end point has been on the catalog for a while; see catalog issue 132. It isn't clear why it disappeared in the first place, but that was a while ago. I've been working on restoring and enhancing our SOS service; with some luck and Shane's help on an issue with the 52N SOS sos-injector-db, we'll be live again very soon. Hopefully (fingers crossed) by the end of the the day!

@robragsdale , it sounds like if the SOS endpoint is back online today, it'll be auto-reharvested by the NGDC geoportal overnight, and show up on the Catalog the following morning (Friday)?

Right now NANOOS does not have any services listed in the Catalog. I would like to resolve this quickly because we just launched the new version.

Thanks for your help and attention, Rob. Yes, we're very cognizant of this too, and that's why I'm pushing hard on several fronts (52N SOS, OSU ROMS THREDDS, and ERDDAP OSU climatology / NANOOS WAF).

[emiliom]

The NANOOS SOS endpoint went online last night (10/1; with limited offerings initially, but that's a side issue). It got harvested by NGDC, resulting in updated information available here (unprocessed records) and here (processed records). And no errors! However, it did not make it into GeoPortal today; it was my understanding that would happen automatically the day after (re)harvesting? Though @amilan17 did say elsewhere that it takes "a day or two", but she was including the IOOS Catalog as well, not just Geoportal.

Sorry to prod. I'll be traveling and offline for ~12 days starting on Sunday, so I'd really like to have this service endpoint in good shape all the way to the IOOS Catalog before I leave ... ideally by tomorrow (Friday).

BTW, NGDC's harvesting of the SOS endpoint unfortunately happened last night at a time when there was only one offering present :( I added many more a few hours later. I hope / assume that tonight's reharvesting will update NGDC's WAF with the new set of offerings, and that updated information in turn is what will make it to Geoportal and the IOOS Catalog. And so on.

Same goes for the THREDDS OSU ROMS endpoint that was restored yesterday; it did not come back into Geoportal today.

[dpsnowden]

Anna, Is it possible to do a manual harvest of the waf into Geoportal ? I'd like to get that resolved before Emilio begins his travel if possible.

On Thursday, October 2, 2014, Emilio Mayorga notifications@github.com wrote:

The NANOOS SOS endpoint went online last night (10/1; with limited offerings initially, but that's a side issue). It got harvested by NGDC, resulting in updated information available here (unprocessed records) http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/NANOOS/iso_u/ and here (processed records) http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/NANOOS/iso/. And no errors! However, it did not make it into GeoPortal today http://www.ngdc.noaa.gov/geoportal/catalog/search/browse/browse.page; it was my understanding that would happen automatically the day after (re)harvesting? Though @amilan17 https://github.com/amilan17 did say elsewhere that it takes "a day or two" https://github.com/ioos/catalog/issues/195#issuecomment-57502238, but she was including the IOOS Catalog as well, not just Geoportal.

Sorry to prod. I'll be traveling and offline for ~12 days starting on Sunday, so I'd really like to have this service endpoint in good shape all the way to the IOOS Catalog before I leave ... ideally by tomorrow (Friday).

BTW, NGDC's harvesting of the SOS endpoint unfortunately happened last night at a time when there was only one offering present :( I added many more a few hours later. I hope / assume that tonight's reharvesting will update NGDC's WAF with the new set of offerings, and that updated information in turn is what will make it to Geoportal and the IOOS Catalog. And so on.

Same goes for the THREDDS OSU ROMS endpoint https://github.com/ioos/catalog/issues/195 that was restored yesterday; it did not come back into Geoportal today.

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/57#issuecomment-57747606.

Excuse my brevity, Sent from Gmail Mobile.

[robragsdale]

@emiliom The NANOOS 52N made it into Geoportal last night (http://www.ngdc.noaa.gov/geoportal/catalog/search/resource/details.page?uuid=%7BAFCA152E-AB1C-4C5D-94A6-D1ABE62D9387%7D). It will be harvested by the Catalog later today or tonight (the Catalog harvests every 8 hours).

[robragsdale]

@emiliom The OSU ROMS endpoints are in Geoportal now and in the Catalog.

[emiliom]

Thank you, @robragsdale . I see both in Geoportal, but not in the Catalog. I'll check back in a couple of hours.

[amilan17]

I think that the records are there now. Can you double check?

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Oct 3, 2014 at 5:49 AM, Derrick Snowden notifications@github.com wrote:

Anna, Is it possible to do a manual harvest of the waf into Geoportal ? I'd like to get that resolved before Emilio begins his travel if possible.

On Thursday, October 2, 2014, Emilio Mayorga notifications@github.com wrote:

The NANOOS SOS endpoint went online last night (10/1; with limited offerings initially, but that's a side issue). It got harvested by NGDC, resulting in updated information available here (unprocessed records) http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/NANOOS/iso_u/ and here (processed records) http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/NANOOS/iso/. And no errors! However, it did not make it into GeoPortal today http://www.ngdc.noaa.gov/geoportal/catalog/search/browse/browse.page; it was my understanding that would happen automatically the day after (re)harvesting? Though @amilan17 https://github.com/amilan17 did say elsewhere that it takes "a day or two" https://github.com/ioos/catalog/issues/195#issuecomment-57502238, but she was including the IOOS Catalog as well, not just Geoportal.

Sorry to prod. I'll be traveling and offline for ~12 days starting on Sunday, so I'd really like to have this service endpoint in good shape all the way to the IOOS Catalog before I leave ... ideally by tomorrow (Friday).

BTW, NGDC's harvesting of the SOS endpoint unfortunately happened last night at a time when there was only one offering present :( I added many more a few hours later. I hope / assume that tonight's reharvesting will update NGDC's WAF with the new set of offerings, and that updated information in turn is what will make it to Geoportal and the IOOS Catalog. And so on.

Same goes for the THREDDS OSU ROMS endpoint https://github.com/ioos/catalog/issues/195 that was restored yesterday; it did not come back into Geoportal today.

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/57#issuecomment-57747606.

Excuse my brevity, Sent from Gmail Mobile.

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/57#issuecomment-57784113.

[amilan17]

Disregard last email - just finished the chain of emails confirming that the records are in Geoportal now. At any time, I can run synchronization in geoportal manually if you need to see results quicker. Please don't hesitate to ask.

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Oct 3, 2014 at 9:41 AM, Anna Milan - NOAA Federal < anna.milan@noaa.gov> wrote:

I think that the records are there now. Can you double check?

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Oct 3, 2014 at 5:49 AM, Derrick Snowden notifications@github.com wrote:

Anna, Is it possible to do a manual harvest of the waf into Geoportal ? I'd like to get that resolved before Emilio begins his travel if possible.

On Thursday, October 2, 2014, Emilio Mayorga notifications@github.com wrote:

The NANOOS SOS endpoint went online last night (10/1; with limited offerings initially, but that's a side issue). It got harvested by NGDC, resulting in updated information available here (unprocessed records) http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/NANOOS/iso_u/ and here (processed records) http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/NANOOS/iso/. And no errors! However, it did not make it into GeoPortal today http://www.ngdc.noaa.gov/geoportal/catalog/search/browse/browse.page;

it was my understanding that would happen automatically the day after (re)harvesting? Though @amilan17 https://github.com/amilan17 did say elsewhere that it takes "a day or two" https://github.com/ioos/catalog/issues/195#issuecomment-57502238, but she was including the IOOS Catalog as well, not just Geoportal.

Sorry to prod. I'll be traveling and offline for ~12 days starting on Sunday, so I'd really like to have this service endpoint in good shape all the way to the IOOS Catalog before I leave ... ideally by tomorrow (Friday).

BTW, NGDC's harvesting of the SOS endpoint unfortunately happened last night at a time when there was only one offering present :( I added many more a few hours later. I hope / assume that tonight's reharvesting will update NGDC's WAF with the new set of offerings, and that updated information in turn is what will make it to Geoportal and the IOOS Catalog. And so on.

Same goes for the THREDDS OSU ROMS endpoint https://github.com/ioos/catalog/issues/195 that was restored yesterday; it did not come back into Geoportal today.

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/57#issuecomment-57747606.

Excuse my brevity, Sent from Gmail Mobile.

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/57#issuecomment-57784113.

[emiliom]

Thanks for the offer, @amilan17. @robragsdale, the two updated/restored services are still not on the Catalog (9:50am PDT). If those endpoints were added to Geoportal last night (meaning at least before 2am PDT), shouldn't they be in the Catalog by now, given the 8-hour refresh cycle? Should we wait another couple of hours, or is this looking fishy?

[emiliom]

I'm sorry to pester, but I'd REALLY like to see those services back on the catalog! They're still not there. It's 3pm Eastern (close to the end of the day), so I'm getting a little nervous.

@dpsnowden , is it possible for @lukecampbell to run a manual update just this once? The registry README says that "The IOOS Catalog will automatically harvest records from the NGDC Geoportal every 8 hours", but it doesn't specify at what times those 8-hr-cycle runs occur (eg, 0:00, 8:00, 16:00). FYI, I'm referring to NANOOS 52N SOS and THREDDS end points (those links point to their geoportal details pages, with their uuid's)

It'd be really helpful to not only see the records in the catalog, but also have just enough time to comment if there are first-order issues.

Thanks!

[lukecampbell]

I just started a job to reindex the services.

[robragsdale]

@lukecampbell how long will reindexing take? When should they show up in the catalog?

[lukecampbell]

I'm going to update the README at some point soon but we harvest the datasets every 24 hours at 7:10AM UTC. When we harvest we look at the metadata, the geospatial extents of the data, we run a compliance check on the data if applicable.

Every 24-hours at 6:30AM UTC we reindex our services based on the NGDC geoportal. Geoportal publishes the service endpoints for where we can get the data.

We have two outstanding issues with harvesting the data that we're taking a look at. Our third party libraries that we use to harvest is having issues parsing unicode characters and the other has to do with malformed metadata. Every harvesting cycle we lose anywhere between 400 ~ 700 datasets out of the 4000 due to the service being unavailable, malformed metadata or a bug on our part. We're working on a logging system that will log and maintain the status of harvests to give data providers and the catalog users a very clean interface to monitor their services.

[lukecampbell]

The reindex of the services is complete and I just started reharvesting the datasets. I'm not positive how long it takes to do.

[lukecampbell]

I just manually harvested data.nanoos.org

[robragsdale]

Thanks @lukecampbell. I've updated the README but please take a look to see if you have anything that should be added.

[robragsdale]

merge pull request Update README.md #59

[lukecampbell]

Can you folks add me to the owners list for IOOS?

[emiliom]

I love you, @lukecampbell ! I'd bear your child if I could :) I swear, no one has ever been as excited as me at seeing the content of an SOS endpoint harvested and parsed into a digestible form. It's absolutely wonderful (and a big relief) to see that our 52N SOS endpoint was handled seamlessly, all the way to extracting the datasets, nicely separated by platform type (ie, the effort I put into mapping to proper IOOS platform types paid off very visibly).

Thanks also to everyone who helped along the way: @dpsnowden , @robragsdale , @amilan17 , and of course my mom and my wife.

[lukecampbell]

I'm quite excited as well when things work well. Success is a string of small-wins and I'll take a win where I can get it.

[emiliom]

Success is a string of small-wins and I'll take a win where I can get it.

Indeed. But when a chain of sequential wins happen along a workflow, well, it's nirvana (and very rare!).