Inconsistent ncSOS harvesting

[abirger]

While some of the ncSOS instances listed for the same provider indicate harvested datasets, others do not. For example, CARICOOS' "Physical Oceanography Group, University of Maine VIA Wqm historical Buoy Sensor" has a harvested dataset, but SWAN model simulation for Puerto Rico and the US Virgin Islands - does not. It seems that MARACOOS and PacIOOS have the same problem.

I noticed that the all ncSOS in catalog have old-style URL with "version=1.0.0" instead of "acceptVersions=1.0.0". May that be a reason for the harvesting problem?

[lukecampbell]

The datasets you've posted are returning 404: Not Found from the service providers.

The URL for these services comes from geoportal. We don't generate the URL ourselves, if version=1.0.0 is in the URL then it should be coming from geoportal that way.

[lukecampbell]

http://www.ngdc.noaa.gov/geoportal/rest/document?id=%7BF4DFA9A6-8376-43A6-9F12-ABC6013E275B%7D

<srv:containsOperations>
<srv:SV_OperationMetadata>
<srv:operationName>
<gco:CharacterString>GetCapabilities</gco:CharacterString>
</srv:operationName>
<srv:DCP gco:nilReason="unknown"/>
<srv:connectPoint>
<gmd:CI_OnlineResource>
<gmd:linkage>
<gmd:URL>
http://dm1.caricoos.org/thredds/sos/buoys/Historical/VIA/DSG_VIA.wqm.cbibs.historical.nc?service=SOS&version=1.0.0&request=GetCapabilities
</gmd:URL>

[lukecampbell]

Same with SWAN

http://www.ngdc.noaa.gov/geoportal/rest/document?id=%7B7D8975EE-83C6-4F94-B5E6-EFBF1100753E%7D

[lukecampbell]

As far as I can tell this is a data provider issue.

[abirger]

So, if a provider updates the ncSOS to the version that requires different request parameter, that updated service has to be re-registered with geoportal, otherwise catalog will not harvest properly, right?

[dpsnowden]

The outstanding action is to get CariCOOS to disable ncSOS as an access mechanism for gridded data. This is also related to https://github.com/ioos/registry/issues/66. @jcapella, does this make sense?

[jcapella]

You mean to disable the service for gridded data only?

On 12/4/2014 1:59 PM, Derrick Snowden wrote:

The outstanding action is to get CariCOOS to disable ncSOS as an access mechanism for gridded data. This is also related to ioos/registry#66 https://github.com/ioos/registry/issues/66. @jcapella https://github.com/jcapella, does this make sense?

— Reply to this email directly or view it on GitHub https://github.com/ioos/catalog/issues/271#issuecomment-65674054.

[dpsnowden]

Correct, just gridded data.

Thanks!

[jcapella]

Ok.  

We went through something similar a while ago in regards to
duplicate service entries in the catalogs causing registry
problems.  As I remember, due to inheritance it was necessary to
deleted the explicit, duplicate service entries in the sub-catalogs
and the problem was solved (or Anna never brought it up again?).

So, just to be sure, now I have to remove the ncsos service entry in
the master catalog and add it in the non-gridded-data sub catalogs? 
Please advice so that in fixing the wrong entries I don't break
those that work well.

Jorge

On 12/5/2014 6:57 AM, Derrick Snowden
  wrote:

Correct, just gridded data.

  Thanks!
  —
    Reply to this email directly or view
      it on GitHub.

[kknee]

@jcapella yes, I believe your proposed approach should do the trick. thanks!

[lukecampbell]

Is this still an issue?

[abirger]

@lukecampbell , CariCOOS has upgraded to ncSOS v1.2 since then. I don't know if this specific issue is still there; however, testing the CariCOOS services revealed some issues of the ncSOS itself. They are documented, and waiting for resolution.