Cannot access CO-OPS data with pyoos

[ocefpaf]

We cannot access any CO-OPS SOS data with pyoos after 2017-6-16.

https://gist.github.com/ocefpaf/9f111a85360cd43a42fd14beec3d78bf

I know we should be moving away from pyoos but right now we have many notebooks that rely on it to access CO-OPS.

Ping @rsignell-usgs, @jbosch-noaa, and @mwengren here for some advice on how to proceed. I am not sure what are the alternatives available. (Maybe I am just out of the loop of the latest changes in the service.)

[ocefpaf]

My attempts to use an all catalog solution by searching for TABLEDAP and SOS also failed:

https://gist.github.com/ocefpaf/be4f1605894a84254120881c9c84d0dc

Here is a summary of the endpoints found for a 9 day "Boston Light Swim"-like search:

URL: https://data.ioos.us/csw
URL: https://gamone.whoi.edu/csw

In [25]:
dap_urls
Out[25]:
['http://oos.soest.hawaii.edu/thredds/dodsC/pacioos/hycom/global',
 'http://thredds.secoora.org/thredds/dodsC/SECOORA_NCSU_CNAPS.nc',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/204p1_rt.nc',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/162p1_rt.nc',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/166p1_rt.nc',
 'http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_FVCOM_OCEAN_BOSTON_FORECAST.nc',
 'http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_FVCOM_OCEAN_SCITUATE_FORECAST.nc',
 'http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc',
 'http://geoport-dev.whoi.edu/thredds/dodsC/coawst_4/use/fmrc/coawst_4_use_best.ncd',
 'http://www.neracoos.org/thredds/dodsC/UMO/DSG/SOS/A01/Accelerometer/HistoricRealtime/Agg.ncml',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/179p1_rt.nc',
 'http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_FVCOM_OCEAN_MASSBAY_FORECAST.nc',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/231p1_rt.nc',
 'http://oos.soest.hawaii.edu/thredds/dodsC/hioos/satellite/dhw_5km',
 'http://thredds.secoora.org/thredds/dodsC/G1_SST_GLOBAL.nc',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/201p1_rt.nc',
 'http://thredds.cdip.ucsd.edu/thredds/dodsC/cdip/realtime/036p1_rt.nc']

In [26]:
sos_urls
Out[26]:
['http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/036p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/166p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/162p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/204p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/179p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/231p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://www.neracoos.org/thredds/sos/UMO/DSG/SOS/A01/Accelerometer/HistoricRealtime/Agg.ncml?service=SOS&version=1.0.0&request=GetCapabilities',
 'http://thredds.cdip.ucsd.edu/thredds/sos/cdip/realtime/201p1_rt.nc?service=SOS&version=1.0.0&request=GetCapabilities']

In [24]:
erddap_urls
Out[24]:
[]

We should find NDBC 44013 and NOS.CO-OPS 8443970 there, but all we got is NDBC 44029 (http://www.neracoos.org/thredds/sos/UMO/DSG/SOS/A01/Accelerometer/HistoricRealtime/Agg.ncml).

And many cdip.ucsd moorings too :-/

[jbosch-noaa]

Not sure how to help. Perhaps post an issue on the pyoos repo?

[ocefpaf]

Not sure that is a pyoos issue. Pyoos tries to fetch data from https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS but it seems that the data is not there.

I am out of the loop of the changes in the services but I assumed I should be able to find the same data using the catalog but, as you can see from my attempts above, that is not true.

I don't want to hardcode an endpoint to these buoys b/c that defeats the purpose of the catalog driven approach, but I am not sure what we should do next.

@rsignell-usgs do you know if anything has change in the last month? Should we stop relaying on https://opendap.co-ops.nos.noaa.gov/ioos-dif-sos/SOS ? If so what are the alternatives available?

[ocefpaf]

I just remembered that we had the issue with cdip.ucsd showing up even with the bounding box does not include it: https://github.com/SECOORA/skill_score/issues/197

[ocefpaf]

It turns out that one of the buoys we expect to find, NOS.CO-OPS 8443970, disabled the water temperature sensor in Jun 17 2017 4:40PM. See https://tidesandcurrents.noaa.gov/stationhome.html?id=8443970

That means pyoos is doing the right thing there. I will investigate other bounding boxes and time-spans to make sure there are no other issues with the pyoos search and I will open another one to follow up on the catalog search.