Fix broken TAMU dap urls

[rsignell-usgs]

I wrote a little script to query the NGDC CSW for all the OPeNDAP endpoints and found 2785 links but 1100 of them either timed out after 2 seconds or gave 404 errors.

The bad ones are here:https://github.com/rsignell-usgs/system-test/blob/master/Theme_1_Baseline/bad.csv

@mkhoward, about 500 of these bad links are from tamu, and looks like: 'http://barataria.tamu.edu/thredds/dodsC/nam_gom_monthly/vgrd/nam_vgrd_gom_201312.nc' 'http://barataria.tamu.edu/thredds/dodsC/nam_gom_monthly/dswrf/nam_dswrf_gom_200901.nc'

So the immediate problem is that these are timing out -- it looks like the THREDDS server on barataria: http://barataria.tamu.edu/thredds is down.

But the other problem is that we should be harvesting the endpoint for the aggregated data, not the granule datasets.

Can you please provide the THREDDS catalog link that contains the aggregated data? Or better still, create a WAF of ISO metadata that NGDC can harvest?

Thanks, Rich

[rsignell-usgs]

@amilan17, I heard from the TAMU guys that the machine this data resided on had a internal fire and is being rebuilt. So it will be a while before this data comes back online. Can it be suspended?

[amilan17]

Should we keep the previous harvest available until then? Or Clean those out too?

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Sep 26, 2014 at 1:14 PM, Rich Signell notifications@github.com wrote:

@amilan17 https://github.com/amilan17, I heard from the TAMU guys that the machine this data resided on had a internal fire and is being rebuilt. So it will be a while before this data comes back online. Can it be suspended?

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-57007508.

[robragsdale]

@mkhoward @felimongayanilo should we do a cleanout of the TAMU DAP urls from the server http://barataria.tamu.edu/thredds?

[rsignell-usgs]

I would vote to remove these, since we should have the aggregations, not granules in the catalog anyway.

[fgayanilo]

@mkhoward yes I think we should cleanout

[amilan17]

GCOOS is scheduled for a clean up.

https://www.ngdc.noaa.gov/docucomp/collectionSource/list?recordSetId=2604644&componentId=&serviceType=&serviceStatus=&serviceUrl=&search=List+Collection+Sources

You can view the result in EMMA here tomorrow:

http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/GCOOS/iso/

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Sep 26, 2014 at 2:05 PM, FELIMON GAYANILO notifications@github.com wrote:

@mkhoward https://github.com/mkhoward yes I think we should cleanout

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-57013344.

[rsignell-usgs]

@amilan17 , Awesome. Thank you!

[rsignell-usgs]

and.... They are gone! Proof is in cell [35] here. http://nbviewer.ipython.org/gist/rsignell-usgs/3a340219cc62f5919059

[mkhoward]

Anna,

My TDS guy says he has created a WAF to replace the TDS catalog files that had a separate file for each month. He said he sent it to catalog@ioos.noaa.gov mailto:catalog@ioos.noaa.gov on 14-November-2014.

I’m not seeing it on any of the GCOOS pages.

This is the link he sent http://barataria.tamu.edu/iso/ http://barataria.tamu.edu/iso/ Does this work for you?

Best Regards,

Matt

On Sep 29, 2014, at 3:56 PM, Anna Milan notifications@github.com wrote:

GCOOS is scheduled for a clean up.

https://www.ngdc.noaa.gov/docucomp/collectionSource/list?recordSetId=2604644&componentId=&serviceType=&serviceStatus=&serviceUrl=&search=List+Collection+Sources

You can view the result in EMMA here tomorrow:

http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/GCOOS/iso/

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Sep 26, 2014 at 2:05 PM, FELIMON GAYANILO notifications@github.com wrote:

@mkhoward https://github.com/mkhoward yes I think we should cleanout

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-57013344.

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-57228529.

+---------------------------------------------------------------------------------------------------+ | Dr. Matthew K. Howard Research Scientist | | Department of Oceanography Voice: (979)-862-4169 | | Texas A&M University FAX: (979)-847-8879 | | College Station, TX 77843-3146 Mobile: (979)-696-2026 | | http://gcoos.org mkhoward@tamu.edu | +---------------------------------------------------------------------------------------------------+

[robragsdale]

Hi Matt,

IOOS.Catalog@noaa.gov is the correct address. I did not get the url, but I will register http://barataria.tamu.edu/iso/ now and let you know by Friday if harvest was successful. The WAF looks good. Thanks for setting up the WAF.

Thanks, Rob

On Wed, Nov 26, 2014 at 1:12 PM, mkhoward notifications@github.com wrote:

Anna,

My TDS guy says he has created a WAF to replace the TDS catalog files that had a separate file for each month. He said he sent it to catalog@ioos.noaa.gov mailto:catalog@ioos.noaa.gov on 14-November-2014.

I’m not seeing it on any of the GCOOS pages.

This is the link he sent http://barataria.tamu.edu/iso/ < http://barataria.tamu.edu/iso/> Does this work for you?

Best Regards,

Matt

On Sep 29, 2014, at 3:56 PM, Anna Milan notifications@github.com wrote:

GCOOS is scheduled for a clean up.

https://www.ngdc.noaa.gov/docucomp/collectionSource/list?recordSetId=2604644&componentId=&serviceType=&serviceStatus=&serviceUrl=&search=List+Collection+Sources

You can view the result in EMMA here tomorrow:

http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/GCOOS/iso/

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Sep 26, 2014 at 2:05 PM, FELIMON GAYANILO < notifications@github.com> wrote:

@mkhoward https://github.com/mkhoward yes I think we should cleanout

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-57013344.

— Reply to this email directly or view it on GitHub < https://github.com/ioos/registry/issues/56#issuecomment-57228529>.

+---------------------------------------------------------------------------------------------------+

| Dr. Matthew K. Howard Research Scientist | | Department of Oceanography Voice: (979)-862-4169 | | Texas A&M University FAX: (979)-847-8879 | | College Station, TX 77843-3146 Mobile: (979)-696-2026 | | http://gcoos.org mkhoward@tamu.edu | +---------------------------------------------------------------------------------------------------+

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-64687904.

Rob Ragsdale U.S. IOOS Program Phone: 252-518-5957 http://www.ioos.noaa.gov

[dpsnowden]

Matt,

The email is ioos.catalog@noaa.gov. But, now that you've posted this issue, consider us notified. @robragsdale will make sure the update gets in the queue.

Derrick

On Wed, Nov 26, 2014 at 1:12 PM, mkhoward notifications@github.com wrote:

Anna,

My TDS guy says he has created a WAF to replace the TDS catalog files that had a separate file for each month. He said he sent it to catalog@ioos.noaa.gov mailto:catalog@ioos.noaa.gov on 14-November-2014.

I’m not seeing it on any of the GCOOS pages.

This is the link he sent http://barataria.tamu.edu/iso/ < http://barataria.tamu.edu/iso/> Does this work for you?

Best Regards,

Matt

On Sep 29, 2014, at 3:56 PM, Anna Milan notifications@github.com wrote:

GCOOS is scheduled for a clean up.

https://www.ngdc.noaa.gov/docucomp/collectionSource/list?recordSetId=2604644&componentId=&serviceType=&serviceStatus=&serviceUrl=&search=List+Collection+Sources

You can view the result in EMMA here tomorrow:

http://www.ngdc.noaa.gov/metadata/published/NOAA/IOOS/GCOOS/iso/

Anna ~~~~~~~ Anna.Milan@noaa.gov, 303-497-5099 NOAA/NESDIS/NGDC

http://www.ngdc.noaa.gov/metadata/emma ~~~~~~~

On Fri, Sep 26, 2014 at 2:05 PM, FELIMON GAYANILO < notifications@github.com> wrote:

@mkhoward https://github.com/mkhoward yes I think we should cleanout

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-57013344.

— Reply to this email directly or view it on GitHub < https://github.com/ioos/registry/issues/56#issuecomment-57228529>.

+---------------------------------------------------------------------------------------------------+

| Dr. Matthew K. Howard Research Scientist | | Department of Oceanography Voice: (979)-862-4169 | | Texas A&M University FAX: (979)-847-8879 | | College Station, TX 77843-3146 Mobile: (979)-696-2026 | | http://gcoos.org mkhoward@tamu.edu | +---------------------------------------------------------------------------------------------------+

— Reply to this email directly or view it on GitHub https://github.com/ioos/registry/issues/56#issuecomment-64687904.

Derrick Snowden System Architect US IOOS http://www.ioos.noaa.gov 1100 Wayne Ave, Suite 1225 Silver Spring, MD 20912 +1 301 427 2464 (o), +1 301 427 2073 (f)

Find us on Facebook http://www.facebook.com/usioosgov

[mkhoward]

Thanks,

On Nov 26, 2014, at 1:41 PM, Derrick Snowden notifications@github.com wrote:

Matt,

The email is ioos.catalog@noaa.gov. But, now that you've posted this issue, consider us notified. @robragsdale will make sure the update gets in the queue.

Derrick

Apparently, the directions said to let the bird chill in the sink a few hours….

+---------------------------------------------------------------------------------------------------+ | Dr. Matthew K. Howard Research Scientist | | Department of Oceanography Voice: (979)-862-4169 | | Texas A&M University FAX: (979)-847-8879 | | College Station, TX 77843-3146 Mobile: (979)-696-2026 | | http://gcoos.org mkhoward@tamu.edu | +---------------------------------------------------------------------------------------------------+

[robragsdale]

@mkhoward harvest into the test WAF was successful. The process is described in the registry README file. In a few sentences, the service URL is harvested into a production WAF next. It is harvested a day later by the Geoportal server hosted by NGDC. The Catalog will harvest the URL the next day. I will keep watching this as it progresses through the steps.

[robragsdale]

@mkhoward the WAF harvest was successful all the way through to the Catalog