Ingest data from directly from existing West Coast TDS

[kknee]

@rsignell-usgs apparently this came up during the meeting. Do you have the URL?

[rsignell-usgs]

Do you have any more context? Not exactly sure what this is referring to.

[kknee]

Eoin mentioned that the West Coast COMT team had their own TDS and that we should ingest datasets into sciWMS directly from there. He said you'd have more info on URL? I assumed this was discussed during COMT meeting in early August.

[rsignell-usgs]

Ah, okay. That was the additional context I needed. Here is one:

[rsignell-usgs]

@kknee , the top level catalog for the UCSC modeling datasets is at: http://oceanmodeling.pmc.ucsc.edu:8080/thredds/catalog.html but I'm not sure what catalogs are to be included (if any) for the COMT work.

The best time series catalog http://oceanmodeling.pmc.ucsc.edu:8080/thredds/catalog/ccsnrt/fmrc/catalog.html is already being harvested by IOOS (as evidenced by the screen grab of the NGDC geoportal catalog above)

I'm bringing in Chris Edwards @cedwards2 here, hoping he knows which runs are approriate to include for the COMT work.

[cedwards2]

Hi. You could ingest any of the ones that are listed on the ucsc oceanmodeling webpage, but for the COMT catalog it may be best to wait a few weeks. We'll have 2 new NRT catalogs that should be improved over the NRT system that's currently on there.

[rsignell-usgs]

Chris, sounds good. Thanks!

[rsignell-usgs]

@kknee , does this satisfy your need for now?

[kknee]

@rsignell-usgs Yup, thanks! @cedwards2 can you let us know when the 2 new NRT catalogs are ready?

[kknee]

@cedwards2 there's a lot of interest in getting the west coast data into sciWMS. Should we get started with what's currently on the oceanmodeling page or continue to wait for your updates?

[cedwards2]

Hi.

Ok. This morning I updated our site to serve our near-real-time physical/biological assimilative product. You can find more information at http://oceanmodeling.pmc.ucsc.edu:8080/thredds/catalog/ccsnrt_physbio/fmrc/catalog.html?dataset=ccsnrt_physbio/fmrc/CCSNRT_Phys_Bio_Aggregation_best.ncd

Please note that this is a new product at this point and still developing. I may recreate the whole archive as we find and correct bugs. Will that cause a problem for you?

[kknee]

@cedwards2 my vote is that we get it in, but make sure all potential users are aware that the product is under development. Updating the archive is no problem, just need to manage expectations of users.

[rsignell-usgs]

Yes, we should add this to the catalog. @cedwards2, perhaps you could create a global attribute called "summary" in your catalog, and give a slightly longer description of this simulation, including any caveats about the experimental nature. As an example, here's the featureCollection aggregation NcML (including a "summary") we use for the USGS east coast forecast model:

   <featureCollection name="coawst_4_use" featureType="FMRC" harvest="true" path="coawst_4/use/fmrc">
    <metadata inherited="true">
     <documentation type="summary">ROMS USE Output from COAWST</documentation>
     <serviceName>allServices</serviceName>
    </metadata>
    <collection spec="/usgs/vault0/coawst/coawst_4/Output/use/coawst_us_#yyyyMMdd_HH#.nc$"
     olderThan="6 min"/>
    <update startup="true" rescan="0 34 4-5 * * ? *" trigger="allow"/>

    <protoDataset choice="Penultimate">
     <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
      <attribute name="title" type="String"
       value="COAWST Forecast System : USGS : US East Coast and Gulf of Mexico (Experimental)"/>
      <attribute name="summary" type="String"
       value="Experimental forecast model product from the USGS Coupled Ocean Atmosphere Wave Sediment-Transport (COAWST) modeling system. Data required to drive the modeling system include parametric wave parameters derived from Wave Watch III, wind and atmospheric surface inputs derived from the National Centers for Environmental Prediction (NCEP) North American Mesoscale and Global Forecasting System models, and climatology fields obtained from the Florida State University Center for Ocean-Atmospheric Predictions Studies (COAPS)  HYCOM model. The COAWST model above is run on a 5km (336x896) grid which encompasses the US East Coast and the Gulf of Mexico." /> 
      <variable name="h">
       <attribute name="standard_name" type="String" value="sea_floor_depth" />
      </variable>      
      <variable name="temp">
       <attribute name="_FillValue" type="float" value="0.0"/>
       <attribute name="standard_name" type="String" value="sea_water_potential_temperature" />
      </variable>
      <variable name="salt">
       <attribute name="standard_name" type="String" value="sea_water_salinity" />
       <attribute name="units" type="String" value="0.001"/>
       <attribute name="_FillValue" type="float" value="0.0"/>
      </variable>
      <variable name="Hwave">
       <attribute name="_FillValue" type="float" value="0.0"/>
       <attribute name="standard_name" type="String" value="sea_surface_wave_significant_height" />
      </variable>
      <variable name="u">
       <attribute name="standard_name" type="String" value="x_sea_water_velocity" />
       <attribute name="_FillValue" type="float" value="0.0"/>
      </variable>
      <variable name="v">
       <attribute name="standard_name" type="String" value="y_sea_water_velocity" />
       <attribute name="_FillValue" type="float" value="0.0"/>
      </variable>
      <variable name="ubar">
       <attribute name="standard_name" type="String" value="barotropic_x_sea_water_velocity" />
       <attribute name="_FillValue" type="float" value="0.0"/>
      </variable>
      <variable name="vbar">
       <attribute name="standard_name" type="String" value="barotropic_y_sea_water_velocity" />
       <attribute name="_FillValue" type="float" value="0.0"/>
      </variable>
      <variable name="zeta">
       <attribute name="standard_name" type="String" value="water_surface_height_above_reference_datum" />
       <attribute name="_FillValue" type="float" value="0.0"/>
      </variable>
     </netcdf>
    </protoDataset>
    <fmrcConfig regularize="false" datasetTypes="Best"/>
</featureCollection>

[cedwards2]

Hi. Ok. I made some changes to the Summary explaining that fields will be replaced on an ongoing basis as issues are encountered. Take a look.

[kknee]

@brandonmayer can you tackle? I don't have permission to assign anyone in this repo.

[rsignell-usgs]

@kknee , you should have permission now.

[kknee]

This is now in the COMT catalog: http://comt.sura.org/thredds/remoteCatalogService?catalog=http://oceanmodeling.pmc.ucsc.edu:8080/thredds/catalog/ccsnrt_physbio/fmrc/catalog.xml

@cedwards2 - can you provide some info about where these model results fit into the COMT data explorer (http://testbedwww.sura.org/explorer/#) - we current categorize models by type - estuarine hypoxia, shelf hypoxia, tropical inundation, and extratropical inundation.

we'll also need to know which variables to visualize.

[kknee]

@rluettich just let me know that the category here will be West Coast, so @cedwards2, we just need to know the variables of interest.

[kknee]

@cedwards2 any guidance on the variables of interest for ingesting to testbed? Would a quick call to discuss be easier?