search

ioos / marine_life_data_network

Planning efforts relevant to IOOS Marine Life and IOOS DMAC.
Other
0 stars 2 forks source link

New Task: Provide an open access table of GTS metrics #68

Open MathewBiddle opened 3 months ago

MathewBiddle commented 3 months ago

Who is requesting this?

@ioos/marine-life

What is being requested?

Provide an open access table of all platforms with data submitted to the GTS including WMO id, deployment id, ptt id, platform code and deployment start date.

What is the requested deadline and why?

2025-09-30

What is the current status quo (i.e., what happens if this does not get done)?

The IOOS Office will not have readily available metrics about how many messages the ATN is sending to the GTS. These metrics are used to convey the importance of the system and identify impacts.

What indicates this is done (i.e., how do we know this is complete)?

Provide a description or any other important information.

We have some existing harvesting GTS statistics for ATN by scraping https://stage-ndbc-bufr.srv.axds.co/platforms/atn/smru/profiles/. This assumes that everything posted there makes it to the GTS. We would like to build in some monitoring capability to ensure what we think is going, is actually making it. Maybe we can harvest information from the OSMC ERDDAP https://osmc.noaa.gov/erddap/tabledap/OSMC_flattened.html

Below is a script for collecting the assumed metrics:

Could we check against the OSMC ERDDAP for platform = “TAGGED ANIMAL”:

https://osmc.noaa.gov/erddap/tabledap/OSMC_flattened.htmlTable?platform_id%2Cplatform_code%2Cplatform_type%2Ccountry%2Ctime%2Clatitude%2Clongitude%2Cobservation_depth%2Csst%2Catmp%2Cprecip%2Csss%2Cztmp%2Czsal%2Cslp%2Cwindspd%2Cwinddir%2Cwvht%2Cwaterlevel%2Cclouds%2Cdewpoint%2Cuo%2Cvo%2Cwo%2Crainfall_rate%2Chur%2Csea_water_elec_conductivity%2Csea_water_pressure%2Crlds%2Crsds%2Cwaterlevel_met_res%2Cwaterlevel_wrt_lcd%2Cwater_col_ht%2Cwind_to_direction%2Clon360&platform_type="TAGGED ANIMAL"

mmckinzie commented 1 month ago

The original task as listed above, i.e., provide open access table of all platforms with data submitted to the GTS has been completed. Table is available here: https://osmc.noaa.gov/erddap/tabledap/anibos_metadata.html. ATN platforms can be quired by the program field, select 'AniBOS ATN' or the country field, select 'United States'. These metadata are also being integrated into the OceanOPS dashboard. I update these metadata after new CTD deployments are registered in the ADR and BUFR flags embedded to trigger automated pathway the generates and serves BUFR messages to the NDBC stagging platform.

MathewBiddle commented 1 month ago

Great stuff. I can work with integrating that ERDDAP query into the ioos_metrics package. See https://github.com/ioos/ioos_metrics/issues/58

MathewBiddle commented 1 month ago

It would be good to gather some details on how data are getting to that ERDDAP. Is there any latency? Is it automated? What are the requirements on ATN's side to release data to the OSMC ERDDAP?

When looking at the ERDDAP response I see a lot less data than we are posting on https://stage-ndbc-bufr.srv.axds.co/platforms/atn/smru/profiles/. Any ideas about why there is a difference?

mmckinzie commented 1 month ago

My understanding is that OMSC ERRDAP harvests data directly from the GTS. No requirement on ATN to release data to OMSC. But I can confirm this with Kevin O'Brien.

I'm not sure why only the 2024 metadata is appearing in the tableDAP service there definitely should be more deployments listed. I need to send Kevin some updated metadata from recent summer deployments, so I'll check on that as well.

mmckinzie commented 2 weeks ago

AniBOS platform metadata has been updated to include all AniBOS ATN and AniBOS IMOS platforms with data submitted to the GTS from Dec 2019 through present.

Kevin O' Brien confirmed that OMSC harvests animal borne profile data submitted to the GTS as BUFR or TESAC messages automatically in near real-time, including ATN submissions and these data are available via OMSC ERDAPP.