search

ioos / ioos_metrics

Working on creating metrics for the IOOS by the numbers
https://ioos.github.io/ioos_metrics/
MIT License
2 stars 4 forks source link

Add ATN GTS statistics to NDBC station GTS statistics #6

Closed MathewBiddle closed 1 year ago

MathewBiddle commented 2 years ago

Now that ATN is pushing data to GTS, would be nice to know how much. Not sure where we can collect this information...

MathewBiddle commented 1 year ago

Can get info from https://stage-ndbc-bufr.srv.axds.co/platforms/atn/smru/profiles/

currently each directory is a deployment (16 as of 7/19, last updated May 2022).

MathewBiddle commented 1 year ago

IOOS_GTS_ATN numbers_21Jul22.xlsx

1 profile = 1 message

MathewBiddle commented 1 year ago

@mmckinzie are only smru profiles going to GTS?

My computation for "Number of records ATN pushed to the GTS" will be computed by recursively searching https://stage-ndbc-bufr.srv.axds.co/platforms/atn/smru/profiles/ for files ending in .bufr and counting them up.

I can probably get more fancy with the calculation if we want to parse the .csv files too.

MathewBiddle commented 1 year ago

script written and metrics on monthly profiles going to GTS are in https://github.com/MathewBiddle/ioos_by_the_numbers/commit/f4fdc8b7b94cac9fa0d5b775b27fb4c4d74fc84c

metrics can be found at https://github.com/MathewBiddle/ioos_by_the_numbers/blob/main/gts/GTS_ATN_monthly_totals.csv

mmckinzie commented 1 year ago

@MathewBiddle thanks for working on this. Numbers look great. Using the .bufr files is appropriate.

Only thought would be to distinguish bw temp only from temp and salinity profiles and tally them separately. But no easy way to distinguish which deployments are temp and which are temp/salinity from information available in the staging platform. But I could provide you with those details or possibly see if axiom can add in an instrument field to directory to distinguish. i.e., CTD-SRDL and CTD/F-SRDL have temp and salinity profiles, GPS-SRDL are temp only.

MathewBiddle commented 1 year ago

I should add that I am using the dates listed on the file index to serve as the "Date when messages were sent to the GTS"

For example, https://stage-ndbc-bufr.srv.axds.co/platforms/atn/smru/profiles/ct169-594-21/

If this is incorrect, we should reopen this issue and hash out what a reasonable approach is.

In response to:

Only thought would be to distinguish bw temp only from temp and salinity profiles and tally them separately.

I did find a python package https://pdbufr.readthedocs.io/en/latest/index.html that allows me to crack open the bufr messages and parse the data. I started testing in https://github.com/MathewBiddle/ioos_by_the_numbers/blob/main/read_bufr.py. If you can provide a criteria for distinguishing the differences, I'm sure we can compute those values. For example, crack open the bufr message and check if it contains salinity data...

mmckinzie commented 1 year ago

@MathewBiddle Nice. I still think the total number of profiles metric is useful and yes, it is OK to use date listed in file index. But thanks for also looking into parsing temp only from temp/salinity. And yes, one way to go about it would be to crack open the messages and check if they contain salinity data.

We are also hoping to set-up a secure tableDAP server which all nations AniBOS programs including the ATN can provide limited but relevant metadata about platforms with data submitted to the GTS in either BUFR or TESAC format. In the future, that or something similar could be used to reference which platforms have temp only, which have temp/salinity and sooner than later temp/salinity/fluorescence (for now CTD/F tags only submit the T/S data to the GTS regardless if sent as BUFR or TESAC messages)

I am also working with OceanOPS to officially get these platforms listed under GOOS AniBOS and the AniBOS ATN program. OceanOPS will also harvest the BUFR and TESAC messages using provided platform WMO id, header and node to provide access and generate additional statistics such as time delay bw profile generation and arrival at GTS, they will also likely tally number of submitted messages. The 2020-2023 AniBOS details and stats will be included in the next OceanOPS report card.