search

GEUS-Glaciology-and-Climate / pypromice

Process AWS data from L0 (raw logger) through Lx (end user)
https://pypromice.readthedocs.io
GNU General Public License v2.0
12 stars 4 forks source link

Update positions #107

Closed patrickjwright closed 1 year ago

patrickjwright commented 1 year ago

This PR addresses https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues/issues/31

The AWS_station_locations.csv file will now be a complete representation of all stations with data in aws-l3/level_3, including the positions and timestamps associated with the last transmission (even discontinued stations).

We have also addressed errors in position for ZAK_L by correcting the toml file.

patrickjwright commented 1 year ago

This has been tested with manual running of getBUFR --positions --dev. AWS_station_locations.csv is updating as expected.

I will also mention: the other way to do this, rather than creating the positions_seed dict in wmo_config.py, would be to look for any discontinued stations present in aws-l3/level_3 that are not present in aws-l3/tx. Then read those files, do the smoothing, and get the most recent positions and timestamps. However, this is just extra and unnecessary hourly computation for stations that no longer exist and the config object seems like a simpler solution. Downside is that it will have to be manually updated (like other entires in this config file), but it is all in one place and should need infrequent updating.

PennyHow commented 1 year ago

I will also mention: the other way to do this, rather than creating the positions_seed dict in wmo_config.py, would be to look for any discontinued stations present in aws-l3/level_3 that are not present in aws-l3/tx. Then read those files, do the smoothing, and get the most recent positions and timestamps. However, this is just extra and unnecessary hourly computation for stations that no longer exist and the config object seems like a simpler solution. Downside is that it will have to be manually updated (like other entires in this config file), but it is all in one place and should need infrequent updating.

The logic with checking aws-l3/tx only works if we update and remove old tx messages when raw data comes in for a station (and therefore we no longer need to store all tx messages for the period we have overlap with the raw data). We currently don't have routines in place for this, and have not yet had to do this. Like you say though, it should only need to be done infrequently at least if we have to do it manually.