search

GEUS-Glaciology-and-Climate / pypromice

Process AWS data from L0 (raw logger) through Lx (end user)
https://pypromice.readthedocs.io
GNU General Public License v2.0
14 stars 4 forks source link

implement writing of position csv file to aws-l3 #78

Closed patrickjwright closed 1 year ago

patrickjwright commented 1 year ago

This uses already built-in building of positions dict to write a csv file to the aws-l3 repo.

The positions are the most recent point on 3 month linear regression trends. There is some inconsistency, in that we only look for and use modem positions when GPS positions are missing for stations that make it through to BUFR submission. There are many scenarios in getBUFR where a station is skipped or rejected for BUFR processing. I have captured all of these situations and we will still write position metadata to the csv file for these stations (but only using GPS, not modem).

EDIT: The above was fixed! See following commits and comments.

This will require to run as getBUFR --positions in processing script.

patrickjwright commented 1 year ago

Well, it was easy to add the combine_first method to fetch_old_positions(). One more commit to address this. Now, we will use both GPS and modem-derived positions for writing to the csv file, regardless of whether the station made it through BUFR processing or was skipped/rejected.

patrickjwright commented 1 year ago

Also note that this csv file is "data driven". It only makes entries for stations that have a file in the aws-l0 tx directory, and if no valid GPS or modem positions can be found in previous 3 months, then the entry will be empty.

The timestamp is the observation time on the linear trend that was used for the position.

csv file head looks like:

stid,timestamp,lat,lon,alt
CEN1,2021-06-12 18:00:00,77.181783,-61.11583,1888.8
CEN2,2022-11-28 16:00:00,77.181974,-61.115571,1892.2
CP1,2023-01-03 12:00:00,69.871914,-47.040055,1946.0
DY2,2023-01-03 12:00:00,66.482366,-46.29364,2120.6
EGP,2023-01-03 00:00:00,75.626863,-35.96938,2665.6
HUM,2023-01-01 12:00:00,78.528898,-56.844461,1968.5
JAR,2023-01-03 12:00:00,69.492115,-49.720693,907.0
JAR_O,2023-01-03 12:00:00,69.494688,-49.67066,923.9
KAN_L,2023-01-03 00:00:00,67.09374,-49.96555,632.9
KAN_M,2023-01-03 00:00:00,67.070402,-48.892871,
KAN_U,2023-01-03 00:00:00,67.000595,-47.032157,1823.0
KPC_L,2023-01-03 00:00:00,79.910797,-24.080782,360.5
KPC_Lv3,2023-01-03 13:00:00,79.913878,-24.093321,373.6