GEUS-Glaciology-and-Climate / pypromice

Process AWS data from L0 (raw logger) through Lx (end user)
https://pypromice.readthedocs.io
GNU General Public License v2.0
12 stars 4 forks source link

Appended transmission file needs to be cropped to match period indicated in toml file #179

Open BaptisteVandecrux opened 10 months ago

BaptisteVandecrux commented 10 months ago

Inherited from https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues/issues/60

If a modem is being moved from one AWS to another, but is not properly (or immediately) reported in the toml file, it can happen that transmissions from one station are appended to another station. When the toml file is eventually updated, the wrong data, appended in the meantime, remains on the wrong station.

Until now, the solution was that someone would open the transmission file AWS-L0/tx/<stid>_<imei>.txt and remove the rows corresponding to erroneous transmission. It would be much more reliable if pypromice could check (and crop if necessary) the AWS-L0/tx/<stid>_<imei>.txt so that the content of that file only correspond to the time period described in the toml file for that given IMEI number.

Just for reporting another related situation:

In 2022 a logger box and associated modem from CP1 were re-installed as a new station called CEN2. Because CEN2 inherited from an IMEI that had already data, and in spite of the dates being set right in the CEN2 toml file, the past transmissions from CP1 ended up in the CEN2 data file (see https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-data-issues/issues/50 for illlustrations). I needed to remove manually the CP1 data from CEN2 transmission file. This could have been done instantaneously by pypromice.