search

GEUS-Glaciology-and-Climate / pypromice

Process AWS data from L0 (raw logger) through Lx (end user)
https://pypromice.readthedocs.io
GNU General Public License v2.0
14 stars 4 forks source link

Mixing temporal resolutions when merging files with combine_first #297

Open BaptisteVandecrux opened 2 weeks ago

BaptisteVandecrux commented 2 weeks ago

If an hourly STM table covering 2022-2024 is retrieved from the field and is added after a raw 10 min file covering 2022-2023, then the merging of L1 datasets into the L1A dataset will have a strange behavior: https://github.com/GEUS-Glaciology-and-Climate/pypromice/blob/3813c77eec8220ac174dae52699b62e00221fa04/src/pypromice/process/aws.py#L138

The latest hourly data ds_latest_hourly = AWS.L1[-1] will be merged with the previous raw dataset ds_previous_10min = AWS.L1[-2] with : ds_merged = ds_latest_hourly.combine_first(ds_previous_10min )

In that statement, the round-hours lines from the hourly file will be unchanged, but 10 min data will be inserted between these values.

BaptisteVandecrux commented 2 weeks ago

Due to this this, winter daily tx have been injected at KAN_U's hourly data: billede

Again, it all comes from the fact that combine_first only looks at the order at which the files come and is not setting different priorities to the different data types (10 min > STM > tx).