search

geoschem / integrated_methane_inversion

Integrated Methane Inversion workflow repository.
https://imi.readthedocs.org
MIT License
28 stars 24 forks source link

IMI BCs not reproducible with current replace_outliers scheme #148

Closed laestrada closed 1 year ago

laestrada commented 1 year ago

Name and Institution (Required)

Name: Lucas Estrada Institution: Harvard ACMG

Description of your issue or question

The current iteration of calculate_bias.py (v2023-06) uses a threshold value that removes bias corrections greater than the threshold to prevent anomalously high bias corrections. However, this threshold is determined by finding the 99th percentile of bias. The percentile changes depending on the input start and end dates, which inhibits reproducibility for the Boundary conditions.

For the next iteration of BCs we should either remove the outlier replacement or update it to use a set amount of time eg. 1-month. Thanks @nicholasbalasus for finding this bug.

nicholasbalasus commented 1 year ago

When we fix this, we should also make the simple change to only do this when generating for April 2018: https://github.com/geoschem/integrated_methane_inversion/blob/5ac2ac6b814e1e78771933ec2da6cefa99662b6b/src/write_BCs/run_BCs.sh#L78-L79

nicholasbalasus commented 1 year ago

Another thing that is relevant when not generating for the full period (April 2018 - present): if we are going to keep our temporal averaging as is, we need to run GC and get TROPOMI data for ~20 days before the start date requested. Right now, if you do:

startdate: 20230401 enddate: 20230530

It will only run GC and get TROPOMI for 2023.04.01-2023.06.20. It should get it for 2023.03.11-2023.06.20 (or so).

nicholasbalasus commented 1 year ago

Closed by #156