search

alan-turing-institute / python-cmethods

Bias Correction/Adjustment Tools written in Python for Climatic Research
GNU General Public License v3.0
0 stars 0 forks source link

Group option does not work on quantile_mapping and quantile_delta_mapping methods #1

Open crangelsmith opened 1 year ago

crangelsmith commented 1 year ago

Describe the bug

We cannot implement debiasing using groups on the following methods: quantile_mapping and quantile_delta_mapping.

To Reproduce

Running from clim-recal, in directory python/debiasing

python run_debiasing.py --obs ../../data/test_data_debias/obs --contr ../../data/test_data_debias/simh  --scen ../../data/test_data_debias/simp --shape ../../data/Scotland/Scotland.bbox.shp -v tasmax --method quantile_delta_mapping --output ../../data/output_test --group time.month

You can also add a grouping option to cell 7 from this notebook.

Error message

File "/Users/crangelsmith/Projects/clim-recal/python/debiasing/run_cmethods.py", line 146, in run_debiasing
    result = cm.adjust_3d(
             ^^^^^^^^^^^^^
  File "/Users/crangelsmith/Projects/clim-recal/python/debiasing/python-cmethods/cmethods/CMethods.py", line 149, in adjust_3d
    result[lat, lon] = method(
                       ^^^^^^^
  File "/Users/crangelsmith/Projects/clim-recal/python/debiasing/python-cmethods/cmethods/CMethods.py", line 634, in quantile_delta_mapping
    if group is not None: return cls.grouped_correction(method='quantile_delta_mapping', obs=obs, simh=simh,
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crangelsmith/Projects/clim-recal/python/debiasing/python-cmethods/cmethods/CMethods.py", line 244, in grouped_correction
    computed_result = func_adjustment(obs=m_obs, simh=m_simh, simp=m_simp, kind=kind, group=None, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crangelsmith/Projects/clim-recal/python/debiasing/python-cmethods/cmethods/CMethods.py", line 638, in quantile_delta_mapping
    res = simp.copy(deep=True)
          ^^^^^^^^^^^^^^^^^^^^
TypeError: list.copy() takes no keyword arguments

Additional context

This work in the other debiasing because the methods are based on computation on numpy arrays, so when it's giving a list of arrays the calculation is applied individually to each list. This is not the case for the distribution-based methods (quantile mapping), where no list is expected, but a single time series.

A bug fix should allow running the calculation on each component of the list that is returned when the grouping happens.

btschwertfeger commented 1 year ago

The grouping for quantile methods has been removed in https://github.com/btschwertfeger/python-cmethods/commit/e3436310a4f927c86b30d7db137dd718c69fe1f7 since grouping distribution-based methods produces unrealistic high deviations in monthly (if group="time.month") transitions on large data sets. This also occurs when grouping to time.dayofyear, and time.year.

It was removed since the grouping was some kind of leftover of my bachelor theses in which I started implementing this tool and it is not described in any paper (for the mentioned reasons I think).

crangelsmith commented 1 year ago

The grouping for quantile methods has been removed in btschwertfeger@e343631 since grouping distribution-based methods produces unrealistic high deviations in monthly (if group="time.month") transitions on large data sets. This also occurs when grouping to time.dayofyear, and time.year.

It was removed since the grouping was some kind of leftover of my bachelor theses in which I started implementing this tool and it is not described in any paper (for the mentioned reasons I think).

Thank you @btschwertfeger for the explanation!

@RuthBowyer FYI.

gmingas commented 11 months ago

Revisiting this. I think we should modify the code to gracefully fail early when grouping for quantile methods is requested. And also add information about this in the relevant docstrings.