pankajkarman / bias_correction

python library for bias correction
MIT License
30 stars 8 forks source link

Application for scenarios #8

Closed phiscu closed 2 years ago

phiscu commented 2 years ago

Dear Pankaj, thank you for your code. It is very convenient! But I'm seeing a strange effect when I'm using datasets that exceed the range of the reference data. I have three temperature datasets as CSV: obs of a weather station, ERA5 data of the closest grid point, and CMIP6 data of the closest gridpoint. SDM with obs and ERA5 works great but using the same function on the corrected ERA5 and CMIP6 results in a time series that is much colder than both training datasets. Using uncorrected ERA5 has similar results although the gap is smaller. The function preserves the relative trend but it appears the resulting values are far too low every time the data to be corrected includes values exceeding the range of the reference data. But isn't climate projection one of the target applications? Or might the problem be my application of the package? plot_bias_correction_exmpl

Thank you for your help and best regards, Phillip

pankajkarman commented 2 years ago

@phiscu from your attached plot, it's apparent the methodology you have applied has just shifted the background trend level. It has not made any modification to the data distribution. I guess there is some issue with the application of the module here. Quantile mapping has been found to inflate or deflate the raw model projected climate change.

Reference: https://hess.copernicus.org/articles/21/2649/2017/

phiscu commented 2 years ago

Dear Pankaj, thx for your instant reply, I appreciate it. Please note that these are annual means of daily data. Looking at the corrected and target data sets the distributions are indeed very similar but not the same. It is very hard for me to find any more error sources as the code is actually very simple (thx to you) and I checked every detail over and over again. I prepared a reproducible example that should work out of the box. Would you check what I'm doing wrong? I'm stuck here for quite a while now... Thanks a lot! example_bias_correction.zip

pankajkarman commented 2 years ago

Hi Philip, I don't see any problem in the results generated by your code (plot attached herewith).

Figure_1

If you think the problem is values exceeding the range of the reference data, you might want to experiment with cdf_threshold for normal_correction which has been discussed by Switanek et al.

pankajkarman commented 2 years ago

One more thing, normal_mapping is the correct keyword for normal SDM not normal_correction. Using normal_correction will trigger modified quantile correction instead of throwing error.

phiscu commented 2 years ago

Hi Pankaj, thanks a lot, your last answer solved the case. On your manual page you show an example using method ='gamma_mapping` but also introduce the functionnormal_correction()`. I didn't see the little but important difference in the keywords and didn't actually use SDM after all. Thanks again! Great package!