search

radix-ai / conformal-tights

👖 Conformal Tights adds conformal prediction of coherent quantiles and intervals to any scikit-learn regressor or Darts forecaster
MIT License
79 stars 3 forks source link

Quantiles not monotonically increasing on test set #23

Open ejhlego opened 3 months ago

ejhlego commented 3 months ago

Hi,

I was keen to use this package since when I've tried to do conformal prediction using quantile regression in the past I've encountered the common issue of quantile forecast results not increasing monotonically.

I see this package aimed to solve that but I am still seeing some NOT monotonic results in my quantiles! It seemed to work before this change feat: support pre-fitted estimators (https://github.com/radix-ai/conformal-tights/pull/19).

I think it could be because I'm passing a fitted model lgbm model into ConformalCoherentQuantileRegressor and so the part that uses XGBoost to model the quantiles doesn't happen possibly.

lsorber commented 3 months ago

Hi @ejhlego, apologies for the delay in responding to this question!

Your analysis of the problem is correct: the fact that you supply a fitted model in combination with PR #19 could break monotonicity.

However, PR #19 is only available on our main branch, it's not yet released to PyPI. This is intentional, as that PR is part of a new version that is still in development.

That means that the solution to your problem is to not install Conformal Tights from GitHub directly, but to use the latest version on PyPI with pip install --upgrade conformal-tights.

Could you check if that solves the issue? If so I'll go ahead and close this one. Thank you!

lsorber commented 2 months ago

Hi @ejhlego, we just released v0.4.0 which brings a number of improvements. Specifically, https://github.com/radix-ai/conformal-tights/pull/28 may help address the issue you reported. If that solves it for you, please let me know so that I can close this issue, thank you!