👖 Conformal Tights

Conformal Tights is a Python package that exports:

• a scikit-learn meta-estimator that adds conformal prediction of coherent quantiles and intervals to any scikit-learn regressor
• a Darts forecaster that adds conformally calibrated probabilistic time series forecasting to any scikit-learn regressor

Features

1. 🍬 Sklearn meta-estimator: add conformal prediction of quantiles and intervals to any scikit-learn regressor
2. 🔮 Darts forecaster: add conformally calibrated probabilistic forecasting to any scikit-learn regressor
3. 🌡️ Conformally calibrated: accurate quantiles, and intervals with reliable coverage
4. 🚦 Coherent quantiles: quantiles increase monotonically instead of crossing each other
5. 👖 Tight quantiles: selects the lowest dispersion that provides the desired coverage
6. 🎁 Data efficient: requires only a small number of calibration examples to fit
7. 🐼 Pandas support: optionally predict on DataFrames and receive DataFrame output

Using

Quick links

1. Installing
2. Predicting quantiles
3. Predicting intervals
4. Forecasting time series

Installing

pip install conformal-tights

Predicting quantiles

Conformal Tights exports a meta-estimator called ConformalCoherentQuantileRegressor that you can use to equip any scikit-learn regressor with a predict_quantiles method that predicts conformally calibrated quantiles. Example usage:

from conformal_tights import ConformalCoherentQuantileRegressor
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor

# Fetch dataset and split in train and test
X, y = fetch_openml("ames_housing", version=1, return_X_y=True, as_frame=True, parser="auto")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=42)

# Create a regressor, equip it with conformal prediction, and fit on the train set
my_regressor = XGBRegressor(objective="reg:absoluteerror")
conformal_predictor = ConformalCoherentQuantileRegressor(estimator=my_regressor)
conformal_predictor.fit(X_train, y_train)

# Predict with the underlying regressor
ŷ_test = conformal_predictor.predict(X_test)

# Predict quantiles with the conformal predictor
ŷ_test_quantiles = conformal_predictor.predict_quantiles(
    X_test, quantiles=(0.025, 0.05, 0.1, 0.5, 0.9, 0.95, 0.975)
)

When the input data is a pandas DataFrame, the output is also a pandas DataFrame. For example, printing the head of ŷ_test_quantiles yields:

Let's visualize the predicted quantiles on the test set:

Predicting intervals

In addition to quantile prediction, you can use predict_interval to predict conformally calibrated prediction intervals. Compared to quantiles, these focus on reliable coverage over quantile accuracy. Example usage:

# Predict an interval for each example with the conformal predictor
ŷ_test_interval = conformal_predictor.predict_interval(X_test, coverage=0.95)

# Measure the coverage of the prediction intervals on the test set
coverage = ((ŷ_test_interval.iloc[:, 0] <= y_test) & (y_test <= ŷ_test_interval.iloc[:, 1])).mean()
print(coverage)  # 96.6%

When the input data is a pandas DataFrame, the output is also a pandas DataFrame. For example, printing the head of ŷ_test_interval yields:

Forecasting time series

Conformal Tights also exports a Darts forecaster called DartsForecaster that uses a ConformalCoherentQuantileRegressor to make conformally calibrated probabilistic time series forecasts. To demonstrate its usage, let's begin by loading a time series dataset:

from darts.datasets import ElectricityConsumptionZurichDataset

# Load a forecasting dataset
ts = ElectricityConsumptionZurichDataset().load()
ts = ts.resample("h")

# Split the dataset into covariates X and target y
X = ts.drop_columns(["Value_NE5", "Value_NE7"])
y = ts["Value_NE5"]  # NE5 = Household energy consumption

# Add categorical covariates to X
X = X.add_holidays(country_code="CH")
X = X.add_datetime_attribute("month")
X = X.add_datetime_attribute("dayofweek")
X = X.add_datetime_attribute("hour")
X_categoricals = ["holidays", "month", "dayofweek", "hour"]

Printing the tail of the covariates time series X.pd_dataframe() yields:

We can now equip a scikit-learn regressor with conformal prediction using ConformalCoherentQuantileRegressor as before, and then equip that conformal predictor with probabilistic time series forecasting using DartsForecaster:

from conformal_tights import DartsForecaster, ConformalCoherentQuantileRegressor
from pandas import Timestamp
from xgboost import XGBRegressor

# Split the dataset into train and test
test_cutoff = Timestamp("2022-06-01")
y_train, y_test = y.split_after(test_cutoff)
X_train, X_test = X.split_after(test_cutoff)

# Now let's:
# 1. Create an sklearn regressor of our choosing, in this case `XGBRegressor`
# 2. Add conformal quantile prediction to the regressor with `ConformalCoherentQuantileRegressor`
# 3. Add probabilistic forecasting to the conformal predictor with `DartsForecaster`
my_regressor = XGBRegressor()
conformal_predictor = ConformalCoherentQuantileRegressor(estimator=my_regressor)
forecaster = DartsForecaster(
    model=conformal_predictor,
    lags=5 * 24,  # Add the last 5 days of the target to the prediction features
    lags_future_covariates=[0],  # Add the current timestamp's covariates to the prediction features
    categorical_future_covariates=X_categoricals,  # Convert these covariates to pd.Categorical
)

# Fit the forecaster
forecaster.fit(y_train, future_covariates=X_train)

# Make a probabilistic forecast 5 days into the future by predicting a set of conformally calibrated
# quantiles at each time step and drawing 500 samples from them
quantiles = (0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.975)
forecast = forecaster.predict(
    n=5 * 24, future_covariates=X_test, num_samples=500, quantiles=quantiles
)

Printing the head of the forecast quantiles time series forecast.quantiles_df(quantiles=quantiles) yields:

Let's visualize the forecast and its prediction interval on the test set:

Contributing