[ENH] possible improvements for `FallbackForecaster`

[fkiraly]

Summarizes some optional features discussed for FallbackForecaster (see first version in https://github.com/sktime/sktime/pull/5779) for future work:

• currently, does not support probabilistic forecasts. This could be suppoted by inheriting from _DelegatedForecaster, similar to MultiplexForecaster.
• being able to specify a number of repeats may be helpful, in case there are stochastic or sporadic failures, e.g., for reasons of random initialization

FYI @ninedigits

[ninedigits]

Would the purpose of this be to create prediction intervals?

[fkiraly]

yes, of point 1. imagine you have forecasters that the user wants to use for prediction intervals, curretly the fallback does not work for that functionality.

[ninedigits]

That makes sense, but what happens if a user has a mix of forecasters capable of a prediction interval and those without, eg if they have an EnsembleForecaster somewhere in the list?

[fkiraly]

but what happens if a user has a mix of forecasters capable of a prediction interval and those without, eg if they have an EnsembleForecaster somewhere in the list?

Interesting question! I see two "obvious" options:

Option 1: the capability tag capability:pred_int for the composite is conjunction, i.e., if one of the fallback forecasters can't make proba predictions, the entire composite can't. No further logic is required, as the "mixed" situation does not occur.

Option 2: the capability tag capability:pred_int for the composite is disjunction, i.e., if at least one of the fallback forecasters can't make proba predictions, the entire composite can. Internally, you make a distinction based on the capability tag capability:pred_int, and if a proba prediction is requested, you query only the proba forecasters. This would require keeping track of "first nonfailing proba index" separately from "first nonfailing index", and possibly even fitting two estimators in fit.

One more involved option 3:

We go with option 1, but introduce another composite that takes two forecasters - one proba and one possibly not, and mixes them in the way that proba methods delegate to one, non-proba prediction methods to the other.

Personally, my preference would be option 1 due to its simplicity.

[ninedigits]

Makes sense to me. Option 2 or 3 would be nice to have, but they also introduce a bit more complexity and raise some potential questions. Eg, how should the system behave if the first non-failing forecaster without prediction intervals projects a value outside the prediction intervals of the first non-failing forecaster equipped with this feature?

I have the code ready for option 1, might need your help in ensuring it fully integrates with the system.

[yarnabrina]

I want to clarify the expected behaviour of probabilistic prediction of falback forecaster. I am adhering to option 1 @fkiraly described above, i.e. it'll have the capability only if all soecified forecasters have the capability.

At this stage, will fallback forecaster try to predict intervals (or quantiles) similar to how it does prediction, i.e. it'll try with current selected one, if fail, go to next, and so on? Or, will it attempt only for current forecaster and won't proceed if that fails?

I think #5847 does the latter, please correct me if I am wrong. In that case, do we need to add the internal methods in this class, or simply updating the tag during __init__ is sufficient?

And if we do the former, I think there's a chance that mean predictions and probabilistic predictions are coming from different forecasters. Apart from possibly being a confusion to users, can it not lead to non-monotonicity in some situations?

(I understand mean does not have a monotonicity property in relation of quantiles, but asking for simplicity of most common situations.)

---

Also, for forecasters that do not have probabilistic capabilities, can we somehow use conformal intervals for them? I am not familiar with its theory and haven't used them in sktime myself, but I suppose that can be used? May be as a wrapper/decorator?

[fkiraly]

In that case, do we need to add the internal methods in this class

I agree with you, @yarnabrina, some logic is still missing, but I was assuming that that is still work in progress.

Apart from possibly being a confusion to users, can it not lead to non-monotonicity in some situations?

Interesting point. Not sure how to deal with that except by clear documentation - do you have an idea?

[fkiraly]

Also, for forecasters that do not have probabilistic capabilities, can we somehow use conformal intervals for them? I am not familiar with its theory and haven't used them in sktime myself, but I suppose that can be used? May be as a wrapper/decorator?

That exists, the wrapper is called ConformalIntervals. It has some efficency issues, if you want to look into it 😄