Wrap models not machines?

[ablaom]

@pat-alt

Congratualations on the launch of this new package 🎉 Great to have the integration with MLJ!

I'm not familiar with conformal prediction, but I nevertheless wonder why this package wraps MLJ machines rather than models. If you wrap models, then you will buy into MLJ's model composition. So, a "conformally wrapped model" will behave like any other model: you can insert in pipeline, can wrap in tuning strategy, and so forth.

New models in MLJ generally implement the "basement level" model API. Machines are a higher level abstraction for: (i) user interaction; and (ii) syntax for building learning networks which are ultimately "exported" as standalone model types.

Here are other examples of model wrapping in MLJ: EnsembleModel (docs), BinaryThresholdPredictor, TunedModel, IteratedModel. What makes things a little complicated is the model hierarchy: the model supertype for the wrapped model depends on the supertype of the atomic model. So for example, we don't just have EnsembleModel we have DeterministicEnsembleModel (for ordinary point predictors) and ProbabilisticEnsembleModel (for probabilistic predictors) but the user only sees a single constructor EnsembleModel; see here. (A longer term goal is to drop the hierarchy in favour of pure trait interface, which will simplify things, but that's a little ways off yet.)

Happy to provide further guidance.

cc @azev77

[azev77]

@ablaom currently MLJ models can give deterministic & probabilistic predictions (GLM predicts an entire distribution). If methods for conformal predictive inference become more developed, can users automatically combine their favorite (appropriate) deterministic predictor w/ their favorite predictive interval method, to make probabilistic predictions?

For example: currently model = LGBMRegressor() gives deterministic predictions would a user be able to create a new model composing a deterministic model LGBMRegressor w/ their favorite (appropriate) predictive interval method e.g. conformal(type=naive) to create a new model w/ probabilistic predictions?

Note: at some point it would be great to compare these w/ predictions/Prediction Intervals from NGBoost.py etc

[ablaom]

For example: currently model = LGBMRegressor() gives deterministic predictions would a user be able to create a new model composing a deterministic model LGBMRegressor w/ their favorite (appropriate) predictive interval method e.g. conformal(type=naive) to create a new model w/ probabilistic predictions?

Yes, I guess that's the design I am suggesting. So, similar to the way BinaryThresholdPredictor wraps any probabilistic predictor and makes it deterministic.

[pat-alt]

Hi @ablaom 👋🏽 Thanks very much for this suggestion and sorry for the delayed response (been battling Covid this week while also trying to finish my JuliaCon proceedings submission 😅 ). I will implement this first thing once I turn back to working on this package some time next week 👍🏽

[pat-alt]

Implemented in #10, but will keep this open for now, because I still want to iron out a few things (some related questions here).

[pat-alt]

18 should now be strictly in line with MLJ @ablaom, so I will close this. Will still need to figure out how to handle downstream tasks like evaluate in the future.

[ablaom]

Did you not need new abstract model subtype(s) at MLJModelInterface? For set-predictions (we already have Interval).

[pat-alt]

Yup, you're right. Have done that now in #20