search

epiforecasts / stackr

Model stacking for predictive ensembles
http://epiforecasts.io/stackr/
MIT License
4 stars 3 forks source link

Future development of the package #10

Open sbfnk opened 1 year ago

sbfnk commented 1 year ago

It would be good think about the future of this package, whether it is worth maintaing and perhaps even developing further and if yes what additional functionality we might want.

My understanding is that there is no other package that does what this one does (optimising mixtures of predictive distributions for CRPS and creating a mixture forecast), and that this is a valid approach that is worth doing further research on / testing its predictive ability. As a minimum, I therefore think that it is worth fixing the failing checks and spending a bit of time on bringing the package back to life with some tests etc. and give it a unique name (#4).

Additionally we could support convolutions via optimising the WIS as currently implemented in https://github.com/epiforecasts/qra and promised there.

If we wanted to go further it should be possible to support optimisation for any score available in https://github.com/epiforecasts/scoringutils and create mixtures and convolutions from that. This would require more development work to support different represenations of forecasts and require some method for optimisation under constraints (e.g. sum of weights = 1) which is difficult to to reliably in the general case. One approach would be to re-implement the scores in stan as is done with CRPS optimisation at the moment but it's not clear to me if there is value in that.

So as a possible destination I think a package that tightly integrates with scoringutils to do general optimisation whilst drawing on better methods where available (e.g. the stan model here for sample-based CRPS optimisation and https://github.com/ryantibs/quantgen for quantile-based WIS optimisation) would be of value if there is no other package that can do the job.

sbfnk commented 1 year ago

Pinging past contributors @nikosbosse @seabbs @yao-yl in case you'd like to contribute thoughts/comments.

seabbs commented 1 year ago

My understanding is that there is no other package that does what this one does (optimising mixtures of predictive distributions for CRPS and creating a mixture forecast), and that this is a valid approach that is worth doing further research on / testing its predictive ability.

This is also my understanding and I agree. I agree it is worth bringing it back to life/functionality.

Additionally we could support convolutions via optimising the WIS as currently implemented in

Agree this is a good idea.

Before integrating in other scores etc. I think really making the core functionality good and doing some evaluation should be the focus. Once that has happened then yes thinking about other scores could make sense.

I like the idea of adding a tight integration with scoringutils.

I'm happy to put some time in to helping with some of the above. I am particularly keen to explore the does it work aspect (of course) but also happy to do some dev.

jonathonmellor commented 5 months ago

Was wondering what the plans are with respect to this package? We are currently (well, last week of the winter season) doing an unweighted mixture approach in our ensemble, would be keen to explore scoring based mixing and making some decisions whether to try out existing packages (e.g. here) or make our own version (probably not stan dependent)

nikosbosse commented 5 months ago

At the moment the package is mostly dormant. @sbfnk had some ideas and thoughts at some point.

sbfnk commented 5 months ago

The plan would still be to eventually implement the suggestions above (with a focus on optimisation towards any given score, not depending on stan) but there's currently no capacity to address this. For what it's worth I think the approach implemented in this package (CRPS based mixture) works to the level of passing tests/examples, as does the WIS-based convolution in https://github.com/epiforecasts/qra

Both approaches could really do with some testing and robustness checking so if you're interested in giving them a go we'd be happy to support where we can. In the longer term and whenever there is capacity (contributions very welcome) it would be good to combine the two into a more flexible package.

sbfnk commented 5 months ago

@jonathonmellor for exploring existing packages possibly also worth looking at https://github.com/epiforecasts/stackr/issues/17 (again, welcoming additions if you find anything interesting/suitable)