salesforce / Merlion

Merlion: A Machine Learning Framework for Time Series Intelligence
BSD 3-Clause "New" or "Revised" License
3.43k stars 302 forks source link

[FEATURE REQUEST] somewhat unintuitive default forecasting model #94

Open MBasalla opened 2 years ago

MBasalla commented 2 years ago

Is your feature request related to a problem? Please describe. It is somewhat unintuitive, that the model returns a standard error in the univariate case and but not in the multivariate case (due to using a tree based forecast in the multivariate case).

Describe the solution you'd like Ideally the default forecaster should return the same output "format" (forecast and standard error) independent of the input data. This would likely require changing the default forecaster for the multivariate case (A probabilistic tree based method like for example ngboost [https://github.com/stanfordmlgroup/ngboost] might be worth investigating for this.)

Describe alternatives you've considered Given that the "ideal" solution is quite complex a lower effort alternative would be to make this behavior more clear in the description and documentation of the default forecast model.

Additional context I think having a default model as an entry point to the forecasting case is a good idea. But the difference in forecast outputs for univariate and multivariate case somewhat defeats this purpose. Intuitively one would probably assume, that the default model has the same features in both cases.

aadyotb commented 2 years ago

@MBasalla thanks for the suggestion. @Emerald01 can you look into enhancing the tree models with uncertainty estimates?

Emerald01 commented 2 years ago

@MBasalla thank you for the suggestion. We used deterministic tree models for the multivariate forecaster, and we are considering to add some probabilistic-based tree models and others for the multivariate so we can have variance measure. It is worth mentioning that the standard error is not the default output for prediction in general because only probabilistic-based algorithms will naturally provide such measure. For deterministic forecaster, we return None for standard deviation.