Chronos: dependencies should be cleaned and refactored

[TheaperDeng]

Updated 7/13/2022 according to comments

Please use this demo page to understand what prompt we will give our users. https://theaperdeng.github.io/complex-installation-document-panel/

Motivation

bigdl-chronos's dependencies is becomming more and more unmanagable under current limited installation options(default and all), especially when we start to support 2 frameworks(torch and tf).

Install Options

• bigdl-chronos - tsdataset, simple sklearn based model(dbscan, threshold)

• bigdl-chronos[pytorch] pytorch based models(lstm, s2s, tcn, autoformer, nbeats, autoformer, autoencoder, doppelganger).

• bigdl-chronos[tensorflow] - tf2 based models(mtnet, autoencoder, lstm, s2s, tcn)

• bigdl-chronos[automl] - lightweight HPO

• bigdl-chronos[distributed] - automodel, autots, xshardstsdataset, distributed forecasters(lstm, s2s, tcn, tcmf, nbeats)(must with another framework option (pytorch/tensorflow))

• bigdl-chronos[all] - all(prophet, arima, light-weight hpo, onnxrt, openvino, quantization)

Which option should I Install

dl backend/ml models	automl	distributed	install option	comment
pytorch	no	no	bigdl-chronos[pytorch]
pytorch with inference optimizations	no	no	bigdl-chronos[pytorch] + onnx, onnxruntime	prompt onnxruntime and onnx install cmd
pytorch	yes	no	bigdl-chronos[pytorch, automl]
pytorch	no	yes	bigdl-chronos[pytorch, distributed]
pytorch	yes	yes	bigdl-chronos[pytorch, distributed, automl]
tensorflow	no	no	bigdl-chronos[tensorflow]
tensorflow	yes	no	bigdl-chronos[tensorflow,automl]
tensorflow	no	yes	bigdl-chronos[tensorflow,distributed]
tensorflow	yes	yes	bigdl-chronos[tensorflow, distributed, automl]
prophet, arima	yes/no	no	bigdl-chronos[automl] + prophet/pmdarima	prompt prophet/pmdarima install cmd
dbscan	no	no	bigdl-chronos
I want all models work fine!	yes	yes	bigdl-chronos[all]

Dependencies details in each option

install option	dependencies
[]	pandas, scikit-learn
[pytorch]	+bigdl-nano[pytorch]
[automl]	+optuna, configspace
[tensorflow]	+bigdl-nano[tensorflow]
[distributed]	+bigdl-orca[automl]
[all]	+pmdarima, prophet, tsfresh, pyarrow, light-weight hpo dependencies, onnx, onnxruntime, openvino, neural-compressor, optuna, configspace, bigdl-nano[tensorflow], bigdl-orca[automl]

[TheaperDeng]

@shane-huang @liangs6212 please have a look and provide some feedback if possible.

[shane-huang]

I think we should make our default install a minimum viable option for the most common or our most recommended usage. In this sense,

• we should not include ml models by default as they are not the most recommended models - maybe use another option, e.g. chronos[ml])
• we should figure out some typical pipelines that we want to recommend to user as baseline, env validation, or first options to try. Default option should be able to run them smoothly and successfully. For example, we can target for supporting LSTM/TCN/Seq2Seq pytorch training and inference on built-in dataset on single node in default install. Then let's figure out which dependencies it needs.

[shane-huang]

maybe separate the lightweight-hpo with other all options, since it only requires optuna and configspace.

• use lw automl on single node: chornos[automl]- for both pytorch and tensorflow
• use distributred automl: chronos[automl, distributed]

[shane-huang]

Another related topic we may consider. Shall we separate chornos further into chronos-forecaster, chronos-detector, and chronos-simulator, chronos-data?

[TheaperDeng]

Another related topic we may consider. Shall we separate chornos further into chronos-forecaster, chronos-detector, and chronos-simulator, chronos-data?

This might make Chronos too complex, and since detector and simulator are not that large and mature, we may just stick to one bigdl-chronos for now.