intel-analytics / analytics-zoo

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
https://analytics-zoo.readthedocs.io/
Apache License 2.0
17 stars 3 forks source link

[automl] MC dropout for model uncertainty and add confidence level #989

Closed shane-huang closed 4 years ago

shane-huang commented 5 years ago

Monte Carlo Dropout is a simple way of adding uncertainty for modeling extreme/unexpected events and for anomaly detection.

Easy to implement. Also provides a way for NN's to calculate confidence level and empirical coverage metrics as statistical methods. Refer to paper: https://arxiv.org/abs/1709.01907

shane-huang commented 5 years ago

API: enable Monte Carlo dropout with tsp.fit(mc=True) add a new interface predict_with_uncertainty for pipeline:

y_pred, y_pred_uncertainty = pipeline.predict_with_uncertainty(input_df, n_iter=100)

y_pred is a dataframe with datetime and value column. value column contains the mean value of the prediction results for n_iter times. y_pred_uncertainty is a numpy array with a shape of [num samples, future_seq_len]

shane-huang commented 4 years ago

added into each model and may need refactor