mc_dropout with predict_likelihood_parameters

[SaltedfishLZX]

Hi, thanks for the prompt response regarding issue #2097 . I'm now able to utilize mc_dropout in historical_forecasts.

My current objective is to incorporate mc_dropout for obtaining epistemic uncertainty and predict_likelihood_parameters for acquiring aleatoric uncertainty. However, when setting predict_likelihood_parameters to True, I encounter an issue where I'm unable to use the sample method. Consequently, I'm limited to obtaining only one result with each call to historical_forecasts with predict_likelihood_parameters=True and mc_dropout=True.

I suspect that the solution might involve distinguishing the sample generation process by using mc_dropout and sample generation process based on the fitted distribution. Now they are using the same num_samples parameter.

[dennisbader]

Hi @SaltedfishLZX , mc dropout and predict_likelihood_parameters shouldn't be used together.

• if you use a likelihood to train your model, then the model learns to predict the likelihood parameters based on some input. Monte carlo dropout will not be used during training. predict_likelihood_parameters will then return the predicted distribution/parameters. These are deterministic. You can also set predict_likelihood_parameters=False and num_sample >> 1 to generate probabilstic samples. Then you can calculate the distribution parameters from the sampled prediction (e.g. with quantiles, mean, std, etc.).
• For the Monte Carlo dropout, your model does not have to be trained using a likelihood. It can be a deterministic model that was trained using a loss_fn. When predicting with mc_dropout, we use monte carlo sampling to generate probabilistic samples. If you train the model using a likelihood instead, you should set predict_likelihood_parameters=False, mc_dropout=True and num_samples >> 1 to generate the samples. Later you could caluclate the distribution parameters from the sampled prediction (e.g. with quantiles, mean, std, etc.)

Hope this helps

[SaltedfishLZX]

Hi @SaltedfishLZX , mc dropout and predict_likelihood_parameters shouldn't be used together.

• if you use a likelihood to train your model, then the model learns to predict the likelihood parameters based on some input. Monte carlo dropout will not be used during training. predict_likelihood_parameters will then return the predicted distribution/parameters. These are deterministic. You can also set predict_likelihood_parameters=False and num_sample >> 1 to generate probabilstic samples. Then you can calculate the distribution parameters from the sampled prediction (e.g. with quantiles, mean, std, etc.).
• For the Monte Carlo dropout, your model does not have to be trained using a likelihood. It can be a deterministic model that was trained using a loss_fn. When predicting with mc_dropout, we use monte carlo sampling to generate probabilistic samples. If you train the model using a likelihood instead, you should set predict_likelihood_parameters=False, mc_dropout=True and num_samples >> 1 to generate the samples. Later you could caluclate the distribution parameters from the sampled prediction (e.g. with quantiles, mean, std, etc.)

Hope this helps

Thanks for your response @dennisbader. My idea is based on the User Guide: Capturing model uncertainty using Monte Carlo Dropout. It mentions that "Monte Carlo Dropout can be combined with other likelihood estimations in Darts, which can be interpreted as a way to capture both epistemic and aleatoric uncertainty."

Therefore, I am considering the possibility of using MC dropout with likelihood. Given that MC dropout with a deterministic model effectively captures model uncertainty, incorporating MC dropout with a likelihood model appears to be a more comprehensive approach for capturing both epistemic and aleatoric uncertainty. It just likes that we utilize MC dropout to obtain the distribution of a distribution (mean, std), and the distribution of the mean may correspond to the epistemic uncertainty.

Could you confirm if my understanding is correct? I would appreciate your guidance on implementing the approach outlined in the User Guide.

[SaltedfishLZX]

• if you use a likelihood to train your model, then the model learns to predict the likelihood parameters based on some input. Monte carlo dropout will not be used during training.

As for training likelihood, I came across an example of TFT setting dropout=0.1 and likelihood=QuantileRegression. It appears that the use of MC dropout for training does not conflict with learning the likelihood, since it is just a way of regularization during training.

In any case, I believe we need to clarify the usage scenarios for MC dropout and the sampling method.