Question: Uncertainty Types

[ayeshans]

Hey @nikitadurasov! 👋

When using the Masksembles method, I want to make a distinction between the different types of uncertainty (model/epistemic, label/aleatoric, and distributional/dataset shift). I noticed it wasn't explicitly stated what kind/s of uncertainty were being estimated through this method, but since the experiments focus on OOD detection, would it be fair to say this is mostly for dataset shift?

[nikitadurasov]

Hey @ayeshans!

Yes, primarily we're benchmarkzing against dataset shift, image corruptions and aleatoric uncertainty. Though we aren't providing any clean way to distinguish different types of uncertainties (eg epistemic and data shift), our model still works rather good with standard methods to approach that: 1) for aleatoric classification uncertainty, we can use entropy of mean distribution 2) for epistemic, we calculate Mutual Information as described in the paper.

Hope that helps! :)

Nikita