Uncertainty quantification: methods, use cases, challenges

[elcorto]

Title

Uncertainty quantification: methods, use cases, challenges

Description

Uncertainty quantification (UQ) methods enable us to equip ML model predictions with "error bars". These are useful in many areas from active learning to out-of-distribution detection and in general to tackle trustworthiness issues.

Whether you are an expert or have never heard of UQ, let's talk. Possible topics:

• Methods: Bayesian basics (e.g. Gaussian Processes), Bayesian NNs, approximate methods (ensembles, Dropout, Laplace, ...)
• Use cases: Do you use UQ? Great, what for? You don't (yet)? Even better!
• Technical: implementation, packages
• Challenges: Technical ("It's too hard, PyTorch doesn't have it so I don't care."), adoption in the community as "standard" tool, a.k.a. communicate / show usefulness.
• Why doesn't ChatGPT say "I'm not sure, I won't answer" rather than inventing stuff?

Organizational

Organizer(s)

• Steve Schmerler s.schmerler@hzdr.de github: @elcorto
• Peter Steinbach p.steinbach@hzdr.de github: @psteinb

Speakers

See above.

Format

Intro talk (if desired), followed by an open discussion.

Timeframe

1-2h.

Number of participants

1-99

[HeidiSeibold]

Super interesting topic! 👏

Maybe we could also discuss how we could bring statistical thinking into ML to improve UQ?

Also interesting in this context:

• https://mindfulmodeler.substack.com/p/understanding-different-uncertainty
• https://royalsocietypublishing.org/doi/10.1098/rsos.201925

[elcorto]

Sure, since it'll be an open discussion, anything goes :) Thanks for the link!

[elcorto]

If possible, a way to show introduction slides would be nice.

[SusanneWenzel]

noted

[SusanneWenzel]

@elcorto if possible, please make a note here on the (rough) number of participants. Also don't forget to make a note here about the outcome of the session and, if applicable, future plans that came out of this session.

[elcorto]

Number of participants: 30

Useful links

• UQ formulated in terms of information theory:
  • https://arxiv.org/abs/2102.11582
  • https://arxiv.org/abs/1803.08533
• The(:tm:) deep ensemble paper
• https://github.com/uncertainty-toolbox/
• talk on GPs, Kernel methods and UQ
• review of UQ for NNs
• Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
• conformal prediction for sklearn-compatible models
• Sources of Uncertainty in Machine Learning -- A Statisticians' View

Specific use cases

• Stochastic Segmentation network

[t-schanz]

After a few of us left, we had kind of a split-group going on, talking about Bayesian networks and the different kind of uncertainties. We decided on all commenting this issue here to find back together and maybe continue our discussion. So if those people read this (or anyone else who is interested in discussing especially the actual "quantification" part) give this a thumbs up and if we are a few people we can try to organize something.

[erathorn]

Thank you for the list of papers and for organizing this session of the unconference!

[elcorto]

After a few of us left, we had kind of a split-group going on, talking about Bayesian networks and the different kind of uncertainties. We decided on all commenting this issue here to find back together and maybe continue our discussion. So if those people read this (or anyone else who is interested in discussing especially the actual "quantification" part) give this a thumbs up and if we are a few people we can try to organize something.

Good idea. In order to stay connected, we could as a first step create a channel in a chat, e.g. https://mattermost.hzdr.de where all of you should be able to log in with Helmholtz AAI. That would move things out of this issue and serve as a central contact point. There we can discuss a follow-up meeting (in person or virtual) and any other activity, share resources, etc. If there is interest, give this comment a :+1: and I'll set something up.

[elcorto]

Here is the invite link for the mattermost team. Please log in using Helmholtz AAI and your institution's login credentials. If you have trouble logging in, contact me (s.schmerler@hzdr.de). See you there!

[elcorto]

I'm taking the liberty to close this issue. If you want to join the chat, please do so as described above.