UQ4ML – Uncertainty Quantification Techniques in Machine Learning Models

[paramnav]

UQ4ML – Uncertainty Quantification Techniques in Machine Learning Models

This session focuses on uncertainty quantification (UQ) techniques in machine learning models and their applications across scientific disciplines. Regardless of the field of research, UQ is an important technique that can lead to more nuanced result interpretation, better understanding of AI models, and more principled decision-making.

We will begin the session by showcasing various UQ techniques that we have implemented in our AI models and discussing their impact on our research. This will be followed by a roundtable discussion on additional possibilities for UQ, its applications, and implications within your own field of research.

Some questions to consider:

1) How to reduce parameter uncertainty:
    Adding more training data (if feasible) – does it reduce or increase uncertainty?
    Incorporating physical constraints or domain knowledge
2) How to reduce model uncertainty:
    Exploring different model architectures
    Improving input features

3) Interpreting UQ: Various approaches and perspective outside our experience in Earth and Environment disciplines.

We invite you to share your experiences:

Are you curious about the knowledge your AI models possess beyond predictions?
Have you applied techniques like Bayesian Neural Networks, Monte Carlo Dropout, Neural Network Ensembles, or Softmax Classification to quantify uncertainty?
What are your takeaways or challenges?
What is the ideal way to quantify uncertainty in your field, and how do you make decisions based on it? How crucial is this knowledge?

Organizational Details:

1) Hosts:
    Everardo González (GEOMAR) (egonzalez@geomar.de)
    Naveenkumar Parameswaran (GEOMAR)(nparameswaran@geomar.de)
2) Format:
    Introductory talk describing various UQ methods
    Discussion round with topics like the questions mentioned above
3) Timeframe: 1.5h
4) Number of participants: 3-20
5) Material:     
    Projector
    Whiteboard or flip chart

[helenehoffmann]

Thank you for your contribution. That sounds interesting. Is this connected or are you aware of the workshop in this topic at the satellite day at Jülich?

Unfortunately you did not use the prepared issue template. could add information about the sessions lengths, number of participants and material needed? Thank you!

[paramnav]

This is not connected to the workshop. I was not completely aware of the workshop. But this could be a good follow-up in discussing the use-cases and the experiences in dealing with UQ in ML. We would like to discuss with people who are dealing with uncertainties in their project or thinking of implementing them.

[helenehoffmann]

@psteinb @codingS3b @elcorto maybe you can coordinate your efforts so it fits together.

[helenehoffmann]

@paramnav

Thank you so much for hosting a session at this Unconference! Your participation is what makes this event truly special. Please, be prepared to give a 1-minute pitch for your topic to share at the beginning of the Unconference.

We’re excited to see you there and can’t wait for an amazing event! Best, The Organizing Team

[paramnav]

https://docs.google.com/document/d/1TfBVaOxEZaIIDdPZzHO6Xen2JP4nImOYVN5RYQS1UQk/edit

[SusanneWenzel]

Dear @paramnav ,

many thanks for your contribution! We hope your session was successful and constructive!

To make this not only a nice experience during the conference but a sustainable format, we would like to release a short result on the Helmholtz AI Website. Please provide us with a little report about the discussion in your session:

• One or two intro sentences about the topic and the respective format of your session
• What was most controversial, or about what did all participants agree on?
• What were the key ideas?
• Do you see any follow up on this discussion?

Depending on the intensity of the session it can be shorter (1/3 page) or longer (one pager).

Please post your report here by end of next week. Ina already confirmed to release that at a HAICON24 subpage at Helmholtz.ai.

Best regards, @helenehoffmann and Susanne

[paramnav]

Dear @helenehoffmann and @SusanneWenzel ,

The topic of the unconference was to discuss about how uncertainty quantification(UQ) is used in applied machine learning(ML). Everardo gave a short presentation of our use case in ocean science and started with different techniques to handle UQ in ML models. We then had a short introduction round where each one of them said, what they do, and their experience with UQ. We had a good mixture of people who wanted to apply UQ in their work, people who already applied UQ, and experts in the theory and development of UQ methods. The unconference was well supported by the workshop on a similar theme. We then had a discussion about which methods do we try first, for our problems. We had some of them present the methods they use, such as softmax, when well calibrated, ensembling(checkpount ensembling, monte carlo dropout), bootstraping etc.

We had a thought provoking discussion on how to evaluate the different methods that we used. In theory, there is no single method that is ideal for a use case, but also not a lot of metrics to measure which method is the best. We had a discussion on how adding random features could tell us about the UQ method that we use. We then had some inputs regarding literature that could be read, for people starting to apply UQ.

We then talked about how to use UQ in certain applications, and how do we reject predictions in case of higher uncertainty and how to fix this threshold. Some other use cases that were discussed were inverse problems(when the uncertainty from the model is lower than the sampling uncertainty), for manual data point rectification, in health for selective classification.

We could see some followups from the discussion, especially from people who wanted to apply UQ in their work. It was also a good networking opportunity for people from different levels of expertise in different fields.

Best regards, @paramnav