[SPRINT] Dey+22 statistical test in RAIL

[torluca]

Dey+22 statistical test in RAIL

Implementation of calibrated predictive distributions (Dey+22) in rail.evaluate

Contacts: Time: All day Main communication channel: Slack GitHub repo: https://github.com/biprateep, https://github.com/LSSTDESC/RAIL

Goals and deliverable

The goal of the sprint activity would be a first implementation of the code developed by Bitrateep Dey in rail.evaluate

Resources and skills needed

Familiarity with rail and qp, Dey+22 work, willingness to learn new things

Detailed description

The idea of the sprint is to kick-off the work of implementing the calibrated predictive distributions presented in Dey+22 into the rail.evaluate module. Bitrateep's code is publicly available and a discussion on how to integrate it into rail needs to be conducted. Bitrateep is going to participate to this sprint hack, helping us in understanding and implementing his code.

[biprateep]

Pasting here some resources and topics for future expansion:

1. Local tests of conditional coverage via PIT: https://proceedings.mlr.press/v161/zhao21b/zhao21b.pdf
2. Recalibration based on conditional coverage: https://arxiv.org/abs/2205.14568 https://arxiv.org/abs/2110.15209

A non-exhaustive list of other tests of conditional coverage that we may want to evaluate and implement in the long run:

1. https://arxiv.org/abs/2106.00394
2. https://arxiv.org/abs/2002.10271
3. (More to be added after literature search)