CP for LLMs

[pat-alt]

Issue set up for Experiment Week

• [x] Study this paper
• [x] Get data from hugging face
• [x] ~Train small transformer model from scratch~
• [x] ~Look at fine-tuning of pre-trained model~

[pat-alt]

cc @florisdenhengst

[pat-alt]

Notes from paper

• Context: LLMs for the task of multiple-choice question answering (MCQA)
• Uncertainty, as estimated through CP, highly correlated with model accuracy
• Working with inductive/split conformal prediction to avoid retraining
• Model: LLaMA-13B
• Data: MMLU benchmark containing MCQA question from 57 domains, 4 possible answers each
• Find that set size is higher on average (closer to 4 for more difficult domains)
• Find that set size negatively correlates with top-1 accuracy and argue that this can be used to filter low-quality predictions
• Also find fairly robust size-stratified coverage, even though it does not appear they used adaptive prediction sets
• Show that coverage guarantees do not hold if exchangeability assumption is violated (see below)

• Code released as notebook: https://github.com/bhaweshiitk/ConformalLLM/blob/main/Conformal_Prediction_for_LLMs.ipynb

[pat-alt]

For work in Julia, we probably want to look at Transformers.jl

[pat-alt]

Idea: for small dataset/model, check if we can use conformal training