search

EleutherAI / elk

Keeping language models honest by directly eliciting knowledge encoded in their activations.
MIT License
182 stars 33 forks source link

Can this code suppot free-form QA task? #298

Open Tan-Hexiang opened 3 months ago

Tan-Hexiang commented 3 months ago

I noticed that the datasets supported in the code are all multiple-choice and classification types, such as IMDB, QNLI, and BoolQ. Can the code in this repository support free-form types of tasks, such as TriviaQA and NQ? For free-form QA tasks, the possible answer space is infinite. How should the CCS method allocate X+ and X- labels for such tasks?

Tan-Hexiang commented 3 months ago

@lauritowal @artkpv @crowsonkb @trisongz

artkpv commented 3 months ago

Free form tasks, imho, can be reduced to a classification tasks by appending question and answer each time like [Original Question] Answer: "[Sampled Answer]" Is the last answer true or false? True / False (contrast pair). And so on for multi-class