Design a way to evaluate question generation approaches

[kcarnold]

• Can we generate questions of various types?
  • [ ] Can we label the type of question using a language model?
  • [ ] Can we come up with prompts that get a LM to generate questions of each type?
• For LLMs that offer logprobs, we can evaluate the logprob of question in context.

[kcarnold]

Data?? https://docs.google.com/spreadsheets/d/1AW-bJkonHvb25VZCdj9HN7ha-h1GQbf0bm0FTjiVOGs/edit#gid=1914027785

[kcarnold]

https://chat.openai.com/share/307afa15-e4bb-4afb-81dc-ba683a87a9f1

[kcarnold]

A possible tool for evaluating: https://github.com/typpo/promptfoo