search

AIToolsLab / questions

Writer-directed questions
0 stars 0 forks source link

Build a feedback-collection tool #12

Open kcarnold opened 1 year ago

kcarnold commented 1 year ago
  1. Collect a bunch of contexts to generate questions for -> list of texts
  2. In batch, ask our LMs to generate questions for each context; aggregate the results. -> list of, for each text, list of questions and metadata (what system generated it). Ask each LM for several questions, log all of them.
  3. Streamlit app: pick a random context from the list, pick some subset of the questions (random?) ask user which one is "best" (most appropriate, most helpful, ...?) -> user's choice of which question. Maybe have them rank the questions? (drag them into best-to-worst?) (or: pick the best)
    • log this in a database:
context, question, system_that_generated, rater_id, rank
kcarnold commented 1 year ago

@souadyakubu showed some initial results on this. I encouraged her to put it on GitHub.