Complete the initial test

[garyfeng]

create a task related to item creation, scoring, or review, and test until we get reasonable responses. Clean up the log

[garyfeng]

See the first example in #1. Will expand on this.

[garyfeng]

Potential ideas

• Fairness review: use ETS guidelines (published), evaluate items and explain. Experiment with prompts to optimize
• Content review: item relatedness, similarity
• Fact check: checking items or responses, may use plugins
• Standards alignment: given an item, align with common core standards (as context)
• item cloning: given an item, change scenarios and parameters but keep content and difficulty
• item template generator: given an item, extract a template

[garyfeng]

For autoGPT, given a workflow document, it should automatically complete the task without human intervention.

How do we enforce the COT via prompt engineering?

• Role assignment: tell GPT that she is an assessment developer
• job description: tasks and orders
• success criteria: how to evaluate and when to stop
• references: basis for evaluation

[garyfeng]

This means we need to customize the "actions" to the job.