aisi-inspect
commented
1 month ago The messages will always a list of messages initialized from the input. I would definitely code against the messages not the input, which will allow for other solvers to be able to tweak the messages in some fashion (not likely, but its nice to not shut the door on this entirely). So something more like this:
async def solve(state: TaskState, generate: Generate) -> TaskState:
input = state.messages.copy()
state.messages = []
for turn in input:
state.messages.append(turn)
state = await generate(state)
return stateIn terms of always_false_scorer(), you should just be able to have no scorer at all in your Task (then eval will just skip the scoring step).
Lovkush-A
jjallaire
I created a toy example using Inspect for multi-turn dialogue. https://github.com/Lovkush-A/inspect_multiturn_dialogue
This might be a lot to ask, but can somebody in AISI skim through and let me know if there is cleaner way of doing things? There is only one notebook to read through, in which i create a dataset, solver, scorer, task and run the evals. I plan to share this around AI safety community so I want to make sure I am using framework properly.
Thank you!