UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
615 stars 114 forks source link

[feature request] live/realtime `inspect view` #559

Open JasonGross opened 1 month ago

JasonGross commented 1 month ago

I'd like to be able to see messages as they are sent and responded to, but it seems the transcript is not written to until the very end. Can there be an option --live-transcript or similar to inspect eval to write to the transcript on every message update (or have a function that is update_transcript or similar), so I can see partial progress?

jjallaire-aisi commented 1 month ago

We are working on this, but it will come in stages. First will be to make the log/transcript streaming (likely moving it into SQLite so we can handle much larger log file sizes). Then we'll hookup UI to the streamed transcript (likely we'll first do this in the terminal and then eventually in inspect view)