UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://UKGovernmentBEIS.github.io/inspect_ai/
MIT License
400 stars 41 forks source link

Not possible to follow the basics Workflow tutorial #12

Closed jralduaveuthey closed 1 month ago

jralduaveuthey commented 1 month ago

I have tried both, installing via pip and cloning the repo, and it is just not possible to follow most of the first tutorial about the basics of Workflows.

Different problems:

aisi-inspect commented 1 month ago

Thank you for this! The Workflow article isn't intended a proper tutorial (it's more as you discovered illustrative examples that don't run w/o additional context). We are indeed planning on creating a proper tutorial that you can follow step by step. In the meantime we'll add a note to the workflow article to make it clear that the code is by and large not executable standalone).

aisi-inspect commented 1 month ago

Add a clarifying note to the docs on this here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/140c3d63ad42125213bffaa799147135c81857f3