search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
627 stars 118 forks source link

Add more documentation on CLI #868

Open mrahtz opened 2 days ago

mrahtz commented 2 days ago

Hey folks!

As far as I can tell, the documentation at https://inspect.ai-safety-institute.org.uk (which is otherwise fantastic, by the way!) doesn't have any detailed documentation on the CLI.

For example, today, I wanted to know how the --log-dir and --trace options behave - but googling for aisi inspect "--log-dir" basically the only result was https://inspect.ai-safety-institute.org.uk/eval-logs.html, which wasn't obviously what I was looking for (I had to read through the page to find that it did actually have a paragraph on --log-dir), and googling for aisi inspect "--trace" didn't return any results.

So it would be great to have more docs on the CLI - all the different commands that can be run, and what all the arguments to each command does.

Thanks!

jjallaire commented 2 days ago

Agree 100%, we will take care of this soon! In the meantime (acknowledging that its a poor substitute), try inspect --help and then e.g. inspect eval --help for individual commands. Minimally, we'll add an automated step to our doc build to generate this same output. I believe that Click will also give us a way to interrogate the CLI as JSON which will also enable us to create a nicer page for each command.