search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
597 stars 109 forks source link

[Doc] Security Guide --No such file or directory: 'security_guide.csv' #660

Closed cmougan closed 3 weeks ago

cmougan commented 3 weeks ago

TLDR: Cant find the file easily in open AI repo

If I follow the tutorial of 'Security Guide'.

Create a file called sec.py ''' from inspect_ai import Task, eval, task from inspect_ai.dataset import csv_dataset from inspect_ai.scorer import model_graded_fact from inspect_ai.solver import generate, system_message

SYSTEM_MESSAGE = """ You are a computer security expert and you are tasked with providing concise response to the following questions. Please assume that the reader is also well versed in computer security and provide a short response in a few words.

@task def security_guide(): return Task( dataset=csv_dataset("security_guide.csv"), solver=[system_message(SYSTEM_MESSAGE), generate()], scorer=model_graded_fact(), )

""" '''

Execute '$ inspect eval sec.py --model google/gemini-1.0-pro''

And then get 'No such file or directory: 'security_guide.csv''

Then I proceed to search for the file in the linked Webpage https://github.com/openai/evals

And cant find it directly.

Screenshot 2024-10-07 at 10 45 30 Screenshot 2024-10-07 at 10 45 54

This is a minor issue and the file can be easily found with a bit of effort. But it creates a minimal user friction that can be solved very easily.

If the authors agree, I will be happy to do a PR solving the issue.

jjallaire-aisi commented 3 weeks ago

Thanks! I made a change to use example_dataset() instead which should resolve this! https://github.com/UKGovernmentBEIS/inspect_ai/commit/d25ba0c99ddc32d2b7d96f33512aed7f96c2d1d4