Create a file called sec.py
'''
from inspect_ai import Task, eval, task
from inspect_ai.dataset import csv_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message
SYSTEM_MESSAGE = """
You are a computer security expert and you are tasked with
providing concise response to the following questions.
Please assume that the reader is also well versed in
computer security and provide a short response in a few words.
TLDR: Cant find the file easily in open AI repo
If I follow the tutorial of 'Security Guide'.
Create a file called sec.py ''' from inspect_ai import Task, eval, task from inspect_ai.dataset import csv_dataset from inspect_ai.scorer import model_graded_fact from inspect_ai.solver import generate, system_message
SYSTEM_MESSAGE = """ You are a computer security expert and you are tasked with providing concise response to the following questions. Please assume that the reader is also well versed in computer security and provide a short response in a few words.
@task def security_guide(): return Task( dataset=csv_dataset("security_guide.csv"), solver=[system_message(SYSTEM_MESSAGE), generate()], scorer=model_graded_fact(), )
""" '''
Execute '$ inspect eval sec.py --model google/gemini-1.0-pro''
And then get 'No such file or directory: 'security_guide.csv''
Then I proceed to search for the file in the linked Webpage https://github.com/openai/evals
And cant find it directly.
This is a minor issue and the file can be easily found with a bit of effort. But it creates a minimal user friction that can be solved very easily.
If the authors agree, I will be happy to do a PR solving the issue.