search

openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.76k stars 2.58k forks source link

Local run doesn't save logs to disk #1459

Closed charles-somm closed 8 months ago

charles-somm commented 8 months ago

Describe the bug

After running a successful eval, the records are supposed to be saved to tmp/evallogs, but I can't find them. Setting the --local-run flag explicitly doesn't work either.

However, setting a --record_path explicitly works and correctly writes the results to the provided path.

Proposed fix

I managed to fix the issue by changing the default record_path /tmp/evallogs/ to a relative path tmp/evallogs/ in oaieval.py. I'm not sure if this is a bug or if I'm missing something.

To Reproduce

Run an eval with the CLI command: oaieval gpt-3.5-turbo <eval-name>

Code snippets

# evals/cli/oaieval.py - line 178

 record_path = (
        f"/tmp/evallogs/{run_spec.run_id}_{args.completion_fn}_{args.eval}.jsonl"
        if args.record_path is None
        else args.record_path
    )

OS

Ubuntu 22.04.3 (WSL)

Python version

Python 3.11.4

Library version

latest

charles-somm commented 8 months ago

Sorry I was obviously missing something. The logs are correctly written to the absolute path.