openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.36k stars 2.55k forks source link

Request to change arithmetical_puzzles prompting #1448

Open ArcticBeat05 opened 6 months ago

ArcticBeat05 commented 6 months ago

Describe the feature or improvement you're requesting

arithmetical_puzzles prompts does not allow proper reasoning The following is added at the end of every prompt..."Do not give any reasoning or logic for your answer."

In a previous PR, usama-openai requested to change the prompting of eval math_for_5th-grader to add better prompting to allow the LLM to reason.

https://github.com/openai/evals/pull/1293#pullrequestreview-1517023638

Request to change the prompt for arithmetical_puzzles as well.

Additional context

No response