symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
137 stars 5 forks source link

Track query tokens and save them to CSV #347

Open bauersimon opened 1 month ago

bauersimon commented 1 month ago

Problem

With the release of OpenAI o1-preview and o1-mini, the costs also include internal reasoning tokens. Therefore, pricing comparisons with classical one-shot models is not possible. I.e. o1-mini costs $15 and Sonnet 3.5 costs $18 but Sonnet is so much cheaper cause it does not have intermediate reasoning.

Solution (optional)

Track the actual tokens of each query from the model response according to https://platform.openai.com/docs/api-reference/chat/object and save them to the CSV.