issues
search
symflower
/
eval-dev-quality
DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
57
stars
3
forks
source link
Extract model costs into log and CSVs
#210
Open
bauersimon
opened
5 days ago
bauersimon
commented
5 days ago
[x] Extract model costs (per M request token + per M response token + per request + per response) and write into csv reports
[ ] Check other API providers too (@bauersimon knows about that e.g. Mistral, Fireworks, Groq, ...)
ruiAzevedo19
commented
2 days ago
Groq Cloud
source:
https://console.groq.com/docs/models
Only supports the following models (pricing: per 1M tokens, input/output):
LLaMA3 8b: $0.05/$0.08
LLaMA3 70b: $0.59/$0.79
Mixtral 8x7b: $0.24/$0.24
Gemma 7b: $0.07/$0.07
Whisper: $0.03/hour transcribed
Note: does not have a model which is in our top 10
Fireworks
source:
https://fireworks.ai/pricing
pricing $/1M tokens (both input and output tokens)
0B - 16B: $0.20
16.1B - 80B: $0.90
MoE 0B - 56B (e.g. Mixtral 8x7B): $0.50
MoE 56.1B - 176B (e.g. DBRX, Mixtral 8x22B): $1.20
Yi Large: $3.00
Note: just has one model which is in our top 10:
llama-3-70b-instruct
Mistral
Only supports the following models (pricing: per 1M tokens, input/output):
open-mistral-7b: $0.25/$0.25
open-mixtral-8x7b: $0.7/$0.7
open-mixtral-8x22b: $2/$6
mistral-small-2402: $1/$3
codestral-2405: $1/$3
mistral-medium-2312: $2.7/$8.1
mistral-large-2402: $4/$12
mistral-embed: $0.1/$0.1
Mistral 7B: $0.75/$0.75
Mistral Small: $2.5/$7.5
Note: does not have a model which is in our top 10
Notes
Looking at the
Groq
and
Mistral
JSON schema, there is no pricing information for models
All the API's require an API key even to list the models
Fireworks
also is missing pricing information for models