search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in HEIM (https://arxiv.org/abs/2311.04287) and vision-language models in VHELM (https://arxiv.org/abs/2410.07112).
https://crfm.stanford.edu/helm
Apache License 2.0
1.95k stars 252 forks source link

Estimate cost for AI21 models #780

Closed teetone closed 2 years ago

teetone commented 2 years ago

New pricing model: https://studio.ai21.com/pricing

teetone commented 2 years ago

I did a dry run with P1 + P2 RunSpecs and estimated:

For 1 random seed:

ai21/j1-large:
cost = pricing.price_per_1k_token * (num_tokens / 1000) + pricing.price_per_request * num_request
$370.26 = $0.03 * (11010522 / 1000) + $0.0003 * 133156

ai21/j1-grande:
cost = pricing.price_per_1k_token * (num_tokens / 1000) + pricing.price_per_request * num_request
$987.37 = $0.08 * (11010522 / 1000) + $0.0008 * 133156

ai21/j1-jumbo:
cost = pricing.price_per_1k_token * (num_tokens / 1000) + pricing.price_per_request * num_request
$3418.41 = $0.25 * (11010522 / 1000) + $0.005 * 133156

Total: $4776.04