search

chenfei-wu / TaskMatrix

Other
34.51k stars 3.32k forks source link

How to reduce token usage #390

Open yaoye7758521 opened 1 year ago

yaoye7758521 commented 1 year ago

The usage fee for the text-davinci-3 model is $0.0200/1K tokens, which is very expensive.On average, around 3000 tokens are required per simple request,Each simple conversation approximately costs 0.1 US dollars,this is very economically unfriendly to users. image I am trying to use a more economical text ada-1 model. self.llm = OpenAI(temperature=0,model_name="text-ada-001") But the maxmum token size of this model is restricted to 2048,which lead to the following Error. openai.error.InvalidRequestError: This model's maximum context length is 2049 tokens, however you requested 2074 tokens (1818 in your prompt; 256 for the completion). Please reduce your prompt; or completion length. So I want to ask if there is a way to reduce the number of tokens per request or split each long request in order to use a more economical ada model. Looking forward to hearing from you,Thanks!

yaoye7758521 commented 1 year ago

image If the ada model is supported, the token cost will be reduced by 50 times the original cost.

ericthomas1 commented 1 year ago

If you can specify GPT-3.5 Turbo, its significantly less expensive @ $0.002 / 1K tokens. REF

yaoye7758521 commented 1 year ago

The maxmum token size of gpt-3.5-tuibo is also 2048.