search

getappmap / appmap-js

Client libraries for AppMap
45 stars 12 forks source link

Show cost estimates when calling the LLM #1842

Closed dividedmind closed 3 weeks ago

dividedmind commented 3 weeks ago

No effort is currently done to summarize it; it's just output in the logs:

$ PYTHONPATH=. python ./appmap/solve.py --path_conda $(conda info --base) --split dev --num_workers 1 --appmap_command ~/projects/appmap-js/packages/cli/built/cli.js --filter marshmallow
$ grep cost logs/solve/marshmallow-code__marshmallow-*/*/*.log
logs/solve/marshmallow-code__marshmallow-1343/1/apply.log:Tokens (prompt/compl/total): 535/30/565, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1343/1/generate.log:Tokens (prompt/compl/total): 13091/31/13122, cost: $0.07
logs/solve/marshmallow-code__marshmallow-1343/1/generate.log:Tokens (prompt/compl/total): 12502/401/12903, cost: $0.06
logs/solve/marshmallow-code__marshmallow-1343/1/list-files.log:Tokens (prompt/compl/total): 786/30/816, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1343/1/plan.log:Tokens (prompt/compl/total): 941/47/988, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1343/1/plan.log:Tokens (prompt/compl/total): 1341/51/1392, cost: $0.01
logs/solve/marshmallow-code__marshmallow-1343/1/plan.log:Tokens (prompt/compl/total): 6183/653/6836, cost: $0.03
logs/solve/marshmallow-code__marshmallow-1359/1/apply.log:Tokens (prompt/compl/total): 912/19/931, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1359/1/generate.log:Tokens (prompt/compl/total): 15315/34/15349, cost: $0.08
logs/solve/marshmallow-code__marshmallow-1359/1/generate.log:Tokens (prompt/compl/total): 14726/778/15504, cost: $0.08
logs/solve/marshmallow-code__marshmallow-1359/1/list-files.log:Tokens (prompt/compl/total): 614/19/633, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1359/1/plan.log:Tokens (prompt/compl/total): 941/47/988, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1359/1/plan.log:Tokens (prompt/compl/total): 1246/67/1313, cost: $0.01
logs/solve/marshmallow-code__marshmallow-1359/1/plan.log:Tokens (prompt/compl/total): 5957/481/6438, cost: $0.03
logs/solve/marshmallow-code__marshmallow-1359/2/apply.log:Tokens (prompt/compl/total): 1099/29/1128, cost: $0.01
logs/solve/marshmallow-code__marshmallow-1359/2/generate.log:Tokens (prompt/compl/total): 24804/14/24818, cost: $0.12
logs/solve/marshmallow-code__marshmallow-1359/2/generate.log:Tokens (prompt/compl/total): 24215/965/25180, cost: $0.13
logs/solve/marshmallow-code__marshmallow-1359/2/list-files.log:Tokens (prompt/compl/total): 587/29/616, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1359/2/plan.log:Tokens (prompt/compl/total): 941/15/956, cost: $0.00
logs/solve/marshmallow-code__marshmallow-1359/2/plan.log:Tokens (prompt/compl/total): 1246/54/1300, cost: $0.01
logs/solve/marshmallow-code__marshmallow-1359/2/plan.log:Tokens (prompt/compl/total): 6014/454/6468, cost: $0.03
kgilpin commented 3 weeks ago

@dividedmind I'm seeing errors like this in the log: 

Failed to calculate number of tokens, falling back to approximate count Error: Unknown model
    at getEncodingNameForModel (/Users/kgilpin/source/appland/appmap-js/node_modules/js-tiktoken/dist/lite.cjs:223:13)
    at encodingForModel (/Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/core/dist/utils/tiktoken.cjs:23:59)
    at ChatOpenAI.getNumTokens (/Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/core/dist/language_models/base.cjs:182:75)
    at /Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/openai/dist/chat_models.cjs:623:42
    at async Promise.all (index 1)
    at async ChatOpenAI.getNumTokensFromMessages (/Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/openai/dist/chat_models.cjs:621:33)
Project info obtained
Tokens (prompt/compl/total): 24622/35/24657, cost: $0.12
dividedmind commented 3 weeks ago

@dividedmind I'm seeing errors like this in the log:

Failed to calculate number of tokens, falling back to approximate count Error: Unknown model
    at getEncodingNameForModel (/Users/kgilpin/source/appland/appmap-js/node_modules/js-tiktoken/dist/lite.cjs:223:13)
    at encodingForModel (/Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/core/dist/utils/tiktoken.cjs:23:59)
    at ChatOpenAI.getNumTokens (/Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/core/dist/language_models/base.cjs:182:75)
    at /Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/openai/dist/chat_models.cjs:623:42
    at async Promise.all (index 1)
    at async ChatOpenAI.getNumTokensFromMessages (/Users/kgilpin/source/appland/appmap-js/node_modules/@langchain/openai/dist/chat_models.cjs:621:33)
Project info obtained
Tokens (prompt/compl/total): 24622/35/24657, cost: $0.12

Is that with the default model? Did you run yarn to update js-tiktoken?

appland-release commented 3 weeks ago

:tada: This PR is included in version @appland/navie-v1.14.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

appland-release commented 3 weeks ago

:tada: This PR is included in version @appland/appmap-v3.148.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: