Add some metrics to prompt logs - Githubissues

continuedev / continue

⏩ Continue is the leading open-source AI code assistant. You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains

https://docs.continue.dev/

Apache License 2.0

19.43k stars 1.69k forks source link

Add some metrics to prompt logs #2995

Open RomneyDa opened 2 days ago

RomneyDa commented 2 days ago

Validations

[X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
[X] I'm not able to find an open issue that requests the same enhancement

Problem

No token metrics on logs

Solution

https://discord.com/channels/1108621136150929458/1306660426276405279 Add latency metrics to prompt logs e.g.

time from hitting send button to response start
speed token/s : time it took to stream the whole answer/nb of tokens in the answer
prompt size in tokens for the request
prompt size in tokens for the response