search

Aider-AI / aider

aider is AI pair programming in your terminal
https://aider.chat/
Apache License 2.0
20.95k stars 1.94k forks source link

REQ: Ability to set max token threshold, at which a confirmation warning would appear #2041

Open 5ocworkshop opened 2 weeks ago

5ocworkshop commented 2 weeks ago

Issue

I am doing quite a bit of debugging at the moment and it has involved having to enable a number of nested stack traces. This results in a lot of output going to the API and it got out of hand rather quickly and suddenly I was submitting $.30 prompts.

It might be nice to have a configuration option where you could set a token threshold to act as a safety valve in a situation like this to cause a warning to appear and require a (Y/N) to continue, so you have a chance to reconsider if you discover a huge number of tokens is about to be submitted and you didn't realize it.

Version and model info

Aider v0.59.1 Main model: claude-3-5-sonnet-20240620 with diff edit format, prompt cache, infinite output Weak model: claude-3-haiku-20240307 Git repo: .git with 25 files

paul-gauthier commented 5 days ago

This is a bit tricky. Do we set a token threshold? Or a dollar threshold? Active by default? If so, with what default threshhold?

I'm concerned that it's potentially bad to have this turned on by default. But that people won't figure out they might want to turn this on until after they've already learned a hard lesson ;)

5ocworkshop commented 5 days ago

This is a bit tricky. Do we set a token threshold? Or a dollar threshold? Active by default? If so, with what default threshhold?

I'm concerned that it's potentially bad to have this turned on by default. But that people won't figure out they might want to turn this on until after they've already learned a hard lesson ;)

You make excellent points. I guess in the case of Sonnet 3.5 the 200K threshold maybe is the "oh crap maybe I need to set something lower than maxing out the token budget" :). I've run afoul of this during some iterative debugging where I also had maybe six files open, two of which we quite large.

I don't think you necessarily enable it by default, but you could highlight on launching aider whether a limit is set or not, to "nudge" people that it is a settable option they might want to consider? "Max tokens currently set to: Model maximum. To set a lower hard or soft threshold use the --token-limit-[soft|hard] option"?