Open 5ocworkshop opened 2 weeks ago
This is a bit tricky. Do we set a token threshold? Or a dollar threshold? Active by default? If so, with what default threshhold?
I'm concerned that it's potentially bad to have this turned on by default. But that people won't figure out they might want to turn this on until after they've already learned a hard lesson ;)
This is a bit tricky. Do we set a token threshold? Or a dollar threshold? Active by default? If so, with what default threshhold?
I'm concerned that it's potentially bad to have this turned on by default. But that people won't figure out they might want to turn this on until after they've already learned a hard lesson ;)
You make excellent points. I guess in the case of Sonnet 3.5 the 200K threshold maybe is the "oh crap maybe I need to set something lower than maxing out the token budget" :). I've run afoul of this during some iterative debugging where I also had maybe six files open, two of which we quite large.
I don't think you necessarily enable it by default, but you could highlight on launching aider whether a limit is set or not, to "nudge" people that it is a settable option they might want to consider? "Max tokens currently set to: Model maximum. To set a lower hard or soft threshold use the --token-limit-[soft|hard] option"?
Issue
I am doing quite a bit of debugging at the moment and it has involved having to enable a number of nested stack traces. This results in a lot of output going to the API and it got out of hand rather quickly and suddenly I was submitting $.30 prompts.
It might be nice to have a configuration option where you could set a token threshold to act as a safety valve in a situation like this to cause a warning to appear and require a (Y/N) to continue, so you have a chance to reconsider if you discover a huge number of tokens is about to be submitted and you didn't realize it.
Version and model info
Aider v0.59.1 Main model: claude-3-5-sonnet-20240620 with diff edit format, prompt cache, infinite output Weak model: claude-3-haiku-20240307 Git repo: .git with 25 files