Open ErikBjare opened 2 weeks ago
Right now we just count the token length of the chat log, we should capture the actual spend.
Not sure if very high priority, not obvious how to do it the right way.
Might be interesting to set a token/message limit for evals, stopping long runs leading nowhere (not very common).
Right now we just count the token length of the chat log, we should capture the actual spend.
Not sure if very high priority, not obvious how to do it the right way.
Might be interesting to set a token/message limit for evals, stopping long runs leading nowhere (not very common).