mongodben
commented
3 months ago hi Eric,
Good idea. I think this is something worth including natively in the framework b/c i could imagine the use case is fairly generic + many if not most LLM APIs include token usage info in the response so it'd be a low-cost thing to add.
However, just FYI, i don't think we're going to be able to get to this in the immediate future given some other features we're working on.
We might implement a limit to the number of calls being made (based on user account). Also, we would like to measure/track the usage of LLM calls. For that, I would like to get the usage response (number of tokens used by LLM) from the LLM call, accessible in the API response, or possibly stored in the /conversations collection.