search

Future-House / paper-qa

High accuracy RAG for answering questions from scientific documents with citations
Apache License 2.0
6.23k stars 586 forks source link

Agent LLM Token Usage and Cost #603

Open irenemadejski opened 3 days ago

irenemadejski commented 3 days ago

Is there a way to retrieve the token counts for the agent llm? I am using different models for the agent llm and the main/summarize llm, and I am currently using the cost and token_counts to retrieve this info. Only the cost and token counts of the main/summary llm are being returned in the token_counts dictionary, however the agent llm is not in the token_counts dictionary.

dosubot[bot] commented 3 days ago

To retrieve the token counts for the agent LLM, ensure that the add_tokens method in the Answer class is correctly processing the agent LLM's results. The method checks if the result is an instance of Message and if result.info contains both "model" and "usage" keys. If these conditions are not met, the token usage won't be added to the token_counts dictionary [1].

Make sure that the agent LLM's results are being passed with the necessary information. If the Message instances for the agent LLM do not have the required "model" and "usage" keys in result.info, they will be skipped. You might need to verify that the agent LLM's outputs are being correctly formatted and passed to this method.

To continue talking to Dosu, mention @dosu.

whitead commented 3 days ago

@irenemadejski What setting are you using? Some settings do not use the agent at all. But, yes you should get a token summary that includes token usage for all steps