search

run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
36.08k stars 5.14k forks source link

[Bug]: Anthropic LLMs as REACT agents stop before they complete the answer #13480

Closed gich2009 closed 5 months ago

gich2009 commented 5 months ago

Bug Description

Anthropic models are good when used as REACT agents. They follow the thought, action, input flow quite well but when they generate the answer, the generation always cuts off before the entire answer is produced.

Version

llama-index-core==0.10.35 llama-index-llms-anthropic==0.1.11

Steps to Reproduce

Run a React agent with an anthropic llm as the underlying model.

Relevant Logs/Tracbacks

HTTP Request: POST https://api.anthropic.com/v1/messages?beta=tools "HTTP/1.1 200 OK"
Thought: I have calculated the estimated net GHG emission reductions for the project using the equations from the methodology and the project-specific values. I can now generate section 4.4 of the PDD.
Answer: 4.4 Net GHG Emission Reductions and Removals

The net GHG emission reductions of the project are calculated using Equation 1 from the VCS VMR0006 Methodology for Installation of High Efficiency Firewood Cookstoves v1.2:

ERy = ∑∑(By,savings,i,j × N0,i,j × ny,i,j × μy × fNRB,y × NCVbiomass × (EFwf,CO2 + EFwf,non CO2) × AdjLE × (1-ud))

Where:
ERy = Emission reductions in year y (tCO2e)
By,savings,i,j = Quantity of woody biomass saved per project device i and batch j in year y (tonnes) 
N0,i,j = Number of project devices commissioned
ny,i,j = Proportion of commissioned project devices operating in year y
μy = Adjustment for continued use of pre-project devices in year y
fNRB,y = Fraction of non-renewable biomass (%)
NCVbiomass = Net calorific value of the non-renewable woody biomass (TJ/tonne)
EFwf,CO2 = CO2 emission factor for non-renewable woody biomass (tCO2/TJ)
EFwf,non CO2 = Non-CO2 emission factor for non-renewable woody biomass (tCO2e/TJ)
AdjLE = Adjustment factor for leakage (fraction) 
ud = Uncertainty deduction for fnrb (%)

The quantity of woody biomass saved (By,savings,i,j) is determined using the procedures in AMS-II.G, as referenced by the methodology. The leakage adjustment factor (AdjLE) is determined using the procedures in AMS-II.G and fixed at validation.

Plugging in the project-specific values:
By,savings,i,

('4.4 Net GHG Emission Reductions and Removals\n\nThe net GHG emission reductions of the project are calculated using Equation 1 from the VCS VMR0006 Methodology for Installation of High Efficiency Firewood Cookstoves v1.2:\n\nERy = ∑∑(By,savings,i,j × N0,i,j × ny,i,j × μy × fNRB,y × NCVbiomass × (EFwf,CO2 + EFwf,non CO2) × AdjLE × (1-ud))\n\nWhere:\nERy = Emission reductions in year y (tCO2e)\nBy,savings,i,j = Quantity of woody biomass saved per project device i and batch j in year y (tonnes) \nN0,i,j = Number of project devices commissioned\nny,i,j = Proportion of commissioned project devices operating in year y\nμy = Adjustment for continued use of pre-project devices in year y\nfNRB,y = Fraction of non-renewable biomass (%)\nNCVbiomass = Net calorific value of the non-renewable woody biomass (TJ/tonne)\nEFwf,CO2 = CO2 emission factor for non-renewable woody biomass (tCO2/TJ)\nEFwf,non CO2 = Non-CO2 emission factor for non-renewable woody biomass (tCO2e/TJ)\nAdjLE = Adjustment factor for leakage (fraction) \nud = Uncertainty deduction for fnrb (%)\n\nThe quantity of woody biomass saved (By,savings,i,j) is determined using the procedures in AMS-II.G, as referenced by the methodology. The leakage adjustment factor (AdjLE) is determined using the procedures in AMS-II.G and fixed at validation.\n\nPlugging in the project-specific values:\nBy,savings,i,')
dosubot[bot] commented 5 months ago

To address the issue where the Anthropic models used as REACT agents cut off the answer generation prematurely, it's likely related to the max_tokens parameter setting. This parameter controls the maximum number of tokens the model generates, and if the output exceeds this limit, it will be truncated. Here's how you can tackle this:

  1. Adjust the max_tokens Parameter: Increase the max_tokens value when initializing the Anthropic class to accommodate longer outputs. For example:

    llm = Anthropic(model="your-model-name", max_tokens=1024)

  2. Ensure Proper Tokenization: Verify that the tokenization process matches your expectations and that no post-processing steps or API response size limitations are truncating the model's output.

  3. Check Callbacks and Decorators: Make sure that decorators and callback functions wrapping the main methods do not unintentionally truncate the output.

  4. Update Your Packages: Confirm that you're using the latest versions of llama-index-core and llama-index-llms-anthropic, as they may contain fixes or improvements.

  5. Implement Debugging and Logging: Add detailed logging around the API call and response handling to identify if the truncation occurs during these stages.

If these steps do not resolve the issue, consider reaching out to the support or community forums for llama-index-core and llama-index-llms-anthropic with a detailed description of your problem and any relevant code snippets.

#### Details **Sources** - [llama-index-integrations/llms/llama-index-llms-anthropic/llama_index/llms/anthropic/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-anthropic/llama_index/llms/anthropic/base.py) - [llama-index-core/llama_index/core/node_parser/text/token.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/node_parser/text/token.py) - [llama-index-core/llama_index/core/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/utils.py) - [CHANGELOG.md](https://github.com/run-llama/llama_index/blob/main/CHANGELOG.md) - [llama-index-legacy/llama_index/legacy/llms/anthropic.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/llms/anthropic.py) - [llama-index-integrations/multi_modal_llms/llama-index-multi-modal-llms-anthropic/llama_index/multi_modal_llms/anthropic/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/multi_modal_llms/llama-index-multi-modal-llms-anthropic/llama_index/multi_modal_llms/anthropic/base.py) - [llama-index-integrations/llms/llama-index-llms-anthropic/llama_index/llms/anthropic/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/llms/llama-index-llms-anthropic/llama_index/llms/anthropic/utils.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

gich2009 commented 5 months ago

Thanks @dosu. Don't know why it didn't occur to me to change the max_tokens/num_output parameter.