Issue: load_summarization_chain in 'map_reduce' mode not breaking up document

omnific9 commented 1 year ago

Issue you'd like to raise.

I'm using the load_summarization_chain with the type of 'map_reduce' in the following fashion:

    summary_prompt = ChatPromptTemplate.from_template(
        "Write a long-form summary of the following text delimited by triple backquotes. "
        "Include detailed evidence and arguments to make sure the generated paragraph is convincing. "
        "Write in Chinese."
        "```{text}```"
    )

    combine_prompt = ChatPromptTemplate.from_template(
        "Write a long-form summary of the following text delimited by triple backquotes. "
        "Include all the details as much as possible."
        "Write in Chinese."
        "```{text}```"
    )
    combined_summary_chain = load_summarize_chain(llm=llm, chain_type="map_reduce", map_prompt=summary_prompt,
                                                  combine_prompt=combine_prompt, return_intermediate_steps=True)

    summary, inter_steps = combined_summary_chain({"input_documents": data, "token_max": 8000})

I've checked several tutorials and there doesn't seem to be anything wrong with this. But I keep getting this error: openai.error.InvalidRequestError: This model's maximum context length is 8192 tokens. However, your messages resulted in 15473 tokens. Please reduce the length of the messages.

It's as if the map_reduce chain isn't breaking up the data and running as if it's a stuffing chain instead. What did I do wrong?

Suggestion:

No response

dosubot[bot] commented 1 year ago

🤖

Hello,

Thank you for reaching out and providing detailed information about your issue. Based on the context you've provided and similar issues in the LangChain repository, there are a few things that might be causing the InvalidRequestError you're encountering.

Firstly, the token_max parameter in the load_summarize_chain function is used to limit the maximum number of tokens in the combined document. If the combined document would exceed the token_max, the reduce_documents_chain will need to reduce the content in some way to meet the token limit.

In your case, you've set token_max to 8196. However, the error message indicates that your messages resulted in 15473 tokens, which exceeds the model's maximum context length of 8192 tokens. This suggests that the documents you're trying to process are too large and are not being split into smaller parts as expected.

One possible solution is to increase the token_max value or ensure your documents are smaller. However, please note that no single document should be longer than the token_max value, as this would raise a ValueError.

Another possible issue could be the incorrect usage or configuration of the LLMChain and ReduceDocumentsChain instances in the MapReduceDocumentsChain class. You should check the configuration and usage of these instances to ensure they are correctly processing the documents and not exceeding the model's maximum context length.

Lastly, based on a similar issue in the LangChain repository, you might want to switch from OpenAI to ChatOpenAI. Instead of importing from langchain import OpenAI, you should import from langchain.chat_models import ChatOpenAI.

I hope this helps! If you're still encountering issues, please provide more information about your usage and configuration of the LLMChain and ReduceDocumentsChain instances, as well as the size and content of the documents you're trying to process. This will help us better understand and address your issue.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

dosubot[bot] commented 11 months ago

Hi, @omnific9! I'm Dosu, and I'm helping the LangChain team manage our backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, you were experiencing an issue with the load_summarization_chain function in 'map_reduce' mode. It seems that the chain is not breaking up the data as expected. In response, I suggested increasing the token_max value or ensuring the documents are smaller. I also mentioned checking the configuration and usage of the LLMChain and ReduceDocumentsChain instances.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!

metaswang commented 9 months ago

Hi I believe this issue will still confuse you if you keep max_tokens NOT none when init llm before run your map_reduce chain.

llm = ChatOpenAI(temperature=0, max_tokens=MAX_TOKENS)

langchain-ai / langchain