dosubot[bot]
commented
1 year ago ๐ค
Hello,
Thank you for raising this issue. The error you're encountering is due to the fact that the total number of tokens being sent to OpenAI is exceeding the model's maximum context length of 4097 tokens. This can happen when you're trying to send a large amount of data from a Graph DB at once.
There are a few ways to resolve this issue:
- Adjust the
max_tokensparameter: You can adjust themax_tokensparameter in theOpenAIinstance of theLLMChain. This parameter determines the maximum number of tokens to generate in the completion. However, the total token count of your prompt andmax_tokenscannot exceed the model's context length. Most models have a context length of 2048 tokens, but newer models, like the one you're using, support up to 4096 tokens. If your prompt is already using most of these tokens, the response might be cut short due to this limitation. Here's how you can adjust themax_tokensparameter:
from langchain import OpenAI, ConversationChain, LLMChain, PromptTemplate
from langchain.memory import ConversationBufferWindowMemory
template = """..."""
prompt = PromptTemplate(input_variables=["history", "human_input"], template=template)
chatgpt_chain = LLMChain(
llm=OpenAI(temperature=0, max_tokens=1000), # Adjust this value as needed
prompt=prompt,
verbose=True,
memory=ConversationBufferWindowMemory(k=2),
)Please adjust the max_tokens value according to your needs, keeping in mind the limitations of the model you're using. You can find more details about this in the issue #7981.
- Split your data into smaller chunks: If adjusting the
max_tokensparameter doesn't resolve the issue, you can try splitting your data into smaller chunks that fit within the token limit. You can use a method similar to theget_sub_promptsmethod in the LangChain codebase to split your data from the Graph DB into smaller chunks that fit within the token limit. Here is a simplified version of theget_sub_promptsmethod that you can use as a starting point:
def get_sub_prompts(
self,
params: Dict[str, Any],
prompts: List[str],
stop: Optional[List[str]] = None,
) -> List[List[str]]:
"""Get the sub prompts for llm call."""
if stop is not None:
if "stop" in params:
raise ValueError("`stop` found in both the input and default params.")
params["stop"] = stop
if params["max_tokens"] == -1:
if len(prompts) != 1:
raise ValueError(
"max_tokens set to -1 not supported for multiple inputs."
)
params["max_tokens"] = self.max_tokens_for_prompt(prompts[0])
sub_prompts = [
prompts[i : i + self.batch_size]
for i in range(0, len(prompts), self.batch_size)
]
return sub_promptsIn this method, prompts is the list of data from the Graph DB and self.batch_size is the maximum number of tokens that can be sent to OpenAI at once. The method returns a list of sub-prompts, each of which is a list of data that fits within the token limit.
You can adjust the self.batch_size to ensure that the total number of tokens in each sub-prompt does not exceed the maximum context length of 4097 tokens.
I hope this helps! If you have any further questions, please don't hesitate to ask.
Best, Dosu
Sources
- Agent does not use max_tokens parameter from ChatOpenAI
- conversational-retrieval-agent using AgentTokenBufferMemory : Still cannot prevent maximum content length errors
- RetrievalQAWithSourcesChain causing openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens
- libs/langchain/langchain/llms/openai.py
- libs/langchain/langchain/chat_models/openai.py
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a ๐ if this is helpful and ๐ if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
thehsansaeed
3mpt
bhatsudo
KUMAS503
Issue you'd like to raise.
How to resolve [This model's maximum context length is 4097 tokens] ?
If Graph DB have many nodes, can not send those tokens to OpenAI by one time.
Suggestion:
No response