dosubot[bot]
commented
1 year ago 🤖
Hello @sfc-gh-syang,
I'm here to assist you with your issue. I'm an AI developed to help users like you navigate and understand the LangChain repository.
Based on the information you've provided, it seems like you're encountering two main issues: timeout errors and invalid response object errors from the API when trying to replicate the basic LangChain summarization as per the website documentation using the meta-llama/Llama-2-7b-chat-hf model.
For the timeout errors, it's possible that the document you're trying to process is too large and is causing the request to time out. You mentioned that you've already tried increasing the request_timeout parameter, but it's still not working. It might be worth trying to further increase this parameter or split your document into smaller chunks before processing it.
For the invalid response object errors, it could be due to an issue with the API itself or the request being sent. The LangChain framework has error handling for various types of API errors including openai.error.APIError, openai.error.APIConnectionError, openai.error.RateLimitError, and openai.error.ServiceUnavailableError. If you're seeing these errors, it could indicate an issue with the API or your usage of it.
Regarding the model name, the meta-llama/Llama-2-7b-chat-hf does not seem to be a valid model for the JinaChat API. The model name should be set to jinachat as per the _invocation_params property in the JinaChat class.
Here is the relevant code for the above points:
API Key and Model Name:
@property
def _invocation_params(self) -> Mapping[str, Any]:
"""Get the parameters used to invoke the model."""
jinachat_creds: Dict[str, Any] = {
"api_key": self.jinachat_api_key,
"api_base": "https://api.chat.jina.ai/v1",
"model": "jinachat",
}
return {**jinachat_creds, **self._default_params}Timeouts and Retries:
def _create_retry_decorator(self) -> Callable[[Any], Any]:
import openai
min_seconds = 1
max_seconds = 60
# Wait 2^x * 1 second between each retry starting with
# 4 seconds, then up to 10 seconds, then 10 seconds afterwards
return retry(
reraise=True,
stop=stop_after_attempt(self.max_retries),
wait=wait_exponential(multiplier=1, min=min_seconds, max=max_seconds),
retry=(
retry_if_exception_type(openai.error.Timeout)
| retry_if_exception_type(openai.error.APIError)
| retry_if_exception_type(openai.error.APIConnectionError)
| retry_if_exception_type(openai.error.RateLimitError)
| retry_if_exception_type(openai.error.ServiceUnavailableError)
),
before_sleep=before_sleep_log(logger, logging.WARNING),
)API Errors:
retry=(
retry_if_exception_type(openai.error.Timeout)
| retry_if_exception_type(openai.error.APIError)
| retry_if_exception_type(openai.error.APIConnectionError)
| retry_if_exception_type(openai.error.RateLimitError)
| retry_if_exception_type(openai.error.ServiceUnavailableError)
),Sources:
I hope this helps! If you have any other questions or need further clarification, feel free to ask.
Best, Dosu
Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
sfc-gh-syang
System Info
Trying to replicate the basic langchain summarization type = Stuff usage as on website documentation, but it is not able to process long doc.
Test model used: meta-llama/Llama-2-7b-chat-hf
Test result: tried its original demo code, it did not work, and gives two types of error alternatively after dozens of trials: Error 1: timeout error (I've referred to the related langchain timeout discussion here to increase request_timeout=600/3600 secs, but still get the timeout error. checked my host is on)
Error 2: Invalid response object from API Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.._completion_with_retry in 4.0 seconds as it raised APIError: Invalid response object from API: '{"object":"error","message":"Expecting value: line 1 column 1 (char 0)","code":50001}' (HTTP response code was 400).
Who can help?
Hi @hwchase17 , include this issue with you, as I am using llm chat wrappers like
from langchain.llms.openai import OpenAI,from langchain.chains.combine_documents.stuff import StuffDocumentsChainandfrom langchain.chains.summarize import load_summarize_chainetc and error comes from there Hi @agola11 , include you w/ this issue re callbacks, as in the errors mentioned above, the Error2: goes to -> --> stuff_chainself.combine_docs(docs, callbacks=_run_manager.get_child(), other_keys)--> 171 return self.llm_chain.predict(callbacks=callbacks, inputs) --> 257 return self(kwargs, callbacks=callbacks)[self.output_key] --> 310 raise e --> 304 self._call(inputs, run_manager=run_manager) ---> 93 response = self.generate([inputs], run_manager=run_manager)
Information
Related Components
Reproduction
To get something out from langchain summarization, I downsized the model from llama2-7b to "Sheared-LLaMA-1.3B" and replaced the original example's doc with a shorter doc "". I can see from langchain w/ verbose started w/ > Entering new StuffDocumentsChain chain..., > Entering new LLMChain chain... w/ print out the original full doc, but it returns'> Finished chain. returns "Got any creative ideas for a 10 year old’s birthday?"'
But I got strange result as below:
(my local host heart beats well)
Try the same code with a small piece of business-related conversation doc(~ 1k words) w/ StuffDocumentsChain returns something relevant, but load_summarize_chain only gives
More problem comes, when I double the size on my test doc(4k words, definitely have to be>4096 tokens, that's the point of using summarization), it returns Error 1 and Error 2 mentioned at the beginning, same as attempting replication langchain documentation, so get the trouble it is not able to process long doc gain.
Expected behavior
returns some relavent result. though understand it does not have to be good, as I am using toy model 1.3Brelevant