Error while using RetrievalQA chain type of Langchain for vector retrieval using FastChat LLM model which is hosted on Endpoint(GPU machine).

[Smitraj007]

Our LLM model is on endpoint :

from langchain.indexes import VectorstoreIndexCreator

os.environ["OPENAI_API_BASE"] = "my_endpoint_ip" os.environ["OPENAI_API_KEY"] = "empty" embedding = OpenAIEmbeddings(model="text-embedding-ada-002") local_llm = OpenAI(model="gpt-3.5-turbo")

So while performing below retrieval task we are not able to fetch llm model from endpoint. qa_chain = RetrievalQA.from_chain_type(llm=local_llm, chain_type="stuff", retriever=retriever, return_source_documents=True)

ERROR : ConnectionResetError

ConnectionResetError Traceback (most recent call last) File ~/SageMaker/AmazonSageMaker-IAG/virtual_environment/lib/python3.10/site-packages/urllib3/connectionpool.py:714, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 713 # Make the request on the httplib connection object. --> 714 httplib_response = self._make_request( 715 conn, 716 method, 717 url, 718 timeout=timeout_obj, 719 body=body, 720 headers=headers, 721 chunked=chunked, 722 ) ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

[KenwayCJZX]

same error

[Minxiangliu]

You can refer to this. https://github.com/lm-sys/FastChat/issues/2303#issuecomment-2151829969