search

opea-project / GenAIExamples

Generative AI Examples is a collection of GenAI examples such as ChatQnA, Copilot, which illustrate the pipeline capabilities of the Open Platform for Enterprise AI (OPEA) project.
https://opea.dev
Apache License 2.0
230 stars 148 forks source link

[k8s ChatQnA] retriever_deploy deployment failed due to "http://tei-embedding-svc.default.svc.cluster.local:6006/**" failed to access #313

Closed moting9 closed 3 months ago

moting9 commented 3 months ago

Dear experts: I followed these steps to run k8s chatQnA example. https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/docker/xeon/README.md https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/kubernetes/manifests/README.md

retriever-deploy deployment shows " requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://tei-embedding-svc.default.svc.cluster.local:6006/"

I have added proxy config below in "qna_configmap_xeon.yaml" and made sure tei-embedding, tei-reranking, tgi-service successfully downloaded 3 models. _http_proxy: http://proxy.xxx.xxxxx.com:xxx https_proxy: http://proxy.xxx.xxxxx.com:xxx noproxy: 10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,xxxxx.com

I suspect network or proxy config relevant issue. any suggestions are appreciated. Below are logs.

NAMESPACE NAME READY STATUS RESTARTS AGE default chaqna-xeon-backend-server-deploy-79cbbb7b-4q9t6 1/1 Running 0 58m default embedding-deploy-7bb6df68f5-4k8kr 1/1 Running 0 58m default llm-deploy-5c85678ddd-sjbsn 1/1 Running 0 58m default redis-vector-db-6db798f98d-56vqz 1/1 Running 0 58m default reranking-deploy-87c5bf6cd-r4cph 1/1 Running 0 58m default retriever-deploy-6bb494f5bd-9ggnd 0/1 CrashLoopBackOff 14 ( ago) 58m default tei-embedding-service-deploy-78fbbcbf67-xvx9v 1/1 Running 0 58m default tei-reranking-service-deploy-6cdb544d49-fmfj8 1/1 Running 0 58m default tgi-service-deploy-78f488ff9f-tlmg8 1/1 Running 0 58m

kubectl get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR chaqna-xeon-backend-server-svc NodePort 10.103.16.9 8888:32441/TCP 69m app=chaqna-xeon-backend-server-deploy embedding-svc ClusterIP 10.104.135.74 6000/TCP 69m app=embedding-deploy kubernetes ClusterIP 10.96.0.1 443/TCP 6d llm-svc ClusterIP 10.104.216.23 9000/TCP 69m app=llm-deploy redis-vector-db ClusterIP 10.103.250.80 6379/TCP,8001/TCP 69m app=redis-vector-db reranking-svc ClusterIP 10.103.2.220 8000/TCP 69m app=reranking-deploy retriever-svc ClusterIP 10.110.168.101 7000/TCP 69m app=retriever-deploy tei-embedding-svc ClusterIP 10.103.168.217 6006/TCP 69m app=tei-embedding-service-deploy tei-reranking-svc ClusterIP 10.107.35.174 8808/TCP 69m app=tei-reranking-service-deploy tgi-svc ClusterIP 10.104.172.25 9009/TCP 69m app=tgi-service-deploy

kubectl logs retriever-deploy-6bb494f5bd-9ggnd Neither CUDA nor MPS are available - defaulting to CPU. Note: This module is much faster with a GPU. Downloading detection model, please wait. This may take several minutes depending upon your network connection. Parsing 10k filing doc for NIKE data/nke-10k-2023.pdf Progress: |███████████████████████████████▒Downloading recognition model, please wait. This may take several minutes depending upon your network connection. Progress: |██████████████████████████████████████████████████| 100.0% CompleteDone preprocessing. Created 270 chunks of the original pdf Traceback (most recent call last): File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status response.raise_for_status() File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://tei-embedding-svc.default.svc.cluster.local:6006/

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/user/comps/retrievers/langchain/redis/ingest.py", line 99, in ingest_documents() File "/home/user/comps/retrievers/langchain/redis/ingest.py", line 88, in ingestdocuments = Redis.from_texts( ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 485, in fromtexts instance, = cls.from_texts_return_keys( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 418, in from_texts_return_keys keys = instance.add_texts(texts, metadatas, keys=keys) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 705, in add_texts embeddings = embeddings or self._embeddings.embed_documents(list(texts)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain_community/embeddings/huggingface_hub.py", line 95, in embed_documents responses = self.client.post( ^^^^^^^^^^^^^^^^^ File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/inference/_client.py", line 273, in post hf_raise_for_status(response) File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status raise HfHubHTTPError(str(e), response=response) from e huggingface_hub.utils._errors.HfHubHTTPError: 504 Server Error: Gateway Timeout for url: http://tei-embedding-svc.default.svc.cluster.local:6006/ /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:149: UserWarning: Field "model_name_orpath" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( [2024-06-24 05:15:52,520] [ INFO] - CORS is enabled. [2024-06-24 05:15:52,521] [ INFO] - Setting up HTTP server [2024-06-24 05:15:52,521] [ INFO] - Uvicorn server setup on port 7000 INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:7000 (Press CTRL+C to quit) [2024-06-24 05:15:52,533] [ INFO] - HTTP server setup successful Traceback (most recent call last): File "/home/user/comps/retrievers/langchain/redis/retriever_redis.py", line 71, in vector_db = Redis.from_existing_index( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 562, in from_existing_index raise ValueError( ValueError: Redis failed to connect: Index rag-redis does not exist.

lianhao commented 3 months ago

Please add ".default.svc.cluster.local" to the no_proxy config too

moting9 commented 3 months ago

Please add ".default.svc.cluster.local" to the no_proxy config too

Thanks very much! It fixed this problem.

I met with a new problem "Internal Server Error" when accessing the RAG service.: curl http://10.97.233.140:8888/v1/chatqna -H "Content-Type: application/json" -d '{ "messages": "What is the revenue of Nike in 2023?" }' Internal Server Error

My GenAIExamples repo version is: _commit a0b94b540180ddba7892573b2d9ce8b0eb16b403 (HEAD -> main, origin/main, origin/HEAD) Author: Ying Chun Guo yingchun.guo@intel.com Date: Wed Jun 19 15:24:27 2024 +0800

Add image build job in docker compose e2e xeon test in CI (#304)_

My GenAIComp version is _commit e77190cc373c0d8a4d826ae1ac2abe5623467513 (HEAD -> main, origin/main, origin/HEAD) Author: lvliang-intel liang1.lv@intel.com Date: Thu Jun 20 09:10:58 2024 +0800

Fix llm tgi test issue (#217)

* add preload env in run.sh_

kubectl logs chaqna-xeon-backend-server-deploy-79cbbb7b-vb9lx /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:160: UserWarning: Field "model_name_orpath" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( [2024-06-25 01:21:22,788] [ INFO] - CORS is enabled. [2024-06-25 01:21:22,788] [ INFO] - Setting up HTTP server [2024-06-25 01:21:22,789] [ INFO] - Uvicorn server setup on port 8888 INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit) [2024-06-25 01:21:22,825] [ INFO] - HTTP server setup successful 504 INFO: 10.244.0.0:33203 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 121, in handle_request result_dict = await self.megaservice.schedule(initial_inputs={"text": prompt}, llm_parameters=parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 49, in schedule response, node = await done_task ^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 98, in execute return await response.json(), cur_node ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1176, in json raise ContentTypeError( aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('http://embedding-svc:6000/v1/embeddings') 504 INFO: 10.244.0.0:6929 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 121, in handle_request result_dict = await self.megaservice.schedule(initial_inputs={"text": prompt}, llm_parameters=parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 49, in schedule response, node = await done_task ^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 98, in execute return await response.json(), cur_node ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1176, in json raise ContentTypeError( aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('http://embedding-svc:6000/v1/embeddings') 504 INFO: 10.244.0.0:52013 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 121, in handle_request result_dict = await self.megaservice.schedule(initial_inputs={"text": prompt}, llm_parameters=parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 49, in schedule response, node = await done_task ^^^^^^^^^^^^^^^ File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 98, in execute return await response.json(), cur_node ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1176, in json raise ContentTypeError( aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('http://embedding-svc:6000/v1/embeddings')

lianhao commented 3 months ago

please add those XXX_SERVICE_HOST_IP's values into no_proxy too, e.g. embedding-svc,retriever-svc, etc.

moting9 commented 3 months ago

Thanks! the solution works.