Open arun-gupta opened 1 month ago
@arun-gupta,
Could you please share the Docker logs for the TEI embedding serving container? The trace indicates that the embedding microservice is unable to connect to the TEI serving.
requests.exceptions.ConnectionError: (MaxRetryError("HTTPConnectionPool(host='10.0.0.233', port=6006): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7911a3e91850>: Failed to establish a new connection: [Errno 113] No route to host'))"), '(Request ID: 84ec5480-daab-48e1-a7cf-fe52d8644f1d)')
I used the stock Docker Compose file, so it should be using the latest
tag.
@arun-gupta In order to debug this network issue, could you get logs from below commands?
@arun-gupta is the network issue still there in Oracle Cloud? if needed, we could also debug in Oracle Cloud.
@louie-tsai ssh to OCI instance has been timing out for a few weeks now. I tried again and still the same status. Will try creating a different account and then debug.
Seems like the container name is changed from embedding-tei-server
to tei-embedding-server
. Here are the logs now:
ubuntu@opea-demo:~$ sudo docker container logs tei-embedding-server
2024-10-14T21:07:14.858232Z INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "BAA*/***-****-**-v1.5", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: true, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "646b7204f7d4", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
2024-10-14T21:07:14.858411Z INFO hf_hub: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hf-hub-0.3.2/src/lib.rs:55: Token file not found "/root/.cache/huggingface/token"
2024-10-14T21:07:14.920793Z INFO download_pool_config: text_embeddings_core::download: core/src/download.rs:38: Downloading `1_Pooling/config.json`
2024-10-14T21:07:15.367035Z INFO download_new_st_config: text_embeddings_core::download: core/src/download.rs:62: Downloading `config_sentence_transformers.json`
2024-10-14T21:07:15.549873Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:21: Starting download
2024-10-14T21:07:15.549901Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:23: Downloading `config.json`
2024-10-14T21:07:15.745925Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:26: Downloading `tokenizer.json`
2024-10-14T21:07:16.123502Z INFO download_artifacts: text_embeddings_backend: backends/src/lib.rs:313: Downloading `model.onnx`
2024-10-14T21:07:16.220576Z WARN download_artifacts: text_embeddings_backend: backends/src/lib.rs:317: Could not download `model.onnx`: request error: HTTP status client error (404 Not Found) for url (https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/model.onnx)
2024-10-14T21:07:16.220604Z INFO download_artifacts: text_embeddings_backend: backends/src/lib.rs:318: Downloading `onnx/model.onnx`
2024-10-14T21:07:18.125150Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:32: Model artifacts downloaded in 2.575276248s
2024-10-14T21:07:18.154670Z INFO text_embeddings_router: router/src/lib.rs:199: Maximum number of tokens per request: 512
2024-10-14T21:07:18.158270Z INFO text_embeddings_core::tokenization: core/src/tokenization.rs:28: Starting 64 tokenization workers
2024-10-14T21:07:18.549449Z INFO text_embeddings_router: router/src/lib.rs:241: Starting model backend
2024-10-14T21:07:19.827070Z WARN text_embeddings_router: router/src/lib.rs:267: Backend does not support a batch size > 8
2024-10-14T21:07:19.827108Z WARN text_embeddings_router: router/src/lib.rs:268: forcing `max_batch_requests=8`
2024-10-14T21:07:19.827395Z WARN text_embeddings_router: router/src/lib.rs:319: Invalid hostname, defaulting to 0.0.0.0
2024-10-14T21:07:19.831059Z INFO text_embeddings_router::http::server: router/src/http/server.rs:1778: Starting HTTP server: 0.0.0.0:80
2024-10-14T21:07:19.831083Z INFO text_embeddings_router::http::server: router/src/http/server.rs:1779: Ready
2024-10-14T21:34:17.720707Z INFO embed{total_time="49.431255ms" tokenization_time="854.099µs" queue_time="1.20308ms" inference_time="47.03138ms"}: text_embeddings_router::http::server: router/src/http/server.rs:706: Success
Invoking the following command gives an error:
ubuntu@opea-demo:~$ curl http://${host_ip}:8888/v1/chatqna \
-H "Content-Type: application/json" \
-d '{
"messages": "What is the revenue of Nike in 2023?"
}'
Internal Server Error
Here is the list of containers:
ubuntu@opea-demo:~$ sudo docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38c6f8885d63 opea/chatqna-ui:latest "docker-entrypoint.s…" 45 minutes ago Up 45 minutes 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp chatqna-xeon-ui-server
f4a2073d9b27 opea/chatqna:latest "python chatqna.py" 45 minutes ago Up 45 minutes 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp chatqna-xeon-backend-server
16d874753ad2 opea/dataprep-redis:latest "python prepare_doc_…" 45 minutes ago Up 45 minutes 0.0.0.0:6007->6007/tcp, :::6007->6007/tcp dataprep-redis-server
1df85d99c0f1 opea/retriever-redis:latest "python retriever_re…" 45 minutes ago Up 4 minutes 0.0.0.0:7000->7000/tcp, :::7000->7000/tcp retriever-redis-server
350fc041dceb ghcr.io/huggingface/text-generation-inference:sha-e4201f4-intel-cpu "text-generation-lau…" 45 minutes ago Up 45 minutes 0.0.0.0:9009->80/tcp, [::]:9009->80/tcp tgi-service
8bd8fb156225 redis/redis-stack:7.2.0-v9 "/entrypoint.sh" 45 minutes ago Up 45 minutes 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp, 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp redis-vector-db
4a36cc2d29d6 ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 "text-embeddings-rou…" 45 minutes ago Up 45 minutes 0.0.0.0:8808->80/tcp, [::]:8808->80/tcp tei-reranking-server
646b7204f7d4 ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 "text-embeddings-rou…" 45 minutes ago Up 45 minutes 0.0.0.0:6006->80/tcp, [::]:6006->80/tcp tei-embedding-server
Here is the next error: https://github.com/opea-project/GenAIExamples/issues/949
@arun-gupta we might need to have Oracle Cloud environment to check this issue. please share us how to connect to Oracle cloud offline.
thanks
Priority
Undecided
OS type
Ubuntu
Hardware type
Xeon-ICX
Installation method
Deploy method
Running nodes
Single Node
What's the version?
0.9
Description
Embedding microservice fails with Ubuntu on Oracle Cloud
Reproduce steps
https://gist.github.com/arun-gupta/b4c13de906e5fe66024018ba48e3f6c3
Raw log