mkbhanda
commented
1 month ago @kevinintel and @lvliang-intel any chance we can revert to TGI 1.4 till the Hugging face bug is resolved. This example otherwise fails.
Open wsfowler opened 1 month ago
mkbhanda
commented
1 month ago @kevinintel and @lvliang-intel any chance we can revert to TGI 1.4 till the Hugging face bug is resolved. This example otherwise fails.
ashahba
commented
1 month ago Thanks @mkbhanda
I'm testing with 1.5 today and I'm not seeing this issue:
$ docker compose -f compose.yaml logs reranking -f
...
reranking-tei-xeon-server | [2024-07-31 23:49:43,238] [ INFO] - CORS is enabled.
reranking-tei-xeon-server | [2024-07-31 23:49:43,239] [ INFO] - Setting up HTTP server
reranking-tei-xeon-server | [2024-07-31 23:49:43,239] [ INFO] - Uvicorn server setup on port 8000
reranking-tei-xeon-server | INFO: Waiting for application startup.
reranking-tei-xeon-server | INFO: Application startup complete.
reranking-tei-xeon-server | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
reranking-tei-xeon-server | [2024-07-31 23:49:43,255] [ INFO] - HTTP server setup successfuland:
$ docker compose -f compose.yaml ps | grep reranking
reranking-tei-xeon-server opea/reranking-tei:latest "python reranking_te…" reranking 10 minutes ago Up 9 minutes 0.0.0.0:8000->8000/tcp
tei-reranking-server ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 "text-embeddings-rou…" tei-reranking-service 10 minutes ago Up 9 minutes 0.0.0.0:8808->80/tcpWould you please try again?
Thanks.
ashahba
commented
1 month ago Is it possible that v0.7 containers are being mixed with latest code?
wsfowler
commented
1 month ago I'll test again, but the only mixing of code happening is that I'm using the latest compose.yaml file with the v0.7 containers and I don't see major differences for the reranking service, if I look at the history I don't see any major changes beyond the TEI version since v0.7.
I retested this morning and I'm getting the same error:
2024-08-01T13:00:54.638796Z INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "BAA*/***-********-**rge", revision: None, tokenization_workers: None, dtype: None, pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: true, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "e6c5ab9b2197", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
2024-08-01T13:00:54.638872Z INFO hf_hub: /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hf-hub-0.3.2/src/lib.rs:55: Token file not found "/root/.cache/huggingface/token"
2024-08-01T13:00:54.681575Z INFO download_pool_config: text_embeddings_core::download: core/src/download.rs:38: Downloading `1_Pooling/config.json`
2024-08-01T13:00:54.863118Z INFO download_new_st_config: text_embeddings_core::download: core/src/download.rs:62: Downloading `config_sentence_transformers.json`
2024-08-01T13:00:54.878546Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:21: Starting download
2024-08-01T13:00:54.878558Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:23: Downloading `config.json`
2024-08-01T13:00:54.960966Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:26: Downloading `tokenizer.json`
2024-08-01T13:00:55.108601Z INFO download_artifacts: text_embeddings_backend: backends/src/lib.rs:313: Downloading `model.onnx`
2024-08-01T13:00:55.126338Z WARN download_artifacts: text_embeddings_backend: backends/src/lib.rs:317: Could not download `model.onnx`: request error: HTTP status client error (404 Not Found) for url (https://huggingface.co/BAAI/bge-reranker-large/resolve/main/model.onnx)
2024-08-01T13:00:55.126364Z INFO download_artifacts: text_embeddings_backend: backends/src/lib.rs:318: Downloading `onnx/model.onnx`
2024-08-01T13:00:55.166679Z INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:32: Model artifacts downloaded in 288.133574ms
2024-08-01T13:00:55.783229Z WARN text_embeddings_router: router/src/lib.rs:195: Could not find a Sentence Transformers config
2024-08-01T13:00:55.783257Z INFO text_embeddings_router: router/src/lib.rs:199: Maximum number of tokens per request: 512
2024-08-01T13:00:55.784582Z INFO text_embeddings_core::tokenization: core/src/tokenization.rs:28: Starting 32 tokenization workers
2024-08-01T13:01:02.836291Z INFO text_embeddings_router: router/src/lib.rs:241: Starting model backend
Error: Could not create backend
Caused by:
Could not start backend: Failed to create ONNX Runtime session: Deserialize tensor roberta.encoder.layer.13.intermediate.dense.bias failed.GetFileLength for /data/models--BAAI--bge-reranker-large/snapshots/55611d7bca2a7133960a6d3b71e083071bbfc312/onnx/model.onnx_data failed:Invalid fd was supplied: -1Just tested with the v0.7 release of the repo and it takes the version of TEI back to 1.2, so it seems like things are working. Let me see if I build the v0.8 containers if things work. Also, with the v0.7 release the conversational UI is deployed but not working.
wsfowler
commented
1 month ago Just tested with the v0.8 release of the repo, building the containers and I'm running into the original problem with the TEI 1.5 container for reranking.
root@ip-172-31-19-70:/opt/GenAIExamples/ChatQnA/docker/xeon# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e233647b23ef opea/chatqna-ui:latest "docker-entrypoint.s…" 36 seconds ago Up 34 seconds 0.0.0.0:5173->5173/tcp, :::5173->5173/tcp chatqna-xeon-ui-server
e253767c1551 opea/chatqna:latest "python chatqna.py" 36 seconds ago Up 35 seconds 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp chatqna-xeon-backend-server
ef38c496621f opea/embedding-tei:latest "python embedding_te…" 36 seconds ago Up 35 seconds 0.0.0.0:6000->6000/tcp, :::6000->6000/tcp embedding-tei-server
6b2488c0716c opea/dataprep-redis:latest "python prepare_doc_…" 36 seconds ago Up 35 seconds 0.0.0.0:6007-6009->6007-6009/tcp, :::6007-6009->6007-6009/tcp dataprep-redis-server
d51236335642 opea/retriever-redis:latest "python retriever_re…" 36 seconds ago Up 35 seconds 0.0.0.0:7000->7000/tcp, :::7000->7000/tcp retriever-redis-server
335c1b57fde2 opea/llm-tgi:latest "bash entrypoint.sh" 36 seconds ago Up 35 seconds 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp llm-tgi-server
257f047b5fea opea/reranking-tei:latest "python reranking_te…" 36 seconds ago Up 35 seconds 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp reranking-tei-xeon-server
6ad87af9aab0 ghcr.io/huggingface/text-generation-inference:2.1.0 "/tgi-entrypoint.sh …" 36 seconds ago Up 36 seconds 0.0.0.0:9009->80/tcp, :::9009->80/tcp tgi-service
2e22f0e71955 ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 "text-embeddings-rou…" 36 seconds ago Up 36 seconds 0.0.0.0:6006->80/tcp, :::6006->80/tcp tei-embedding-server
5b16201f676b ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 "text-embeddings-rou…" 36 seconds ago Exited (1) 27 seconds ago tei-reranking-server
ad47210bbf87 redis/redis-stack:7.2.0-v9 "/entrypoint.sh" 36 seconds ago Up 36 seconds 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp, 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp redis-vector-dbGenAIExamples Repo:
root@ip-172-31-19-70:/opt/GenAIExamples/ChatQnA/docker/xeon# git status
HEAD detached at a2437e8GenAIComps Repo
root@ip-172-31-19-70:/opt/GenAIComps# git status
HEAD detached at f37ed79
chensuyue
commented
1 month ago I did some local test with v0.8 and didn't found any issue. Did you build local image with --no-cache? Or you can do docker system prune --all to clean up the caches before build the image.
mkbhanda
commented
1 month ago @wsfowler will consider building a versioned Terraform script for ChatQnA. Indeed there was some version mixup -- the github clone of the V0.8 and beyond) latest and the docker image repository containing V 0.7. Given we are not releasing any fixes till v0.9, that seems the more practical approach so we have working example sooner than later. Note that @wsfowler spotted an issue filed on HF site for TEI 1.5
wsfowler
commented
1 month ago I'll run some more tests using v0.8 with the commands you mention.
@mkbhanda One issue I will run into creating the versioned Terraform/Ansible script is that the docker compose files today specify latest for the OPEA containers. For instance if I switch the repo to the v0.7 release, after the v0.8 containers are published, then start using the OPEA docker compose file then it will attempt to pull down the latest containers. So some thought may need to be given to using specific versions for the OPEA containers in the docker compose files and others.
wsfowler
commented
1 month ago @chensuyue Just did a clean build of the v0.8 containers again and I'm still seeing the same error. I did the docker system prune --all and I made sure to build the containers using the --no-cache command. I used the build commands straight from the readme.
mkbhanda
commented
1 month ago I'll run some more tests using v0.8 with the commands you mention.
@mkbhanda One issue I will run into creating the versioned Terraform/Ansible script is that the docker compose files today specify
latestfor the OPEA containers. For instance if I switch the repo to thev0.7release, after thev0.8containers are published, then start using the OPEA docker compose file then it will attempt to pull down the latest containers. So some thought may need to be given to using specific versions for the OPEA containers in the docker compose files and others.
@kevinintel and @chensuyue - our use of latest tagged images in docker-compose and manifests is nice, but it causes issues when we have bugs in what is released, as in the case of ChatQnA thanks to a Hugging Face image issue. What if we could send in an argument to docker-compose that lets us pull images with a particular tag? Does such a feature exist?
Alternately, could we with each release expect images and their use to be in sync, like OPEA V0.9 uses V0.9 images of the micro components explicitly (not latest)? Then we could at least use a tagged version of all files, and when we git clone we can pull using a tag.
chensuyue
commented
1 month ago I'll run some more tests using v0.8 with the commands you mention. @mkbhanda One issue I will run into creating the versioned Terraform/Ansible script is that the docker compose files today specify
latestfor the OPEA containers. For instance if I switch the repo to thev0.7release, after thev0.8containers are published, then start using the OPEA docker compose file then it will attempt to pull down the latest containers. So some thought may need to be given to using specific versions for the OPEA containers in the docker compose files and others.@kevinintel and @chensuyue - our use of latest tagged images in docker-compose and manifests is nice, but it causes issues when we have bugs in what is released, as in the case of ChatQnA thanks to a Hugging Face image issue. What if we could send in an argument to docker-compose that lets us pull images with a particular tag? Does such a feature exist?
Alternately, could we with each release expect images and their use to be in sync, like OPEA V0.9 uses V0.9 images of the micro components explicitly (not latest)? Then we could at least use a tagged version of all files, and when we git clone we can pull using a tag.
Yes, we have proposed such solution to freeze all the deps version and image tag in the release branch form v0.9.
I was updating my Terraform/Ansible recipe this morning and I ran into this error while trying to start the example using the provided
compose.yamlfile.The reranking container then exits.
I found the following issue on the TEI repo: https://github.com/huggingface/text-embeddings-inference/issues/341
If I switch the reranking service to use 1.4, then it starts correctly and the example works.