get asyncio.exceptions.CancelledError error

[DFanny-5]

System Info

Using CPU and kubernetes CMD: infinity_emb v1 --model-name-or-path /mnt/model-storage/ms-marco-MiniLM-L-12-v2 --port 8452 --batch-size 32

Information

• [ ] Docker
• [X] The CLI directly via pip

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

Run the command on local computer works, but run on a kubernetes shows the following error INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8452 (Press CTRL+C to quit) ERROR: Traceback (most recent call last): File "/usr/local/lib/python3.11/contextlib.py", line 222, in aexit await self.gen.athrow(typ, value, traceback) File "/usr/local/lib/python3.11/site-packages/infinity_emb/infinity_server.py", line 82, in lifespan yield File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 741, in lifespan await receive() File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get await getter asyncio.exceptions.CancelledError

Expected behavior

I see INFO: Uvicorn running on http://0.0.0.0:8452 (Press CTRL+C to quit) so the server seems starts correctly? But the asyncio.exceptions.CancelledError cause the pod to crash immediately

[michaelfeil]

I think the server did not start correctly. It error-ed during startup. Can you try running ms-marco from huggingface on your k8s cluster?

[DFanny-5]

I think the server did not start correctly. It error-ed during startup. Can you try running ms-marco from huggingface on your k8s cluster?

Thanks for the quick reply. Here is the process I tried: I get the model by: git clone https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2

Then I run infinity_emb v1 --model-name-or-path /mnt/model-storage/ms-marco-MiniLM-L-12-v2 --port 8452 locally, it it seems success. but after I zip the ms-marco-MiniLM-L-12-v2 folder and upload to the persist volume and unzip it on the k8s, run the same command above get the error I mentioned above:

Run the command on local computer works, but run on a kubernetes shows the following error INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8452/ (Press CTRL+C to quit) ERROR: Traceback (most recent call last): File "/usr/local/lib/python3.11/contextlib.py", line 222, in aexit await self.gen.athrow(typ, value, traceback) File "/usr/local/lib/python3.11/site-packages/infinity_emb/infinity_server.py", line 82, in lifespan yield File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 741, in lifespan await receive() File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive return await self.receive_queue.get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get await getter asyncio.exceptions.CancelledError

[DFanny-5]

I think the uploading is not wrong because the same process works for the ms-marco-TinyBERT-L-2 model

[michaelfeil]

The message is an ungraceful exit. Potentially, because a new replica is rolled out, or not enough resources, or model is saved incorrectly.