search

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
https://inference.readthedocs.io
Apache License 2.0
4.54k stars 356 forks source link

BUG: Accessing a wrong model can cause the entire Docker environment to crash. #1520

Open majestichou opened 3 months ago

majestichou commented 3 months ago

Describe the bug

I downloaded the SFR-Embedding-Mistralmodel weight file from huggingface. I put the model weights in the /home/hw/embedding_and_rerank_model directory and then started the container on machine which ip is xxxx , with the following command docker run -v /home/hw/embedding_and_rerank_model:/root/models -p 9998:9997 --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0 After that, I went to http://xxxx:9998/ui and registered the SFR-Embedding-Mistral model. And then I launched the model. Then I used langchain to access http://xxxx:9998/v1/embeddings/SFR-Embedding-Mistral. Everything is OK. The embedding model worked well. And then I accidentally wrote the model name wrong, the SFR-Embedding-Mistral name was misspelled as SFR-Embedding-MistraT. Then I used langchain to access http://xxxx:9998/v1/embeddings/SFR-Embedding-MistraT. Of course, I didn't have a normal access model. I then changed the name to the correct one (http://xxxx:9998/v1/embeddings/SFR-Embedding-Mistral)and reconnected the model. At this point, the unexpected happened: I couldn't access the model properly, even though I filled in the API correctly. I tried to access the embedding model with Dify, and it also failed.

Accessing a wrong model can cause the entire Docker environment to crash.

To Reproduce

  1. Xinference docker image--V0.11.0
  2. Dify 0.6.6
  3. langchain 0.1.13 langchain-community 0.0.31 langchain-core 0.1.40 langchain-openai 0.0.5 langchain-text-splitters 0.0.1 langchain-together 0.1.0

Expected behavior

Accessing a wrong model can not cause the entire Docker environment to crash. After the model name is corected, the model can still be invoked.

Additional context

As far as I can tell, the wrong model was started using the terminal command line, and then the docker mapped port does not exist.

qinxuye commented 3 months ago

Thanks for your report, we will address it ASAP.

github-actions[bot] commented 3 weeks ago

This issue is stale because it has been open for 7 days with no activity.