search

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
https://inference.readthedocs.io
Apache License 2.0
5.12k stars 413 forks source link

显示启动模型失败，load失败 #1884

Open xunuo2345 opened 3 months ago

xunuo2345 commented 3 months ago

System Info / 系統信息

11

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

Version info / 版本信息

xinference, version 0.13.1

The command used to start Xinference / 用以启动 xinference 的命令

docker run --shm-size=4g --name inferencex -v /apps/users/xrp/model:/opt/inference/model/ -e XINFERENCE_HOME=/opt/inference/model/ -p 9998:9997 --gpus all -d registry.cn-hangzhou.aliyuncs.com/xprobe_xinference/xinference:latest xinference-local -H 0.0.0.0 --log-level debug

Reproduction / 复现过程

docker run --shm-size=4g --name inferencex -v /apps/users/xrp/model:/opt/inference/model/ -e XINFERENCE_HOME=/opt/inference/model/ -p 9998:9997 --gpus all -d registry.cn-hangzhou.aliyuncs.com/xprobe_xinference/xinference:latest xinference-local -H 0.0.0.0 --log-level debug 然后容器中model目录中已经存在deepseek-llm-7b-chat，模型，但是使用xinference launch --model-engine vllm -u deepseek-llm-7b-chat -n deepseek-llm-7b-chat -s 7 -f pytorch --gpu_memory_utilization 0.8命令启动模型直接报错失败；这个命令是默认下载模型权重到model/cache目录吗，不能直接加载model目录下的模型吗？ 111 111

Expected behavior / 期待表现

希望能成功加载模型

xunuo2345 commented 3 months ago

image 切换命令还是失败，因为网络连接原因吗？

qinxuye commented 3 months ago

这个报错一般是网络问题。

github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 7 days with no activity.

github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 7 days with no activity.