vllm方式多卡加载模型卡住，GPU占用率100%

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

Apache License 2.0

4.94k stars 391 forks source link

System Info / 系統信息

cuda版本12.3，卡为4090，python10，本地运行非docker方式。

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

[ ] docker / docker
[X] pip install / 通过 pip install 安装
[ ] installation from source / 从源码安装

Version info / 版本信息

0.15.0和0.15.2均存在相同问题

The command used to start Xinference / 用以启动 xinference 的命令

XINFERENCE_HOME=/data/program/xinference XINFERENCE_MODEL_SRC=modelscope xinference-local --host 0.0.0.0 --port 9997

Reproduction / 复现过程

双卡加载模型就会出现。 ![Uploading 微信图片_20240924195713.jpg…]() ![Uploading 微信图片_20240924195729.png…]()

Expected behavior / 期待表现

期望能找到有效的解决办法。

xorbitsai / inference