多卡启动自定义模型时，报Remote server unixsocket错误

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

https://inference.readthedocs.io

Apache License 2.0

4.55k stars 357 forks source link

多卡启动自定义模型时，报Remote server unixsocket错误 #1474

Closed mrkingsun closed 2 weeks ago

mrkingsun commented 3 months ago

两张卡，第一张（12G/32G）,第二张（1G/32G）。模型是internlm2-chat-7b。

只使用第二张卡加载，显存占用大概30G，可以正常启动；
因为占用快满了，想利用下第一张卡，在注册页面设置了gpu-index为0，1。启动时就报错Remote server unixsocket

codingl2k1 commented 3 months ago

有没有服务端的错误信息？

mrkingsun commented 3 months ago

解决了，检索了一下是docker内存不足，在run的时候加上--shm-size=10.24gb即可

github-actions[bot] commented 2 weeks ago

This issue is stale because it has been open for 7 days with no activity.

github-actions[bot] commented 2 weeks ago

This issue was closed because it has been inactive for 5 days since being marked as stale.