search

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
https://inference.readthedocs.io
Apache License 2.0
4.9k stars 388 forks source link

Are you considering supporting NPU and Shengteng series chips? #2234

Open lvshuaigg opened 2 weeks ago

lvshuaigg commented 2 weeks ago

System Info / 系統信息

Operating system version/architecture: Linux version 4.19.90-23.48.v2101.ky10.aarch64 NPU: 8 cards 910 ProB 32G Python: 3.9 NPU driver: 23.0.0 NPU firmware: 7.1.0.3.220 CANN toolkit: 7.0.0.RC1

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

Version info / 版本信息

v0.13.1

The command used to start Xinference / 用以启动 xinference 的命令

xinference-local --host 0.0.0.0 --port 9997

Reproduction / 复现过程

Only the CPU version appears in the system, but not the NPU version. (https://github.com/user-attachments/assets/e8c23e55-bcb7-46bd-ad93-55d5faab897a)

Expected behavior / 期待表现

Looking forward to supporting the NPU version of Shengteng series well

qinxuye commented 2 weeks ago

Just launch model to see if you can choose NPU, now the cluster page cannot show NPU, but you can try to launch.

lvshuaigg commented 2 weeks ago

Just launch model to see if you can choose NPU, now the cluster page cannot show NPU, but you can try to launch.

Hello, the current interface does not show that there is an NPU, and the loaded NPU cannot be used and an error is reported. (https://github.com/user-attachments/assets/b2aa1a8d-9cc2-4446-ac5a-e2e8bfdef1d8) (https://github.com/user-attachments/assets/64676e42-241f-4cf2-888f-3621dd4b9a92)

AnnaYanami-8 commented 1 week ago

System Info / 系統信息

操作系统版本/架构：Linux 版本 4.19.90-23.48.v2101.ky10.aarch64 NPU：8 卡 910 ProB 32G Python：3.9 NPU 驱动程序：23.0.0 NPU 固件：7.1.0.3.220 CANN 工具包：7.0.0.RC1

使用 Docker 运行 Xinference？/ 是否使用 Docker 运行 Xinfernece？

  • [ ] 码头工人 / 码头工人
  • [x] pip install / 通过 pip install 安装
  • [ ] installation from source / 从源码安装

Version info / 版本信息

0.13.1 版

用于启动 Xinference 的命令 / 用以启动 xinference 的命令

xinference-local --host 0.0.0.0 --port 9997

Reproduction / 复现过程

系统中仅显示 CPU 版本，而不显示 NPU 版本。 （https://github.com/user-attachments/assets/e8c23e55-bcb7-46bd-ad93-55d5faab897a)

Expected behavior / 期待表现

期待能很好地支持盛腾系列的 NPU 版

天数智芯的智铠GPU都支持啊

github-actions[bot] commented 4 days ago

This issue is stale because it has been open for 7 days with no activity.