How do I launch the api on a graphics card other than cuda: 0

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

Apache License 2.0

1.76k stars 163 forks source link

Open Stark-zheng opened 2 months ago

Stark-zheng commented 2 months ago

1、deepspeed --include localhost:4 api.py

2、CUDA_VISIBLE_DEVICES=4 python api.py

Neither method can specify cuda to use, so cuda: 0 is used by default

How do I set it?

zhjunqin commented 2 months ago

same question, CUDA_VISIBLE_DEVICES not working