search

punica-ai / punica

Serving multiple LoRA finetuned LLM as one
https://arxiv.org/abs/2310.18547
Apache License 2.0
883 stars 40 forks source link

Support for H100 GPUs? #43

Open LorrinWWW opened 4 months ago

LorrinWWW commented 4 months ago

When I set TORCH_CUDA_ARCH_LIST="8.0 8.6 8.9 9.0", I got compiling errors. And then I found: https://github.com/punica-ai/punica/blob/591b59899f0a20760821785d06b331c8a2e5cb86/.github/workflows/release_wheel.yml#L15

Is there something we does not support yet? Thank you in advance!

Update: Adding +PTX works. But it would be great to have 9.0 supported in the future.

yzh119 commented 4 months ago

Yes, the codebase is compatible with sm90 and we should add 90 to CD.

cc @abcdabcd987