feat(amd): support ROCm detection

bentoml / OpenLLM

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

https://bentoml.com

Apache License 2.0

9.7k stars 616 forks source link

feat(amd): support ROCm detection #950

Closed younseojava closed 3 months ago

younseojava commented 4 months ago

This is first attempt to enable ROCm for AMD GPUs. It consists of

GPU detection with rocm_smi_lib to check non-default location of ROCm library
Add trust_remote_code option for Baichuan model
Skip test for gpu_validate

aarnphm commented 3 months ago

Hi @younseojava, thanks for the PR. For 0.5, we have simplify a lot of the internal logics and decided to use vLLM as the current running backend for now. Which means AMD support will come from vLLM.

I will close the PR for now. Looking forward for future collab :)

aarnphm commented 3 months ago

Actually, there is a code path that we are still using this; I will reopen this.

Can you fix the merge conflict? ths.