engine: AMD GPU support

[hiento09]

Overview

Tasklist

• [ ] Can this be solved via llama.cpp? (e.g. compiled for Vulkan and ROCm)
• [x] https://github.com/janhq/cortex.llamacpp/issues/9
• [ ] https://github.com/janhq/jan/issues/913

Original Post

Problem We need a new nitro bin file that support AMD GPU for both windows and linux

Success Criteria

• new nitro bin file support AMD GPU
• Integrate to Jan app

[tikikun]

hi @hiento09 can we spin up the azure instance of AMD GPU and test

[tikikun]

hi @hiento09 can you set up an instance with the specs like the link below, azure is not working

https://gist.github.com/cgmb/6ae0d118bf357fc4576a7568b85e1c45

[tikikun]

Tutorial to compile nitro on that system above (details in link https://gist.github.com/cgmb/6ae0d118bf357fc4576a7568b85e1c45 ):

First step, install AMD things (for details link: https://gist.github.com/cgmb/6ae0d118bf357fc4576a7568b85e1c45) :

After that just build nitro with below flag:

CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ \
    cmake -H. -Bbuild -DLLAMA_HIPBLAS=ON -DAMDGPU_TARGETS= gfx1011 -DCMAKE_BUILD_TYPE=Release 

and

make

[hiento09]

I am requesting quota for g4ad ec2 instance on aws, will comeback to this task when they open quota for us

[hiento09]

I was able to build nitro with above flag successfully using this docker image https://hub.docker.com/r/rocm/dev-ubuntu-22.04 but did not have AMD GPU for testing the bin file

[hiento09]

AWS reject our quota request, we may need to consider purchasing for some AMD GPU, @tikikun , @dan-jan

[hiepxanh]

I have one, let me test tomorrow 🗡️ @hiento09

Do you have file so I can test? Or what do I need to do to built it if need?

[hiepxanh]

@hiento09 can you provide bin file? I cannot make the build run on my WSL 2 windows

my target CPU is: gfx1032 (radeon 6600xt) if you need

[tikikun]

rename to ROCm support since vulkan already supported

[tikikun]

still have ROCm build left @hiento09 @hiro-v

[louis-jan]

Experimental feature: 0.4.7 - Vulkan

[hiro-v]

@louis-jan No it's not This is AMD RoCm (equivalent to NVIDIA CUDA), not Vulkan. I'm moving this one back to Icebox as of now

[Van-QA]

converting this to epic to close https://github.com/janhq/jan/issues/913

[0xSage]

@Van-QA can we queue this up after trtllm for Cam? 🙏

[Van-QA]

hi @0xSage, if you look at the sprint / status, you can see that it's already in the n‌ext spr‌int

[louis-jan]

From Jan: I think a UX enhancement is needed. (GPU settings)