ericcurtin
commented
4 hours ago Hey @bmahabirbu great to hear from you again. @swarajpande5 was looking into Nvidia support too, maybe you guys can look into it together. For branching, just fork the repo, create a branch and open a PR that way, that's generally how we do things.
Funnily enough I got AMD GPUs working today, by building a container like this:
FROM registry.access.redhat.com/ubi9/ubi:9.4-1214.1726694543
# renovate: datasource=github-releases depName=huggingface/huggingface_hub extractVersion=^v(?<version>.*)
ARG HUGGINGFACE_HUB_VERSION=0.25.1
# renovate: datasource=github-releases depName=containers/omlmd extractVersion=^v(?<version>.*)
ARG OMLMD_VERSION=0.1.5
# renovate: datasource=github-releases depName=tqdm/tqdm extractVersion=^v(?<version>.*)
ARG TQDM_VERSION=4.66.5
ARG LLAMA_CPP_SHA=70392f1f81470607ba3afef04aa56c9f65587664
# renovate: datasource=git-refs depName=ggerganov/whisper.cpp packageName=https://github.com/ggerganov/whisper.cpp gitRef=master versioning=loose type=digest
ARG WHISPER_CPP_SHA=5caa19240d55bfd6ee316d50fbad32c6e9c39528
# vulkan-headers vulkan-loader-devel vulkan-tools glslc glslang python3-pip mesa-libOpenCL-$MESA_VER.aarch64
RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
crb enable && \
dnf install -y epel-release && \
dnf --enablerepo=ubi-9-appstream-rpms install -y git procps-ng vim \
dnf-plugins-core python3-dnf-plugin-versionlock cmake gcc-c++ \
python3-pip python3-argcomplete && \
dnf clean all && \
rm -rf /var/cache/*dnf*
RUN /usr/bin/python3 --version
RUN pip install "huggingface_hub[cli]==${HUGGINGFACE_HUB_VERSION}"
RUN pip install "omlmd==${OMLMD_VERSION}"
RUN pip install "tqdm==${TQDM_VERSION}"
ARG ROCM_VERSION=6.2.2
ARG AMDGPU_VERSION=6.2.2
RUN <<EOF
cat <<EOD > /etc/yum.repos.d/rocm.repo
[amdgpu]
name=amdgpu
baseurl=https://repo.radeon.com/amdgpu/$AMDGPU_VERSION/rhel/9.4/main/x86_64/
enabled=1
priority=50
gpgcheck=1
gpgkey=https://repo.radeon.com/rocm/rocm.gpg.key
[ROCm]
name=ROCm
baseurl=https://repo.radeon.com/rocm/rhel9/$ROCM_VERSION/main
enabled=1
priority=50
gpgcheck=1
gpgkey=https://repo.radeon.com/rocm/rocm.gpg.key
EOD
EOF
RUN dnf config-manager --add-repo https://mirror.stream.centos.org/9-stream/AppStream/$(uname -m)/os/
RUN curl -o /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Official
RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Official
RUN dnf install -y rocm && \
dnf clean all && \
rm -rf /var/cache/*dnf*
RUN git clone https://github.com/ggerganov/llama.cpp && \
cd llama.cpp && \
git reset --hard ${LLAMA_CPP_SHA} && \
cmake -B build -DCMAKE_INSTALL_PREFIX:PATH=/usr -DGGML_CCACHE=0 \
-DGGML_HIPBLAS=1 && \
cmake --build build --config Release -j $(nproc) && \
cmake --install build && \
cd / && \
rm -rf llama.cpp
RUN git clone https://github.com/ggerganov/whisper.cpp.git && \
cd whisper.cpp && \
git reset --hard ${WHISPER_CPP_SHA} && \
make -j $(nproc) && \
mv main /usr/bin/whisper-main && \
mv server /usr/bin/whisper-server && \
cd / && \
rm -rf whisper.cppAnd then figuring out what is the GPU with the largest VRAM via:
rocm-smi --showmeminfo vram --json
{"card0": {"VRAM Total Memory (B)": "8573157376", "VRAM Total Used Memory (B)": "27136000"}, "card1": {"VRAM Total Memory (B)": "536870912", "VRAM Total Used Memory (B)": "371126272"}}Selecting that GPU via:
export HIP_VISIBLE_DEVICES=0
and adding --ngl 99 to llama.cpp command-line. But all the bits still have to be put together properly.
Then this 3b model works with GPU offload via ROCM to AMD GPU:
/usr/bin/ramalama run llama3.2
bmahabirbu
Hi Eric, It's Brian Mahabir, it was a pleasure to meet you at Dev Conf!
I have a bare-bones fully working demo that utilizes Nvidia GPU support. I'd like to have a nv branch on ramalama repo to have a record of the development.
Although you can build llama.cpp with GPU support it doesn't work seamlessly. For example, it takes time to offload the work to Vram so when using GPU you would have to wait about 15-30 seconds before the chatbox appears. However, Ollama handles this much better albeit at the cost of some performance. The plan is to look at how Ollama handles GPU support and integrate those changes for a better experience as it's built ontop of llama.cpp.