Alpaca uses my CPU instead of my GPU (AMD)

frandavid100 commented 4 months ago

I have noticed that Alpaca uses my CPU instead of my GPU. Here's a screenshot showing how it's using almost 40% of my CPU, and only 1% of my GPU.

Captura desde 2024-07-10 06-51-39

I'm using an AMD Radeon RX 6650 XT GPU, which is properly detected by the OS and used by other Flatpak apps like Steam. As you can see in this other screenshot:

Captura desde 2024-07-10 06-54-34

Jeffser commented 4 months ago

Hi, yes this is a problem with ROCM and Flatpaks, I believe this is also a problem with Blender.

Whilst any flatpak can detect and use the GPU for some reason ROCM doesn't work out of the box, there must be a way but I haven't figured it out and it's a bit hard to test since I have an incompatible GPU.

For now I suggest you host an Ollama instance using docker and connect it to Alpaca using the remote connection option.

frandavid100 commented 4 months ago

There's no hurry, I use it sparsely and I can afford it to use the CPU for the time being.

Is there any way I can help to test a possible fix? Is my GPU supposed to be compatible?

loulou64490 commented 4 months ago

Alpaca is based on ollama. Ollama automatically detect CPU and GPU, but when it's executed by flatpak, ollama is contenarized (idk if this word exist lol) and don't have enough privilege to check if GPU can be used. That's what I understand !!

Jeffser commented 4 months ago

yeah that word does exist, though the problem isn't exactly the fact that it is inside a container, the problem is that ROCM doesn't work out of the box.

olumolu commented 3 months ago

I think rocm need to be loaded separately. https://github.com/ollama/ollama/releases/download/v0.2.8/ollama-linux-amd64-rocm.tgz this contains the rocm driver. this is a real issue that need to be fixed.

Jeffser commented 3 months ago

adding that as it is would mean making Alpaca 4 times heavier and not everybody would even need rocm, the problem here is that either the freedesktop runtime or the gnome runtime should include rocm, that or I might not know a better solution right now since I'm still new with flatpak packaging

Jeffser commented 3 months ago

I might finally have a solution where the flatpak accesses the ROCM libraries from the system itself

0chroma commented 3 months ago

adding that as it is would mean making Alpaca 4 times heavier and not everybody would even need rocm, the problem here is that either the freedesktop runtime or the gnome runtime should include rocm, that or I might not know a better solution right now since I'm still new with flatpak packaging

You could always package it as an extension in that case

Jeffser commented 3 months ago

yeah the problem with that is that I would need to make a different package for flathub

TacoCake commented 3 months ago

Any progress on this? Anything you need help with in getting this done?

Jeffser commented 3 months ago

do you have rocm installed on your system? I think I can make Ollama use the system installation

Jeffser commented 3 months ago

if someone has ROCm installed and want to test this run these commands

flatpak override --filesystem=/opt/rocm com.jeffser.Alpaca
flatpak override --env=LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/lib64:/app/lib:/usr/lib/x86_64-linux-gnu/GL/default/lib:/usr/lib/x86_64-linux-gnu/openh264/extra:/usr/lib/sdk/llvm15/lib:/usr/lib/sdk/openjdk11/lib:/usr/lib/sdk/openjdk17/lib:/usr/lib/x86_64-linux-gnu/GL/default/lib com.jeffser.Alpaca

This gives the Flatpak access to /opt/rocm and specifies that it is an available library, the rest are the default Flatpak libraries, just ignore that

frandavid100 commented 3 months ago

How can I install ROCm on my Silverblue machine? I tried to run "rpm-ostree install rocm" but I get a "packages not found" error.

Jeffser commented 3 months ago

I think this should be it

https://copr.fedorainfracloud.org/coprs/cosmicfusion/ROCm-GFX8P/

olumolu commented 3 months ago

How can I install ROCm on my Silverblue machine? I tried to run "rpm-ostree install rocm" but I get a "packages not found" error.

Ask https://discussion.fedoraproject.org/ for help they actually help in this case

Jeffser commented 3 months ago

I was looking around on what Flatpaks include and they have all the stuff needed to run an app with OpenCL (a mesa alternative to ROCm as far as I'm aware) but Ollama can't use it, my recommendation for now is to run Ollama separately of Alpaca and just connect to it as a remote connection

TacoCake commented 3 months ago

Could you use Vulkan instead of trying to use ROCm? Kinda how GTP4ALL does it https://github.com/nomic-ai/gpt4all

Genuine question

TacoCake commented 3 months ago

do you have rocm installed on your system? I think I can make Ollama use the system installation

I don't have ROCm on my system, since it's kind of a headache to install on openSUSE Tumbleweed

olumolu commented 3 months ago

As far as i know the backend ollama use rocm instead of vulkan from front end this not too easy to implement this

Shished commented 3 months ago

GPT4All uses llama.cpp backend while this app uses ollama.

olumolu commented 3 months ago

Yes I don't lnow much about llama.cpp But ollama use rocm I this is really a issue that rocm is not installed on many systems

TacoCake commented 3 months ago

As far as i know the backend ollama use rcom instead of vulkan from front end this not too easy to implement this

GPT4All uses llama.cpp backend while this app uses ollama.

Ahhh I see, sorry for the confusion. If anyone wants to track vulkan support on Ollama:

olumolu commented 3 months ago

Yes if this get merged i hope it will bring vulkan to thos one also. For the time being I dont thing we can do much

Jeffser commented 3 months ago

do you have rocm installed on your system? I think I can make Ollama use the system installation

I don't have ROCm on my system, since it's kind of a headache to install on openSUSE Tumbleweed

I know, it's a headache everywhere including the Flatpak sandbox

francus11 commented 3 months ago

flatpak override --filesystem=/opt/rocm com.jeffser.Alpaca
flatpak override --env=LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/lib64:/app/lib:/usr/lib/x86_64-linux-gnu/GL/default/lib:/usr/lib/x86_64-linux-gnu/openh264/extra:/usr/lib/sdk/llvm15/lib:/usr/lib/sdk/openjdk11/lib:/usr/lib/sdk/openjdk17/lib:/usr/lib/x86_64-linux-gnu/GL/default/lib com.jeffser.Alpaca

I installed rocm on fedora using this tutorial https://fedoraproject.org/wiki/SIGs/HC#Installation Still though, my GPU usage is 0%. Any other suggestions?

Jeffser commented 3 months ago

Today I learned that ROCm is actually bundled with the Ollama binary... So I have no idea what to try now lol

(third line)

Jeffser commented 3 months ago

Ollama says that AMD users should try the propitiatory driver tho

https://github.com/ollama/ollama/blob/main/docs/linux.md#amd-radeon-gpu-support

P-Jay357 commented 2 months ago

Has anyone managed to get it to work with their AMD GPU (I'm using Fedora Workstation 40)? I have a 6950XT, but it's using my CPU and RAM rn. I don't really want to use the proprietary drivers if I don't have to though....

@Jeffser I do love this project though, but it would be 100x better if I could run the LLMs using my GPU!

olumolu commented 2 months ago

Has anyone managed to get it to work with their AMD GPU (I'm using Fedora Workstation 40)? I have a 6950XT, but it's using my CPU and RAM rn. I don't really want to use the proprietary drivers if I don't have to though....

@Jeffser I do love this project though, but it would be 100x better if I could run the LLMs using my GPU!

Yes this will be fixed once ollama merge the vulken support this issue will be fixed else amd need rcom driver along with ollama for amd gpu to function in the ai ml stuff if there is any other option to fix that can be considered as this is a open issue.

Jeffser commented 2 months ago

Can someone with an AMD GPU test if it works when they have the propitiatory driver?

czhang03 commented 2 months ago

I have tried to install rocm via the following command rpm-ostree install rocminfo hipblas on silverblue. There is no problem with secureboot, even with secureblue's hardening boot parameters.

And I can indeed see rocm is installed and detecting my GPU

> rocminfo
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
Runtime Ext Version:     1.4
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES
...
*******                  
Agent 2                  
*******                  
  Name:                    gfx1103                            
  Uuid:                    GPU-XX                             
  Marketing Name:          AMD Radeon 760M                    
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE

I have added the following environment variable via flatseal: LD_LIBRARY_PATH=/var/run/host/usr/lib64/rocm:/app/lib:/usr/lib/x86_64-linux-gnu/GL/default/lib:/usr/lib/x86_64-linux-gnu/openh264/extra:/usr/lib/sdk/llvm15/lib:/usr/lib/sdk/openjdk11/lib:/usr/lib/sdk/openjdk17/lib:/usr/lib/x86_64-linux-gnu/GL/default/lib and enabled host-os file permission.

But it seems like ollama is still not finding the GPU:

> flatpak run com.jeffser.Alpaca

INFO    [main.py | main] Alpaca version: 1.1.1
INFO    [local_instance.py | start] Starting Alpaca's Ollama instance...
2024/08/30 09:51:42 routes.go:1108: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11435 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/var/home/[reducted]/.var/app/com.jeffser.Alpaca/data/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-08-30T09:51:42.913-04:00 level=INFO source=images.go:781 msg="total blobs: 14"
time=2024-08-30T09:51:42.913-04:00 level=INFO source=images.go:788 msg="total unused blobs removed: 0"
time=2024-08-30T09:51:42.913-04:00 level=INFO source=routes.go:1155 msg="Listening on 127.0.0.1:11435 (version 0.3.3)"
time=2024-08-30T09:51:42.914-04:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=[redacted]
INFO    [local_instance.py | start] Started Alpaca's Ollama instance
INFO    [local_instance.py | start] Ollama version: 0.3.3
time=2024-08-30T09:51:45.305-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60102 cpu cpu_avx cpu_avx2]"
time=2024-08-30T09:51:45.305-04:00 level=INFO source=gpu.go:205 msg="looking for compatible GPUs"
time=2024-08-30T09:51:45.307-04:00 level=INFO source=gpu.go:346 msg="no compatible GPUs were discovered"
time=2024-08-30T09:51:45.307-04:00 level=INFO source=types.go:105 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="14.9 GiB" available="8.2 GiB"
[GIN] 2024/08/30 - 09:51:45 | 200 |     935.765µs |       127.0.0.1 | GET      "/api/tags"
[GIN] 2024/08/30 - 09:51:45 | 200 |     526.644µs |       127.0.0.1 | GET      "/api/tags"

czhang03 commented 2 months ago

I have done some debugging with this in distrobox, it seems like ollama is not "properly" reading environmental variables. Here is what I did:

# setup distrobox and install dependencies
> distrobox create && distrobox enter
[in distrobox] > sudo dnf install rocminfo hipblas
[in distrobox] > rocminfo  # to make sure amd gpu is detected.

# install ollama
[in distrobox] > curl -fsSL https://ollama.com/install.sh | sh

# set environmental variable 
[in distrobox] > OLLAMA_DEBUG=1
[in distrobox] > HSA_OVERRIDE_GFX_VERSION=11.0.2

# test ollama
[in distrobox] > ollama serve

I am getting the following output:

level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu_avx2 cuda_v11 cuda_v12 rocm_v60102 cpu cpu_avx]"
level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
level=WARN source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
level=WARN source=amd_linux.go:340 msg="amdgpu is not supported" gpu=0 gpu_type=gfx1103 library=/usr/lib64 supported_types="[gfx1030 gfx1100 gfx1101 gfx1102]"
level=WARN source=amd_linux.go:342 msg="See https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides for HSA_OVERRIDE_GFX_VERSION usage"
level=INFO source=amd_linux.go:360 msg="no compatible amdgpu devices detected"
level=INFO source=gpu.go:347 msg="no compatible GPUs were discovered"
level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="13.4 GiB" available="5.4 GiB"

notice that there is no debug information and in the log gpu_type=gfx1103 seems to suggest that the override variable HSA_OVERRIDE_GFX_VERSION=11.0.2 is not being read, see https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides-on-linux.

IIRC ollama will create another user and use that user to run the service, so I was wondering if that is what causing the problem, and I don't know if alpaca can fix that.

czhang03 commented 2 months ago

Oh I found something interesting, when I enable OLLAMA_DEBUG=1 environment variable, I get the following:

time=2024-08-30T15:00:35.718-04:00 level=DEBUG source=amd_linux.go:371 msg="amdgpu driver not detected /sys/module/amdgpu"

It seems like ollama is looking at amd gpu by looking at /sys/module/amdgpu, this can be fixed by adding /sys/module/amdgpu to file permssions (I think it is read only anyway, so ro is probably not needed).

However, after this is fixed, I have encountered the following error:

time=2024-08-30T15:04:02.463-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /app/bin/rocm"
time=2024-08-30T15:04:02.463-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /var/run/host/usr/lib64/rocm"
time=2024-08-30T15:04:02.463-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /app/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib/x86_64-linux-gnu/GL/default/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib/x86_64-linux-gnu/openh264/extra"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib/sdk/llvm15/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib/sdk/openjdk11/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib/sdk/openjdk17/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib/x86_64-linux-gnu/GL/default/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /opt/rocm/lib"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/lib64"
time=2024-08-30T15:04:02.464-04:00 level=DEBUG source=amd_common.go:16 msg="evaluating potential rocm lib dir /usr/share/ollama/lib/rocm"
time=2024-08-30T15:04:02.464-04:00 level=WARN source=amd_linux.go:395 msg="amdgpu detected, but no compatible rocm library found.  Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install"

It seems like ollama is looking at /var/run/host/usr/lib64/rocm (the rocm library on host) but decided it is not a rocm library. There are probably ways to get around this with more debugging. However, I think the proper way to do this is probably to do a rocm platform and make an alphaca extension that depends on such platfrom, but it is a lot of work...

Jeffser commented 2 months ago

Nice find I will add /sys/module/amdgpu to the flatpak permissions, do you use the propitiatory driver or the open source one?

czhang03 commented 2 months ago

I use the open source one. I managed to fix the previous library not found problem by adding /var/run/host/usr/lib64/rocm/gfx11/lib to LD_LIBRARY_PATH, notice I used gfx11 instead of gfx1100, since gfx1100 is empty on my machine...

At this point, ollama is still complaining about not enough vram, when it seems to be detecting enough. I will post an issue on ollama soon, see if they can help us with this.

Jeffser commented 2 months ago

Good News

So I added /sys/module/amdgpu to the Flatpak manifest on this commit

As we saw before this makes Ollama output this error

amdgpu detected, but no compatible rocm library found. Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install

Now, I made a Flatpak module that installs the ROCm libraries that Ollama provide, this makes Ollama output this different error message

amdgpu is not supported" gpu=0 gpu_type=gfx1010 library=/app/lib/ollama supported_types="[gfx1030 gfx1100 gfx1101 gfx1102 gfx900 gfx906 gfx908 gfx90a gfx940 gfx941 gfx942]

This is because my GPU isn't compatible with Ollama, what matters is that everything is coming together to make AMD support possible without needing to install any drivers or ROCm at a system level

Now, I don't want to make ROCm a requirement on every installation of Alpaca, I'm going to make a Flatpak extension that installs it into the Alpaca libraries, you can see an example of this with the Steam Flatpak which offers Gamescope as an extension on Flathub

Jeffser commented 2 months ago

Update

The extension works! I asked for it to be added to Flathub, by the time I get that approved 2.0.0 should have been released with the extension compatibility added

Jeffser commented 2 months ago

btw, don't ask me why but the ROCm libraries that Ollama provides are... almost 8GB, don't know if that's normal but that's what I have to work with.

They are compressed tho

TheRsKing commented 2 months ago

are the proprietary amd drivers still required? if not it does not work for me

Jeffser commented 2 months ago

are the proprietary amd drivers still required? if not it does not work for me

I'm not sure if the drivers at /sys/module/amdgpu are the proprietary drivers, could you check if you have that directory?

czhang03 commented 2 months ago

@Jeffser that dir exists even on open-source drivers. I have some time to test today, and I will try to see what I can find.

Jeffser commented 2 months ago

well my guess is that it does work on open source drivers, when I was testing it, it outputs that my gpu isn't compatible but it does find the driver

sramkrishna commented 2 months ago

btw, if you need any help with the intel gpus, let me know. (Intel employee, but also a GNOME person)

ghost commented 2 months ago

I installed the add-on, but the llama server crashes. I have the numactl AUR package installed. Here's the entire log: output.log

zrooda commented 2 months ago

Alpaca 2.0.0 falls back to CPU on my RX 7900 GRE, should it already be working? Do I need to specifically install the new extension or?

ghost commented 2 months ago

Alpaca 2.0.0 falls back to CPU on my RX 7900 GRE, should it already be working? Do I need to specifically install the new extension or?

You need to install it, then it should work: flatpak install flathub com.jeffser.Alpaca.Plugins.AMD

P-Jay357 commented 2 months ago

I just downloaded the extension, restarted my PC and had a go with Alpaca. It just says that there was an error with the Ollama instance.

For the Ollama Overrides settings in Alpaca, I have set the HSA_OVERRIDE_GFX_VERSION to 10.3.0 (as I have a 6950xt), but left the others blank. Do I need to set anything for these (as I have and iGPU with my 7800x3D)?

czhang03 commented 2 months ago

@P-Jay357 To the best of my knowledge, if you have a supported GPU, you don't even need to set HSA_OVERRIDE_GFX_VERSION. It is for unsupported GPU to fallback to the closest rocm.

Jeffser commented 2 months ago

For the Ollama Overrides settings in Alpaca, I have set the HSA_OVERRIDE_GFX_VERSION to 10.3.0 (as I have a 6950xt), but left the others blank. Do I need to set anything for these (as I have and iGPU with my 7800x3D)?

I believe that GPU doesn't need an override to work, could you delete that and test it?

czhang03 commented 2 months ago

I am still running into the too little vram problem. I have updated ollama installed the plugin and uses a very small model (smollm 135m)

time=2024-09-11T13:54:46.691-04:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="13.4 GiB" before.free="7.6 GiB" before.free_swap="7.4 GiB" now.total="13.4 GiB" now.free="7.7 GiB" now.free_swap="7.4 GiB"
time=2024-09-11T13:54:46.691-04:00 level=DEBUG source=amd_linux.go:440 msg="updating rocm free memory" gpu=0 name=1002:15bf before="524.2 MiB" now="560.9 MiB"
time=2024-09-11T13:54:46.691-04:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x816740 gpu_count=1
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/var/home/cheng/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-eb2c714d40d4b35ba4b8ee98475a06d51d8080a17d2d2a75a23665985c739b94
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[560.9 MiB]"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:168 msg="gpu has too little memory to allocate any layers" gpu="{memInfo:{TotalMemory:2147483648 FreeMemory:588095488 FreeSwap:0} Library:rocm Variant: MinimumMemory:479199232 DependencyPath:/app/plugins/AMD/lib/ollama EnvWorkarounds:[] UnreliableFreeMemory:false ID:0 Name:1002:15bf Compute:gfx1103 DriverMajor:0 DriverMinor:0}"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:296 msg="insufficient VRAM to load any model layers"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[560.9 MiB]"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:168 msg="gpu has too little memory to allocate any layers" gpu="{memInfo:{TotalMemory:2147483648 FreeMemory:588095488 FreeSwap:0} Library:rocm Variant: MinimumMemory:479199232 DependencyPath:/app/plugins/AMD/lib/ollama EnvWorkarounds:[] UnreliableFreeMemory:false ID:0 Name:1002:15bf Compute:gfx1103 DriverMajor:0 DriverMinor:0}"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:296 msg="insufficient VRAM to load any model layers"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[560.9 MiB]"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:168 msg="gpu has too little memory to allocate any layers" gpu="{memInfo:{TotalMemory:2147483648 FreeMemory:588095488 FreeSwap:0} Library:rocm Variant: MinimumMemory:479199232 DependencyPath:/app/plugins/AMD/lib/ollama EnvWorkarounds:[] UnreliableFreeMemory:false ID:0 Name:1002:15bf Compute:gfx1103 DriverMajor:0 DriverMinor:0}"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:296 msg="insufficient VRAM to load any model layers"
time=2024-09-11T13:54:46.701-04:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[560.9 MiB]"
time=2024-09-11T13:54:46.702-04:00 level=DEBUG source=memory.go:168 msg="gpu has too little memory to allocate any layers" gpu="{memInfo:{TotalMemory:2147483648 FreeMemory:588095488 FreeSwap:0} Library:rocm Variant: MinimumMemory:479199232 DependencyPath:/app/plugins/AMD/lib/ollama EnvWorkarounds:[] UnreliableFreeMemory:false ID:0 Name:1002:15bf Compute:gfx1103 DriverMajor:0 DriverMinor:0}"
time=2024-09-11T13:54:46.702-04:00 level=DEBUG source=memory.go:296 msg="insufficient VRAM to load any model layers"
time=2024-09-11T13:54:46.702-04:00 level=DEBUG source=server.go:101 msg="system memory" total="13.4 GiB" free="7.7 GiB" free_swap="7.4 GiB"
time=2024-09-11T13:54:46.702-04:00 level=DEBUG source=memory.go:101 msg=evaluating library=rocm gpu_count=1 available="[560.9 MiB]"
time=2024-09-11T13:54:46.702-04:00 level=DEBUG source=memory.go:168 msg="gpu has too little memory to allocate any layers" gpu="{memInfo:{TotalMemory:2147483648 FreeMemory:588095488 FreeSwap:0} Library:rocm Variant: MinimumMemory:479199232 DependencyPath:/app/plugins/AMD/lib/ollama EnvWorkarounds:[] UnreliableFreeMemory:false ID:0 Name:1002:15bf Compute:gfx1103 DriverMajor:0 DriverMinor:0}"
time=2024-09-11T13:54:46.702-04:00 level=DEBUG source=memory.go:296 msg="insufficient VRAM to load any model layers"

I think I will try the 3.0.10, which they state that have better error message about insufficient ram.

Jeffser / Alpaca

Alpaca uses my CPU instead of my GPU (AMD) #139

Good News

Update