feat(Inference): supporting gpu layers - Githubissues

containers / podman-desktop-extension-ai-lab

Work with LLMs on a local environment using containers

https://podman-desktop.io/extensions/ai-lab

Apache License 2.0

145 stars 23 forks source link

feat(Inference): supporting gpu layers #1280

Open axel7083 opened 5 days ago

axel7083 commented 5 days ago

What does this PR do?

Screenshot / video of UI

New option Experimental GPU in podman desktop preferences

When enable, new input gpu layers to define

Side by Side playground comparison

https://github.com/containers/podman-desktop-extension-ai-lab/assets/42176370/6a8c5308-dc96-40bc-9200-210bd1fe9477

What issues does this PR fix or reference?

Need https://github.com/containers/podman-desktop-extension-ai-lab/pull/1278 to be merged Fixes https://github.com/containers/podman-desktop-extension-ai-lab/issues/1279

How to test this PR?

Require to be on windows with a WSL podman machine and a NVIDIA GPU

Turn on the experimental flag in the preference
Set a value for the gpu offloading (E.g. 20)
Start the inference server
Go to playground, notice the speed

slemeur commented 1 day ago

Before going forward on the PR, we the experience must be more comprehensive for the end-user:

Provide an explanation about what GPU offloading means, what is the scale for the value to be provided
Ideally, we would like to display some metadata: how much layers the user's GPU could handle and how much layers the model has.