search

ublue-os / boxkit

Build your own custom OCI distrobox container
https://universal-blue.org
Apache License 2.0
178 stars 84 forks source link

Create Bluefin Ollama + WebUI Boxkit #72

Closed mecattaf closed 3 months ago

mecattaf commented 6 months ago

Hey @castrojo I saw the new local LLM justfile in bluefin https://github.com/ublue-os/bluefin/blob/6b58f9836bffb7ec0250296d2809e30e2c9325b6/just/bluefin-tools.just

I was wondering if it would make sense to use the Ollama + WebUI combo within a single distrobox based on bluefin-dx? I had a bad experience with justfiles and think that this feature would be a great use case for the Boxkit logic.

@p5 do you think that Boxkit is a better way to get ollama on ublue devices?

Many thanks!

castrojo commented 6 months ago

How about a dedicated container we could put in ublue-os/toolboxes with what you want in there? ublue-os/ollama or something, we could build it off of wolfi so it's nice and small and then just build it every time there's an update? If you've got time to help out I think this would be great, I don't have time to work on it for a while tho. And we could do rocm6 and cuda versions, etc too?

On Fri, May 3, 2024 at 12:10 PM Thomas Mecattaf @.***> wrote:

Hey @castrojo https://github.com/castrojo I saw the new local LLM justfile in bluefin

https://github.com/ublue-os/bluefin/blob/6b58f9836bffb7ec0250296d2809e30e2c9325b6/just/bluefin-tools.just

I was wondering if it would make sense to use the Ollama + WebUI combo within a single distrobox based on bluefin-dx? I had a bad experience with justfiles and think that this feature would be a great use case for the Boxkit logic.

@p5 https://github.com/p5 do you think that Boxkit is a better way to get ollama on ublue devices?

Many thanks!

— Reply to this email directly, view it on GitHub https://github.com/ublue-os/boxkit/issues/72, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJUT3IB3GZZYNOHP2CQCRTZAOZIZAVCNFSM6AAAAABHFYYXIGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TOOJZG4YTMNA . You are receiving this because you were mentioned.Message ID: @.***>

mecattaf commented 6 months ago

Thanks for the quick reply @castrojo. This could live in ublue-os/toolboxes. Submitted a first PR over there. I chose to build the container off bluefin-cli.

In later iterations we can add a nvidia-specific image, but for now let's do without.

Finally we could set up the systemd units by drawing inspiration from https://github.com/wayblueorg/wayblue/tree/live/config/systemd/system and the systemd module from bluebuild instead of having a justfile that writes the quadlets?