joonas commented 3 months ago

Add Ollama sysext

This creates a sysext for running Ollama on Flatcar.

How to use

Create a Flatcar instance with configuration that looks roughly something like this:

---
variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys: 
        - ${ssh_keys}
storage:
  files:
    - path: /opt/extensions/ollama/ollama-0.3.6-x86-64.raw
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/ollama-0.3.6-x86-64.raw
    - path: /etc/sysupdate.ollama.d/ollama.conf
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/ollama.conf
  links:
    - target: /opt/extensions/ollama/ollama-0.3.6-x86-64.raw
      path: /etc/extensions/ollama.raw
      hard: false
systemd:
  units:
    - name: ollama.service
      enabled: true
      dropins:
        - name: 10-ollama-env-override.conf
          contents: |
            [Service]
            Environment=OLLAMA_MODELS="/var/lib/ollama"
            Environment=HOME="/var/lib/ollama"
            Environment=OLLAMA_RUNNERS_DIR="/var/lib/ollama"
    - name: systemd-sysupdate.timer
      enabled: true
    - name: systemd-sysupdate.service
      dropins:
        - name: ollama.conf
          contents: |
            [Service]
            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C ollama update
        - name: sysext.conf
          contents: |
            [Service]
            ExecStartPost=systemctl restart systemd-sysext

Testing done

I've deployed a DigitalOcean droplet with the above configuration and verified that the expected version of Ollama was available and could be used to inference (following steps from Ollama vision models blog post):

Flatcar Container Linux by Kinvolk stable 3975.2.0 for DigitalOcean
core@flatcar-demo-01 ~ $ ollama --version
ollama version is 0.3.6
core@flatcar-demo-01 ~ $ sudo systemctl start ollama
core@flatcar-demo-01 ~ $ sudo journalctl  -fu ollama
Aug 25 21:37:43 flatcar-demo-01 systemd[1]: Started ollama.service - Ollama.
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: Couldn't find '/var/lib/ollama/.ollama/id_ed25519'. Generating new private key.
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: Your new public key is:
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDzMvDKAScGwUiKb5ydjyOan9f8dAaqZ2uYRZhpPG/eI
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: 2024/08/25 21:37:43 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/var/lib/ollama OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR:/var/lib/ollama OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: time=2024-08-25T21:37:43.828Z level=INFO source=images.go:782 msg="total blobs: 0"
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: time=2024-08-25T21:37:43.829Z level=INFO source=images.go:790 msg="total unused blobs removed: 0"
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: time=2024-08-25T21:37:43.829Z level=INFO source=routes.go:1172 msg="Listening on 127.0.0.1:11434 (version 0.3.6)"
Aug 25 21:37:43 flatcar-demo-01 ollama[1658]: time=2024-08-25T21:37:43.831Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/var/lib/ollama
core@flatcar-demo-01 ~ $ ollama pull llava
pulling manifest 
pulling 170370233dd5... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 4.1 GB                         
pulling 72d6f08a42f6... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 624 MB                         
pulling 43070e2d4e53... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  11 KB                         
pulling c43332387573... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   67 B                         
pulling ed11eda7790d... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   30 B                         
pulling 7c658f9561e5... 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  564 B                         
verifying sha256 digest 
writing manifest 
removing any unused layers 
success 
core@flatcar-demo-01 ~ $ cd /tmp
core@flatcar-demo-01 /tmp $ curl -O https://ollama.com/public/blog/jmb.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  281k  100  281k    0     0  10.1M      0 --:--:-- --:--:-- --:--:-- 10.5M
core@flatcar-demo-01 /tmp $ ollama run llava "describe this image: ./jmb.jpg"
Added image './jmb.jpg'
 The image shows a colorful graffiti-style artwork on what appears to be the side of a building. The central figure is a stylized, fantastical creature with a dragon-like 
appearance, adorned with a crown and holding a scepter in its right hand. Below this creature is a text that reads "Jean Michel Basquiat / AYON," with the word "AYON" 
underlined. The style of the artwork is reminiscent of street art or pop culture aesthetics, characterized by bold outlines and vibrant colors. The building surface has other 
graffiti and images in various styles, contributing to an urban, artistic atmosphere. The background is blue with a hint of orange near the bottom edge of the image.

[ ] Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
[ ] Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

jepio commented 2 months ago

Awesome idea!

joonas commented 2 months ago

I've reworked this a bit to include the environment variables by default and using the tarball-based distribution instead of just downloading the binary as before.

joonas commented 2 months ago

LGTM. Thanks! Note that ollama.service won't start automatically when the sysext will be loaded. If it's not an issue, we can go ahead. :)

hmm, that seems like it probably should happen on boot, I'll take a look at changing that

tormath1 commented 2 months ago

LGTM. Thanks! Note that ollama.service won't start automatically when the sysext will be loaded. If it's not an issue, we can go ahead. :)

hmm, that seems like it probably should happen on boot, I'll take a look at changing that

When a service has to be started when the sysext is loaded, we usually add a drop-in to multi-user.target: https://github.com/flatcar/sysext-bakery/blob/82e4914481f88f09923fe6b25700c1905e650d21/create_wasmcloud_sysext.sh#L92-L94

joonas commented 2 months ago

@tormath1 thanks for the pointer, I thought I had originally already included that, but looks like I'd overlooked it.

Should be good to go now 🙂

flatcar / sysext-bakery