LucasRouckhout commented 4 years ago

Hopefully this is the correct format for asking questions so apologies in advance if this is not the case.

Use case

My use case is quite simple. I want to create a bootable iso from the k3sOS iso which would allow me to have a running instance of our application on any machine by simply running the installer from a USB stick.

Specifically this should also work in an "air-gaped" system, by which I mean either no connected network interface or simply running in a subnet without access to the internet.

Plan of attack

My plan of attack was the following. I would customize the config.yaml and use the write_file entry to add a deployment yaml to the /var/lib/rancher/k3s/server/manifests which, if I understand correctly, would auto deploy the manifest on startup. The only problem left is that I need a way of pre-loading the necessary docker images for the application.

Question

Is possible in the first place? Can I pre-load images like this? and can I auto deploy as described in the k3s documentation? Also, is there a cleaner way of providing files that writing out the contents in a YAML file? Some of these images are rather big and my config.yaml file would become unmanageable.

LucasRouckhout commented 4 years ago

I already found out I can just add files to the k3os/system folder when remastering the ISO.

dweomer commented 4 years ago

I already found out I can just add files to the k3os/system folder when remastering the ISO.

@LucasRouckhout I know of at least one other community user that is doing this exact same thing combined with a custom install script to copy image tarballs onto the new installation.

For now this is the best we've got but I would like to work on (or see the community contribute to) a formal ISO remastering script to make it easier to build custom installation media without the need to muck around with the build system (which isn't complicated, just tedious).

j-landru commented 4 years ago

On #226, I exposed another approach, unifying (with aufs or overlayfs) at startup the K3OS_STATE volume (the K3OS iso) and a second volume labelled K3OS_DELTA (containing persistent files, and possibly container image files). The K3OS_DELTA volume will be used to store persistent data to personalize each cloned node by avoiding copying files by script at startup. This was inspired from unionfs technics used to provide data persistence when booting from a read-only liveDVD/USB. Unfortunately, I didn't find enough time to test this approach recently, I hope that I will soon be able to rework on this idea. If successful, I won't miss to share my personal notes.

dweomer commented 4 years ago

@j-landru I was thinking about your use case recently and I am quite curious to see what you've come up with.

j-landru commented 4 years ago

@dweomer Unfortunately for last three months, due to heavy load in my teaching and lecture activities, my personal kube projects remain in stand by... I hope I'll can investigate k3s/k3os again soon.

At that point I have documented my hybrid (UEFI/BIOS) multiboot k3OS usb keys which will be used to build a small core storage cluster based on k3os/k3s/rook/ceph. My personnal notes on that hybrid (UEFI/BIOS) multiboot K3OS usbkey howto is available as snippsets on a alternate community gitlab : https://framagit.org/snippets/4060

I plan to store persistent cluster config and log files on a second f2fs labeled K3OS_DELTA usb key which will be overlayfs unified at bootup. Untill now I've a unique k3os agent VM mockup, with quick and dirty scripted overlayfs fstab entries.

Here are my "mockup" config files on my master K3OS_STATE labeled storage.

/var/lib/rancher/k3os/config.d/10-write-files-config.yaml

write_files:
- encoding: ""
  content: |-
    # Absolut path to the keymap which should be loaded using loadkmap.
    #KEYMAP="/usr/share/keymaps/xkb/us.map.gz"
    KEYMAP="/usr/share/bkeymaps/fr/fr-nodeadkeys.bmap.gz"
  owner: root:root
  path: /etc/conf.d/loadkmap
  permissions: '0644'
- encoding: ""
  content: |-
    #!/bin/bash
    # mount-k3os-delta script for initial K3OS_DELTA ovelray dirs mount
    # Goal : k3os persistent dirs (/home, /var/lib/rancher, /var/log, /opt, /var/local)
    #        overlayed on a dedicated K3OS_DELTA labeled bloc device
    # mount k3OS_DELTA labeled block device on /k3os/delta
    mkdir -p /k3os/delta || echo "/k3os/delta dir exists"
    mount -L K3OS_DELTA /k3os/delta
    #mount overlay dirs (/home, /var/lib/rancher, /var/log, /opt, /var/local), as at this step, fstab entries not yet mounted ?!
    mount -t overlay overlay -o lowerdir=/var,upperdir=/k3os/delta/var,workdir=/k3os/delta/work/var /var
    #mount -t overlay overlay -o lowerdir=/var/lib/rancher,upperdir=/k3os/delta/var/lib/rancher,workdir=/k3os/delta/work/var/lib/rancher /var/lib/rancher
    #mount -t overlay overlay -o lowerdir=/var/log,upperdir=/k3os/delta/var/log,workdir=/k3os/delta/work/var/log /var/log
    mount -t overlay overlay -o lowerdir=/home,upperdir=/k3os/delta/home,workdir=/k3os/delta/work/home /home
    mount -t overlay overlay -o lowerdir=/opt,upperdir=/k3os/delta/opt,workdir=/k3os/delta/work/opt /opt
    #mount -t overlay overlay -o lowerdir=/var/local,upperdir=/k3os/delta/var/local,workdir=/k3os/delta/work/var/local /var/local
  owner: root:root
  path: /etc/mount-k3os-delta.sh
  permissions: '0755'
- encoding: ""
  content: |-
    # k3os fstab with overlayfs from K3OS_DELTA labeled block device
    /dev/cdrom          /media/cdrom    iso9660         noauto,ro       0 0
    /dev/usbdisk        /media/usb      vfat            noauto,ro       0 0
    # K3OS_DELTA 
    LABEL=K3OS_DELTA    /k3os/delta     f2fs            rw,acl,active_logs=6,background_gc=on,user_xattr 0 1
    #/dev/vdb1           /k3os/delta     f2fs            rw,acl,active_logs=6,background_gc=on,user_xattr 0 1
    # overlay mount for /var/lib/rancher and /var/log
    overlay             /var overlay        lowerdir=/var,upperdir=/k3os/delta/var,workdir=/k3os/delta/work/var 0 0
    #overlay             /var/lib/rancher overlay        lowerdir=/var/lib/rancher,upperdir=/k3os/delta/var/lib/rancher,workdir=/k3os/delta/work/var/lib/rancher 0 0
    #overlay             /var/log        overlay         lowerdir=/var/log,upperdir=/k3os/delta/var/log,workdir=/k3os/delta/work/var/log    0 0
    overlay             /home           overlay         lowerdir=/home,upperdir=/k3os/delta/home,workdir=/k3os/delta/work/home             0 0
    overlay             /opt            overlay         lowerdir=/opt,upperdir=/k3os/delta/opt,workdir=/k3os/delta/work/opt                0 0
    #overlay             /var/local      overlay         lowerdir=/var/local,upperdir=/k3os/delta/var/local,workdir=/k3os/delta/work/var/local   0 0
  owner: root:root
  path: /etc/fstab
  permissions: '0644'
- encoding: ""
  content: |-
    #!/bin/bash
    # rancher update-issue for C.I.R.R.U.S.
    # J. Landru (IMT Lille Douai)
    #    created  : 20190904
    #    modified :          by
    #
    . /etc/os-release
    cat > /etc/issue << "COW"
                   ,        ,                                                                ___
      ,------------|'------'|  _     ____                       ffffff                      / _ \\  ___
     / .           '-'    |-' | |   |___ \\                     ffffff                   ___/ /@> |/ _ \\              
     \\/|             |    |   | | __  __) |  _ _   ___        ff                       / _ \\ \\__/  /@> |
       |   .________.'----'   | |/ / |__ <  / _ \\ / __|      fffff   oooo    r.rrr.   / /@> |      \\__/    ___
       |   |        |   |     |   <  ___) || (_) |\\__ \\     ff      oo oo   rr;  rr   \\ \\__/  C.I.R.R.U.S. ______
       \\___/        \\___/     |_|\\_\\|____/  \\___/ |___/    ff      oo oo   rr          \\____________________________
                                                          ff       oooo   rr         Cloud Iac Réservoir de Ressources
                                                                                       Unifiées pour Sessions de tp
    COW
    cat >> /etc/issue << EOF
    $PRETTY_NAME
    Kernel \r on an \m (\l)

    =================================================================================================================
    NIC              State          Address
    $(ip -br addr show | grep -E -v '^(lo|flannel|cni|veth)')
    =================================================================================================================
    EOF
    cat >> /etc/issue << "EOF"

    Welcome to k3OS for C.I.R.R.U.S. (login with user: rancher)
    EOF
    sed -e "s/\\\/\%/g" /etc/issue | sed -e "s/%%/\\\/g" | sed -e "s/(login with user: rancher)/(login with authorized keys, otherwise go your way \!)/g"> /etc/issue.net
    # reverse to none the following ligne to suppress that banner for remote ssh login
    sed -i  "s/.*Banner.*/Banner \/etc\/issue.net/g"  /etc/ssh/sshd_config
    # for ssh, login with authorized keys, otherwise go your own way !
    sed -i "s/.*PasswordAuthentication.*/PasswordAuthentication no/g" /etc/ssh/sshd_config
  owner: root:root
  path: /root/update-issue
  permissions: '0755'
- encoding: ""
  content: |-
    #!/sbin/openrc-run
    depend() {
        after net-online
        need net
    }
    name="issue"
    command="/root/update-issue"
  owner: root:root
  path: /etc/init.d/issue
  permissions: '0755'
- encoding: ""
  content: |-
    # My new  delta /root/testfile.txt
    Text file test
  owner:  root:root
  path: /root/testfile.txt
  permissions: '0644'
- encoding: ""  
  content: |-
   # My new  delta /root/testfile-2.txt
   Text file test
  owner:  root:root
  path: /root/testfile-2.txt
  permissions: '0644'

/var/lib/rancher/k3os/config.d/30-cmd-config.yaml

init_cmd:
- "/etc/mount-k3os-delta.sh"

/var/lib/rancher/k3os/config.d/70-k3os-config.yaml

k3os:
  modules:
    - kvm
    - nvme
    - overlay

On that mockup VM, K3OS_DELTA storage which contains k3os is now mounted on /k3os/delta mountpoint and K3os persistent directories are overlayfs mounted

k3os-25742 [~]$ mount
/dev/vda1 on / type ext4 (rw,relatime,data=ordered)
/dev/loop1 on /usr type squashfs (ro,relatime)
none on /etc type tmpfs (rw,relatime)
none on /proc type proc (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=403944k,mode=755)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=403944k)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=248418,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
rdma on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,relatime,name=systemd)
/k3os/system/kernel/4.15.0-47-generic/kernel.squashfs on /usr/lib/modules type squashfs (ro,relatime)
/k3os/system/kernel/4.15.0-47-generic/kernel.squashfs on /usr/lib/firmware type squashfs (ro,relatime)
/dev/vda1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /k3os/system type ext4 (ro,relatime,data=ordered)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
/dev/vdb1 on /k3os/delta type f2fs (rw,relatime,lazytime,background_gc=on,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
overlay on /var type overlay (rw,relatime,lowerdir=/var,upperdir=/k3os/delta/var,workdir=/k3os/delta/work/var)
overlay on /home type overlay (rw,relatime,lowerdir=/home,upperdir=/k3os/delta/home,workdir=/k3os/delta/work/home)
overlay on /opt type overlay (rw,relatime,lowerdir=/opt,upperdir=/k3os/delta/opt,workdir=/k3os/delta/work/opt)

That's the point where the project was when suspended...

Next step will be to replace that ugly mount and fstab /etc/mount-k3os-delta.sh with cleaner and more generic script something like, but not yet tested on my mockup...

#!/bin/bash
# mount-k3os-delta script for initial K3OS_DELTA ovelray dirs mount
# Goal : k3os persistent dirs (/home, /var, /opt) overlayed on a dedicated K3OS_DELTA labeled bloc device
#        if a  k3OS_DELTA labeled block device is found then
#                - mount that block device /k3os/delta
#                - overlay mount persistent dirs
#                - append fstab entry accordingly
## mkdir -p /k3os/delta || echo "/k3os/delta dir exists"
## mount -L K3OS_DELTA /k3os/delta
## #mount overlay dirs (/home, /var/lib/rancher, /var/log, /opt, /var/local), as at this step, fstab entries not yet mounted ?!
## mount -t overlay overlay -o lowerdir=/var,upperdir=/k3os/delta/var,workdir=/k3os/delta/work/var /var
## #mount -t overlay overlay -o lowerdir=/var/lib/rancher,upperdir=/k3os/delta/var/lib/rancher,workdir=/k3os/delta/work/var/lib/## rancher /var/lib/rancher
## #mount -t overlay overlay -o lowerdir=/var/log,upperdir=/k3os/delta/var/log,workdir=/k3os/delta/work/var/log /var/log
## mount -t overlay overlay -o lowerdir=/home,upperdir=/k3os/delta/home,workdir=/k3os/delta/work/home /home
## mount -t overlay overlay -o lowerdir=/opt,upperdir=/k3os/delta/opt,workdir=/k3os/delta/work/opt /opt
## #mount -t overlay overlay -o lowerdir=/var/local,upperdir=/k3os/delta/var/local,workdir=/k3os/delta/work/var/local /var/local

#K3OS persitent dir list
PERS_DIRS="home var opt"
# Delta mountpoint
MNT_PNT="/k3os/delta"
# for test and debug
#MNT_PNT="/tmp/k3os/delta"
# fstab target
FST=/etc/fstab
# for test and debug
FST=/tmp/fstab-test

DELTA_BLK=$(blkid -L K3OS_DELTA)

setup_mount_k3os_delta()
{
    if [ -n "${DELTA_BLK}" ]; then
       # K3OS_DELTA block device found.
       echo "k3OS_DELTA block device found on ${DELTA_BLK} \n"
       echo "   => unification of k3os persistent directories by overlayfs mounting \n"
       echo "      and adding overlay fstab entry. \n"
       #
       mkdir -p ${MNT_PNT}
       mount ${DELTA_BLK} ${MNT_PNT}
       for dir in ${PERS_DIRS}; do 
          # create delta/$dir and delta/work/$dir if not already exist
          mkdir -p ${MNT_PNT}/$dir
          mkdir -p ${MNT_PNT}/work/$dir
          #unify  $dir and delta $dir with overlayfs mount
          mount -t overlay overlay -o lowerdir=/$dir,upperdir=${MNT_PNT}/$dir,workdir=${MNT_PNT}/work/$dir /$dir
          # add corresponding overlay fstab entry
          echo "overlay       /$dir overlay     lowerdir=/$dir,upperdir=${MNT_PNT}/$dir,workdir=${MNT_PNT}/work/$dir 0 0" >> ${FST}
          echo "    - /$dir k3os persistent dir and K3OS_DELTA/$dir unified ! \n"
       done
    fi
}

setup_mount_k3os_delta

Note 1 : Even If that script works when launched in early step at k3os bootup sequence, init_cmd for now, I think that piece of script code will be probably better placed inside the initrd. At the end of the initrd sequence just before pivot chrooting on the real root ???

Note 2 : In parallel I plan to build my cirrus cluster in a IPv6 architecture only, not a dual stack one, only IPv6 !!! From my network engeneer point of view, in 2020 new clusters should have to be IPv6, only ingress loadbalancers/proxies should have to be dual stacked. I don't want any more that old and ugly RFC1918 10.0.0.0/8 and associated NAT/PT stuff in my cluster. But that's another philosphical debate and as I understand k3s still have issues on certain IPv6 webapp endpoints... (see https://github.com/rancher/k3s/issues/1478 and https://github.com/rancher/k3s/issues/1268)

rancher / k3os

[Question] Provide deployment YAML and images trough write_files? #385

Use case

Plan of attack

Question