Guidance for setting up RKE on arm64.. specifically on Raspberry Pi 4 nodes

[wolfspyre]

Hi there!

First, Thank you for spending so much time and effort on your documentation. It really helps, and the love y'all put into the documentation shows.

I'd love to see a soup-to-nuts journey documented to getting a highly available rancher cluster and workload cluster up and running on a small herd of Raspberry Pis.

some of the things I encountered while trying this that I didn't see documented coherently anyplace:

• steer users to use the 64 bit images ubuntu releases for Pi found here: https://ubuntu.com/download/raspberry-pi unless y'all are planning on releasing a happy path using ros/k3os/whatever images targeting 64bit pis
  • Raspbian user land is 32bit, even with a 64bit kernel, you're likely to run into edge-case problems that nobody'll like and much pain will be experienced by all.
  • Encourage users to setup the pi eeprom/hardware using raspbian, and then use ubuntu for the base OS.
  • enabling the memory cgroup in ubuntu isn't quite straight forward. You are likely to encounter issues, or ay least unrelenting log noise if you don't enable it. checking /proc/cmdline shows its not enabled by default.. Looking at /boot/firmware/config.txt points you to where the boot args are defined:

root@rkepi-01:~# grep cmdline /boot/firmware/config.txt
cmdline=nobtcmd.txt

which points you to where you need to append the needful: cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

root@rkepi-01:~# cat /boot/firmware/nobtcmd.txt
net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc

So simply append the needful to that line and reboot, and viola:

root@rkepi-01:~# cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=1 cma=64M bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DE:AD:BE:EF:00:00 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
root@rkepi-01:~#

• I didn't see mention anywhere in the docs of adjusting sysctls, by default, ip_forwarding is not enabled.. adjusting /etc/sysctl.d/99-sysctl.conf and uncommenting/enabling the sysctl net.ipv4.ip_forward=1 should do the trick.

• I don't think this is arm/pi specific, but I've found it helpful to use the "log-driver": "journald" option in /etc/docker/daemon.json so as to get docker logs included with all the rest of the systemically available logs via journalctl -fm

• I'd strongly advise encouraging a setup of using a usb->sata adapter and having an ssd for log/docker/etcd.. etcd is noisy, an all those writes will kill a microsd card pretty damned quick...

  • my solution was to use a 64g ssd, 10G for /var/log, 54G for /srv and relocating the /var/lib/docker, /var/lib/etcd, /var/lib/containerd dirs to directories you've created in /srv (or wherever else you choose to mount it at) via something like (untested, typed live) :

export _newmp='srv';
for _basedir in '/var/lib/docker' '/var/lib/etcd' '/var/lib/containerd' '/var/lib/kubelet' '/var/lib/rancher' '/opt'; do
    _newtarget="/${_newmp}/${_basedir}"
    if  [[ -d ${_newtarget} ]]; then
        echo "${_newtarget} directory exists. Not creating."
    else
       echo " ${_newtarget} needs creating. doing the needful.";
       mkdir -p "${_newtarget}"; 
    fi
    if [[ ! -d ${_basedir} ]]; then
      echo "${_basedir} is not a directory, not copying content"; 
    else
       echo "========== relocating content in ${_basedir} to ${_newtarget}";
       echo;
       cd "${_basedir}"; tar cpf - . |( cd "${_newtarget}" && tar xpvf - )
       echo "moving ${_basedir} out of the way and symlinking"
       mv "${_basedir}"  "${_basedir}.relocated"
       ln -s "${_newtarget}" "${_basedir}"
       echo "${_basedir}.relocated">> /tmp/shuffledirs.scratch
    fi
done
echo "mount points shuffled. you'll want to remove the original directories listed below after rebooting"
cat /tmp/shuffledirs.scratch
rm /tmp/shuffledirs.scratch

[btat]

Moved to correct repo now.

[wolfspyre]

uhm….

hi… so….. does experimental mean something other than “work-in-progress, not for production use”?

does it not warrant guidance? or at least pointing out known pitfalls? if you’re not interested in contributing to the documentation on it, that’s one thing.

closing a request for documentation of current state because something is experimental is like saying

R&D doesn’t MAKE things, so they don’t need any resources….

[btat]

uhm….

hi… so….. does experimental mean something other than “work-in-progress, not for production use”?

does it not warrant guidance? or at least pointing out known pitfalls? if you’re not interested in contributing to the documentation on it, that’s one thing.

closing a request for documentation of current state because something is experimental is like saying

R&D doesn’t MAKE things, so they don’t need any resources….

Hi @wolfspyre, apologies I added the wrong comment to this one while transferring issues. It's reopened and moved to the correct repo now.

[wolfspyre]

Hi Billy! Thanks a ton!! Not trying to be a jerk or anything.... I just want to help others avoid unnecessary hairpulling and forehead banging, y'know? IMO ...The only thing worse than no documentation, is BAD documentation....

I hope your day's going wonderfully. ❤️🐺W