cluster creation fails with "Failed to mount API filesystems, freezing"

[mdaniel]

What happened:

$ eksctl anywhere generate clusterconfig eks0 --provider docker --verbosity 10000 > eks0.yaml 2>eks0.log
$ eksctl anywhere create cluster -f eks0.yaml --verbosity 10000 2>&1| tee -a eks0.log
...
Error: failed to create cluster: error creating bootstrap cluster: error executing create cluster: exit status 1, try rerunning with --force-cleanup to force delete previously created bootstrap cluster

Only by looking separately at the `` container logs does one see the actual failure:

$ docker logs $(docker ps -qf label=io.x-k8s.kind.role=control-plane)
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from VM UUID.
INFO: faking /sys/class/dmi/id/product_name to be "kind"
INFO: faking /sys/class/dmi/id/product_uuid to be random
INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
INFO: setting iptables to detected mode: legacy
INFO: Detected IPv4 address: 172.18.0.2
INFO: Detected IPv6 address: fc00:f853:ccd:e793::2
[!!!!!!] Failed to mount API filesystems, freezing.

What you expected to happen:

Cluster cluster creation completes successfully

How to reproduce it (as minimally and precisely as possible):

1. create the following Vagrantfile
2. vagrant up
3. due to the embedded while loop in the provisioning script, there will be a separate process that emits the logs of the eks0-eks-a-cluster container for your convenience

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/impish64"
  config.vm.network "private_network", ip: "192.168.33.105"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "4098"
  end
  config.vm.provision "shell", inline: <<-SHELL
    set -ex
    export DEBIAN_FRONTEND=noninteractive
    if ! command -v gnupg &>/dev/null; then
      apt-get update

      apt-get install -y \
        ca-certificates \
        curl \
        gnupg \
        lsb-release
    fi
    krf=/usr/share/keyrings/docker-archive-keyring.gpg
    if [ ! -e $krf ]; then
      curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o $krf
    fi
    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=$krf] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" >/etc/apt/sources.list.d/docker.list
    apt-get update
    apt-get install -y docker-ce docker-ce-cli containerd.io

    docker version

    if [ ! -e /usr/local/bin/eksctl ]; then
      if [ ! -e /tmp/eksctl ]; then
        curl -fsSL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
        | tar -xz -C /tmp
      fi
      cp /tmp/eksctl /usr/local/bin
    fi
    if [ ! -e /usr/local/bin/eksctl-anywhere ]; then
      EKSA_RELEASE="0.6.0" OS="$(uname -s | tr A-Z a-z)" RELEASE_NUMBER=2
      curl -fsSL "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/${RELEASE_NUMBER}/artifacts/eks-a/v${EKSA_RELEASE}/${OS}/eksctl-anywhere-v${EKSA_RELEASE}-${OS}-amd64.tar.gz" \
      | tar -xzC /usr/local/bin ./eksctl-anywhere
    fi

    eksctl anywhere version
    # since the images are huge, load them if available
    cd ~vagrant
    for i in /vagrant/*.tar ./*.tar; do
      [ -e "$i" ] || continue
      docker load -i "$i"
    done
    eksctl anywhere generate clusterconfig eks0 --provider docker --verbosity 10000 > eks0.yaml 2>eks0.log
    bash -c 'while true; do cid=$(docker ps -qf label=io.x-k8s.kind.role=control-plane || true); if [ -z "$cid" ]; then sleep 2; continue; fi; exec docker logs -f $cid; done' &
    eksctl anywhere create cluster -f eks0.yaml --verbosity 10000 2>&1| tee -a eks0.log
  SHELL
end

Anything else we need to know?:

A cursory search for that error message made it seem it is related to trying to run systemd inside docker: https://stackoverflow.com/questions/64349278/unable-to-start-systemd-container-using-docker-centos-7-8-host-failed-to-moun

Environment:

• EKS Anywhere Release: v0.6.0
• EKS Distro Release: unknown, presumably v1.21.2-eks-d-1-21-6-eks-a-4

[jaxesn]

~Do you know what version of Ubuntu that is you are using? Is it 2204?~

Sorry i see it's 21.10. I think the issue is going to be because of that OS/kernel configured to use Cgroup v2 (correct me if you know better). We recently ran into this due to a docker for max upgrade which switch the underlying vm to be Cgroup v2 based, #789. If that's the case we do not currently have a workaround.

Would you mind trying making your vm based on 20.04(focal) just to confirm?

[mdaniel]

Yes, for sure my kernel is using cgroups v2 because I had to upgrade the docker daemon after upgrading Ubuntu due to an explosion related to that

I tried sniffing around the monster number of disparate eks-distro{,-build-tooling} and eks-anywhere{,-build-tooling} repos and directories looking for what those docker images are doing with systemd, without success. What are those AL2 images doing that requires systemd?

I'll try to remember to fire up a fresh copy of kind when I get back to work in order to find out if kind is broken in the same way

[jaxesn]

Yeah that's the issue. We don't use the upstream kind images in our bootstrap/docker provider containers which are based on Ubuntu. Instead we have an al2 based kind image, mainly because we try to standardized on al2 as much as possible.

The version of systemd, which runs in the kind container to run containerd and other services, is old and it doesn't support Cgroup v2. Since your host is configured with v2, when the kind container launches and systemd tries to mount the various cgroups it ends up bailing out because it's not what it expects.

Unfortunately there really is no workaround for now. Al2022 is due out early next year and we will definitely upgrade. In the time being using a VM configured to use Cgroup v1 would be the best option. We will also watch for more feedback from others and decide if we need to so something short term while waiting on al2022.

[jselleck-sans]

FYI - I'm also running into the same issue.

[mdaniel]

I'm sorry that I missed this bug's birthday, but it appears al2022 is out, seemingly supported by the release notes which I think used to say "preview" but no longer does

I tried v0.13.0 earlier and actually forgot about this ancient cgroups bug but was instantly reminded of it when the cluster failed to come alive

I do see the "backlog" label, but I thought I understood that when AL2002 came out the plan was to "definitely upgrade"

Are you still watching for feedback from others?

[jaxesn]

Unfortunately Al22 has not GA'd yet. The current plan is sometime next year. We are watching that closely and as soon as they officially launch we will look at upgrading our kind images.