search

innobead / kubefire

KubeFire 🔥, creates and manages Kubernetes Clusters using Firecracker microVMs
Apache License 2.0
511 stars 36 forks source link

kubefire install: failed to run script (install-prerequisites.sh): exit status 1 on WSL2 #97

Open ghost opened 6 months ago

ghost commented 6 months ago

Describe the bug

WSL2 can run Firecracker without any modification of underlying WSL2 Kernel. When I run $ kubefire install it produces an FATAL error and could not install (or prepare) kubefire.

To Reproduce

Version: v0.3.10
Build: 35cb33b-20230416001838
kubefire install --log-level trace
DEBU[2024-05-19T16:34:06+09:00]       di.go:46   github.com/innobead/kubefire/internal/di.DelayInit() initializing dependency injection system
DEBU[2024-05-19T16:34:06+09:00]       di.go:119  github.com/innobead/kubefire/internal/di.DelayInit() completed dependency injection system
INFO[2024-05-19T16:34:06+09:00]   script.go:57   github.com/innobead/kubefire/pkg/script.Download() downloading https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites.sh to save /home/seungwoo/.kubefire/bin/v0.3.10/install-prerequisites.sh  force=true version=v0.3.10
INFO[2024-05-19T16:34:06+09:00]   script.go:77   github.com/innobead/kubefire/pkg/script.Run() running script (install-prerequisites.sh)     version=v0.3.10
INFO[2024-05-19T16:34:06+09:00]   script.go:81   github.com/innobead/kubefire/pkg/script.Run() running /home/seungwoo/.kubefire/bin/v0.3.10/install-prerequisites.sh  version=v0.3.10
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + TMP_DIR=/tmp/kubefire
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() ++ uname -m
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + ARCH=x86_64
+ KUBEFIRE_VERSION=v0.3.10
+ CONTAINERD_VERSION=v1.6.20
+ IGNITE_VERION=v0.10.0
+ CNI_VERSION=v1.1.1
+ RUNC_VERSION=v1.1.6
+ '[' -z v0.3.10 ']'
+ '[' -z v1.6.20 ']'
+ '[' -z v0.10.0 ']'
+ '[' -z v1.1.1 ']'
+ '[' -z v1.1.6 ']'
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() ++ sed -E 's/(v[0-9]+\.[0-9]+\.[0-9]+)[a-zA-Z0-9\-]*/\1/g'
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() +++ echo v0.3.10
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + STABLE_KUBEFIRE_VERSION=v0.3.10
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + '[' x86_64 = aarch64 ']'
+ '[' x86_64 = arm64 ']'
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + '[' x86_64 = x86_64 ']'
+ ARCH_SUFFIX=amd64
+ rm -rf /tmp/kubefire
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + mkdir -p /tmp/kubefire
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + pushd /tmp/kubefire
/tmp/kubefire /home/seungwoo
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + trap cleanup EXIT ERR INT TERM
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + check_virtualization
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + '[' amd64 = arm64 ']'
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + lscpu
+ grep 'Virtuali[s|z]ation'
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() Virtualization:                     VT-x
Virtualization type:                full
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + lsmod
+ grep kvm
INFO[2024-05-19T16:34:06+09:00]      log.go:44   github.com/innobead/kubefire/pkg/util.LogWriter.Write() + cleanup
+ rm -rf /tmp/kubefire
Usage:
  kubefire install [flags]

Aliases:
  install, in

Flags:
  -f, --force   Force to install
  -h, --help    help for install

Global Flags:
  -t, --github-token string   GIthub Personal Access Token used to query repo release info
  -l, --log-level string      log level, options: [panic, fatal, error, warning, info, debug, trace] (default "info")

TRAC[2024-05-19T16:34:06+09:00] kubefire.go:74   main.main() exit status 1
github.com/innobead/kubefire/pkg/script.runScript
        /home/runner/work/kubefire/kubefire/pkg/script/script.go:145
github.com/innobead/kubefire/pkg/script.Run
        /home/runner/work/kubefire/kubefire/pkg/script/script.go:82
github.com/innobead/kubefire/cmd/kubefire/cmd.glob..func6
        /home/runner/work/kubefire/kubefire/cmd/kubefire/cmd/install.go:32
github.com/spf13/cobra.(*Command).execute
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068
github.com/spf13/cobra.(*Command).Execute
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
main.main
        /home/runner/work/kubefire/kubefire/cmd/kubefire/kubefire.go:73
runtime.main
        /opt/hostedtoolcache/go/1.20.3/x64/src/runtime/proc.go:250
runtime.goexit
        /opt/hostedtoolcache/go/1.20.3/x64/src/runtime/asm_amd64.s:1598
failed to run script (install-prerequisites.sh)
FATA[2024-05-19T16:34:06+09:00] kubefire.go:75   main.main() failed to run kubefire                        error="failed to run script (install-prerequisites.sh): exit status 1"

Expected behavior Kubefire should've install without any issue.

Environment

PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
kubefire version
Version: v0.3.10
Build: 35cb33b-20230416001838
kubefire info
CONTAINERD              IGNITE                  CNI                     RUNC                            CNIPLUGIN
 (expected: v1.6.20)     (expected: v0.10.0)     (expected: v1.1.1)     1.1.12 (expected: 1.1.6)         (expected: 0.4.0/kubefire-cni-bridge)

Additional context

None.

ghost commented 6 months ago

Seems like the install-prerequisites.sh (https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites.sh) is the issue.

on function check_virtualization, it checks lsmod | grep kvm the ensure a kvm is available on the host. But the kvm is statically enabled on WSL2 Kernel Config

Seems like it can easily fix, If kubefire install provides a optional flags to specify location of the script.

ghost commented 6 months ago

Seems like the install-prerequisites.sh (https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites.sh) is the issue.

on function check_virtualization, it checks lsmod | grep kvm the ensure a kvm is available on the host. But the kvm is statically enabled on WSL2 Kernel Config

Seems like it can easily fix, If kubefire install provides a optional flags to specify location of the script.

➜  ~ lsmod | grep kvm
➜  ~ echo $?
1
ghost commented 6 months ago

Seems like the install-prerequisites.sh (https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites.sh) is the issue. on function check_virtualization, it checks lsmod | grep kvm the ensure a kvm is available on the host. But the kvm is statically enabled on WSL2 Kernel Config Seems like it can easily fix, If kubefire install provides a optional flags to specify location of the script.

➜  ~ lsmod | grep kvm
➜  ~ echo $?
1

I've slightly modified https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites.sh script to skip if the host is WSL2 via check Microsoft is exists on /proc/version, install completed.

function check_virtualization() {
  if [ ${ARCH_SUFFIX} = arm64 ]; then
    return
  fi
  # If Microsoft is included on "/proc/version"
  # do nothing
  if [[ $(grep -i Microsoft /proc/version) ]]; then
    return
  fi

  lscpu | grep "Virtuali[s|z]ation"
  lsmod | grep kvm
}
ghost commented 6 months ago

and I've encountered another issues on create a new cluster via kubefire cluster create demo

kubefire cluster create demo --log-level debug
DEBU[2024-05-19T17:10:13+09:00] initializing dependency injection system
DEBU[2024-05-19T17:10:13+09:00] completed dependency injection system
DEBU[2024-05-19T17:10:13+09:00] forcibly reinitializing dependency injection system
DEBU[2024-05-19T17:10:13+09:00] completed dependency injection system
DEBU[2024-05-19T17:10:13+09:00] getting the latest released version info      bootstrapper=kubeadm
DEBU[2024-05-19T17:10:13+09:00] getting bootstrapper version configurations   bootstrapper=kubeadm
DEBU[2024-05-19T17:10:13+09:00] getting the latest released version info      bootstrapper=kubeadm
INFO[2024-05-19T17:10:14+09:00] initializing cluster configuration            cluster=demo
DEBU[2024-05-19T17:10:14+09:00] &{Name:demo Bootstrapper:kubeadm Pubkey: Prikey: Version:v1.30.1 Image:ghcr.io/innobead/kubefire-opensuse-leap:15.2 KernelImage:ghcr.io/innobead/kubefire-ignite-kernel:4.19.125-amd64 KernelArgs:console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1 ExtraOptions:map[] Deployed:false Master:{Count:1 Memory:2GB Cpus:2 DiskSize:10GB Cluster:0xc000180a00} Worker:{Count:0 Memory:2GB Cpus:2 DiskSize:10GB Cluster:0xc000180a00}}
DEBU[2024-05-19T17:10:14+09:00] getting cluster configurations                cluster=demo
INFO[2024-05-19T17:10:14+09:00] saving cluster configurations                 cluster=demo
INFO[2024-05-19T17:10:14+09:00] creating cluster                              cluster=demo
DEBU[2024-05-19T17:10:14+09:00] getting cluster configurations                cluster=demo
INFO[2024-05-19T17:10:14+09:00] creating master nodes of cluster              cluster=demo started=true
INFO[2024-05-19T17:10:14+09:00] creating node                                 node=demo-master-1
INFO[2024-05-19T17:10:14+09:00] time="2024-05-19T17:10:14+09:00" level=info msg="Removed VM with name \"demo-master-1\" and ID \"e331a2160f2c48d5\""
time="2024-05-19T17:10:14+09:00" level=fatal msg="command [\"dmsetup\" \"create\" \"--verifyudev\" \"ignite-e331a2160f2c48d5-base\"] exited with \"device-mapper: reload ioctl on ignite-e331a2160f2c48d5-base (252:0) failed: Invalid argument\\nCommand failed.\\n\": exit status 1"
ERRO[2024-05-19T17:10:14+09:00] failed to create node                         error="exit status 1" node=demo-master-1
DEBU[2024-05-19T17:10:14+09:00] getting cluster                               cluster=demo
DEBU[2024-05-19T17:10:14+09:00] getting cluster configurations                cluster=demo
DEBU[2024-05-19T17:10:14+09:00] listing nodes of cluster                      cluster=demo
DEBU[2024-05-19T17:10:14+09:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=~demo- -t {{.ObjectMeta.Name}}]
INFO[2024-05-19T17:10:14+09:00] waiting nodes of cluster running              cluster=demo
DEBU[2024-05-19T17:10:14+09:00] listing nodes of cluster                      cluster=demo
DEBU[2024-05-19T17:10:14+09:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=~demo- -t {{.ObjectMeta.Name}}]
INFO[2024-05-19T17:10:14+09:00] initializing cluster                          cluster=demo
DEBU[2024-05-19T17:10:14+09:00] getting the latest released version info      bootstrapper=kubeadm
DEBU[2024-05-19T17:10:14+09:00] getting bootstrapper version configurations   bootstrapper=kubeadm
INFO[2024-05-19T17:10:14+09:00] waiting all nodes initialization finished
DEBU[2024-05-19T17:10:14+09:00] getting node                                  node=demo-master-1
DEBU[2024-05-19T17:10:14+09:00] VM ID   IMAGE   KERNEL  SIZE    CPUS    MEMORY  CREATED STATUS  IPS     PORTS   NAME
DEBU[2024-05-19T17:10:14+09:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.ObjectMeta.Labels.cluster}}]
Usage:
  kubefire cluster create [name] [flags]

Flags:
  -b, --bootstrapper string    Bootstrapper type, options: [kubeadm, k3s, rke2, k0s] (default "kubeadm")
  -c, --config string          Cluster configuration file (ex: use 'config-template' command to generate the default cluster config)
  -o, --extra-options string   Extra options (ex: key=value,...) for bootstrapper
  -f, --force                  Force to recreate if the cluster exists
  -h, --help                   help for create
  -i, --image string           Rootfs container image (default "ghcr.io/innobead/kubefire-opensuse-leap:15.2")
      --kernel-args string     Kernel arguments (default "console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1")
      --kernel-image string    Kernel container image (default "ghcr.io/innobead/kubefire-ignite-kernel:4.19.125-amd64")
      --master-count int       Count of master node (default 1)
      --master-cpu int         CPUs of master node (default 2)
      --master-memory string   Memory of master node (default "2GB")
      --master-size string     Disk size of master node (default "10GB")
      --no-cache               Forget caches
      --no-start               Don't start nodes
  -k, --pubkey string          Public key
  -v, --version string         Version of Kubernetes supported by bootstrapper (ex: v1.18, v1.18.8, empty)
      --worker-count int       Count of worker node
      --worker-cpu int         CPUs of worker node (default 2)
      --worker-memory string   Memory of worker node (default "2GB")
      --worker-size string     Disk size of worker node (default "10GB")

Global Flags:
  -t, --github-token string   GIthub Personal Access Token used to query repo release info
  -l, --log-level string      log level, options: [panic, fatal, error, warning, info, debug, trace] (default "info")

FATA[2024-05-19T17:10:15+09:00] failed to run kubefire                        error="failed to deploy cluster (demo): demo-master-1 node unavailable"

seems like the error is occurred from executing dmsetup.