search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.06k stars 14.08k forks source link

minikube: kvm2 plugin fails due to shared library issue #115878

Open titanous opened 3 years ago

titanous commented 3 years ago

Describe the bug The minikube kvm2 driver does not work at all. This appears to be due to a shared library problem that I do not know how to resolve.

To Reproduce Steps to reproduce the behavior:

  1. minikube start --driver=kvm2 --alsologtostderr
  2. Note the error: 
    libmachine: Error starting plugin binary: fork/exec /home/titanous/.minikube/bin/docker-machine-driver-kvm2: no such file or directory

Expected behavior The kvm2 driver should work.

Additional context

$ ldd .minikube/bin/docker-machine-driver-kvm2
    linux-vdso.so.1 (0x00007fffe1d72000)
    libvirt-qemu.so.0 => not found
    libvirt.so.0 => not found
    libpthread.so.0 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libpthread.so.0 (0x00007ff437c7c000)
    libc.so.6 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libc.so.6 (0x00007ff437abb000)
    /lib64/ld-linux-x86-64.so.2 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib64/ld-linux-x86-64.so.2 (0x00007ff437c9f000)

After setting LD_LIBRARY_PATH=/nix/store/6imd0g7aw49135213pf3s6z92b7q9srz-libvirt-7.0.0/lib it still does not work, but ldd does not complain:

$ ldd .minikube/bin/docker-machine-driver-kvm2
    linux-vdso.so.1 (0x00007ffcea59a000)
    libvirt-qemu.so.0 => /nix/store/6imd0g7aw49135213pf3s6z92b7q9srz-libvirt-7.0.0/lib/libvirt-qemu.so.0 (0x00007ff57a27b000)
    libvirt.so.0 => /nix/store/6imd0g7aw49135213pf3s6z92b7q9srz-libvirt-7.0.0/lib/libvirt.so.0 (0x00007ff579e15000)
    libpthread.so.0 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libpthread.so.0 (0x00007ff579df4000)
    libc.so.6 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libc.so.6 (0x00007ff579c33000)
    libglib-2.0.so.0 => /nix/store/0dvdaqxzvm3qasifkrfcqykckgh7abwz-glib-2.66.4/lib/libglib-2.0.so.0 (0x00007ff579aff000)
    libgobject-2.0.so.0 => /nix/store/0dvdaqxzvm3qasifkrfcqykckgh7abwz-glib-2.66.4/lib/libgobject-2.0.so.0 (0x00007ff579aa6000)
    libgio-2.0.so.0 => /nix/store/0dvdaqxzvm3qasifkrfcqykckgh7abwz-glib-2.66.4/lib/libgio-2.0.so.0 (0x00007ff5798d2000)
    libxml2.so.2 => /nix/store/x1b2xssir8y0282i7qmfqdl3vy717sl1-libxml2-2.9.10/lib/libxml2.so.2 (0x00007ff579763000)
    libaudit.so.1 => /nix/store/zi0h48f1jb88k4xrszg212qld7i3rxcz-audit-2.8.5/lib/libaudit.so.1 (0x00007ff579735000)
    libcap-ng.so.0 => /nix/store/kh0sjbxwq0b3fad350wpx2q4p48z23jj-libcap-ng-0.8.2/lib/libcap-ng.so.0 (0x00007ff57972c000)
    libgnutls.so.30 => /nix/store/1w61bxm2jywck4z598ymrd1676xwh6qj-gnutls-3.6.15/lib/libgnutls.so.30 (0x00007ff579560000)
    libm.so.6 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libm.so.6 (0x00007ff57941d000)
    libnl-3.so.200 => /nix/store/jdhi1vk7svjwzkg7xw2saqrc2zzqhpqa-libnl-3.5.0/lib/libnl-3.so.200 (0x00007ff5793f9000)
    libutil.so.1 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libutil.so.1 (0x00007ff5793f4000)
    libnuma.so.1 => /nix/store/jc9y52i696hv2l02vyf9mnfk453gqfaq-numactl-2.0.14/lib/libnuma.so.1 (0x00007ff5793e3000)
    libapparmor.so.1 => /nix/store/bfjrh208n4dn8vbyacyfybkzkrv3234y-libapparmor-2.13.6/lib/libapparmor.so.1 (0x00007ff5793ce000)
    libyajl.so.2 => /nix/store/yrcs766ipfby60ksajngkwrn7cmljb06-yajl-2.1.0/lib/libyajl.so.2 (0x00007ff5793c2000)
    libssh2.so.1 => /nix/store/52nff18kzg5h4nrbbznh7jwchnaw475a-libssh2-1.9.0/lib/libssh2.so.1 (0x00007ff579381000)
    libtirpc.so.3 => /nix/store/7jlh9bpbdvbqgzfq4cmk28fzbajxwd4d-libtirpc-1.2.7-rc4/lib/libtirpc.so.3 (0x00007ff579351000)
    libcurl.so.4 => /nix/store/8aiy8lz2hj3m514adimkly50bvhj4bbx-curl-7.74.0/lib/libcurl.so.4 (0x00007ff5792b7000)
    libdl.so.2 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libdl.so.2 (0x00007ff5792b0000)
    libgcc_s.so.1 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libgcc_s.so.1 (0x00007ff579296000)
    /lib64/ld-linux-x86-64.so.2 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib64/ld-linux-x86-64.so.2 (0x00007ff57a282000)
    libpcre.so.1 => /nix/store/4clrialdn6i2w6gdsrbpy76qvgdq20vl-pcre-8.44/lib/libpcre.so.1 (0x00007ff57921d000)
    libffi.so.7 => /nix/store/b7ms1xiy6m51xkw6qk1z0qwhxgnwplkv-libffi-3.3/lib/libffi.so.7 (0x00007ff579210000)
    libgmodule-2.0.so.0 => /nix/store/0dvdaqxzvm3qasifkrfcqykckgh7abwz-glib-2.66.4/lib/libgmodule-2.0.so.0 (0x00007ff579207000)
    libz.so.1 => /nix/store/1srmyg1a8cxqwd0hd24rj6kw4lqd61yq-zlib-1.2.11/lib/libz.so.1 (0x00007ff5791ea000)
    libmount.so.1 => /nix/store/6hcqnys5sii21xi821f8yg7mvj7cx5dl-util-linux-2.36.1/lib/libmount.so.1 (0x00007ff57918c000)
    libselinux.so.1 => /nix/store/wm8aiqpj09xlw1h7cqgsiv0m96qn1mww-libselinux-3.0/lib/libselinux.so.1 (0x00007ff57915f000)
    libresolv.so.2 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libresolv.so.2 (0x00007ff579147000)
    libp11-kit.so.0 => /nix/store/c1ss4vfipvqp6k2iywsb3zlgcbm7x2r4-p11-kit-0.23.22/lib/libp11-kit.so.0 (0x00007ff579010000)
    libunistring.so.2 => /nix/store/a8sjzh4lyd9fakbsvbi7gmpnfyasj7hy-libunistring-0.9.10/lib/libunistring.so.2 (0x00007ff578e8d000)
    libtasn1.so.6 => /nix/store/8skdlv4is5dx7k4qv85dky8xqlw56vv0-libtasn1-4.16.0/lib/libtasn1.so.6 (0x00007ff578e77000)
    libnettle.so.8 => /nix/store/0q22n71lh82nv9vr8lzdwm81mwqipaan-nettle-3.6/lib/libnettle.so.8 (0x00007ff578e36000)
    libhogweed.so.6 => /nix/store/0q22n71lh82nv9vr8lzdwm81mwqipaan-nettle-3.6/lib/libhogweed.so.6 (0x00007ff578dec000)
    libgmp.so.10 => /nix/store/mz87xcyjjv0sl6h9bs9d39m85d4qb00n-gmp-6.2.1/lib/libgmp.so.10 (0x00007ff578d48000)
    libssl.so.1.1 => /nix/store/f7jpwsi3yiy98pjv6m06m5j2hxds84cm-openssl-1.1.1j/lib/libssl.so.1.1 (0x00007ff578cb2000)
    libcrypto.so.1.1 => /nix/store/f7jpwsi3yiy98pjv6m06m5j2hxds84cm-openssl-1.1.1j/lib/libcrypto.so.1.1 (0x00007ff5789c5000)
    libgssapi_krb5.so.2 => /nix/store/gi12qahl7d6h26sgj04mm89gimkanf6q-libkrb5-1.18/lib/libgssapi_krb5.so.2 (0x00007ff578972000)
    libkrb5.so.3 => /nix/store/gi12qahl7d6h26sgj04mm89gimkanf6q-libkrb5-1.18/lib/libkrb5.so.3 (0x00007ff578896000)
    libk5crypto.so.3 => /nix/store/gi12qahl7d6h26sgj04mm89gimkanf6q-libkrb5-1.18/lib/libk5crypto.so.3 (0x00007ff578866000)
    libcom_err.so.3 => /nix/store/gi12qahl7d6h26sgj04mm89gimkanf6q-libkrb5-1.18/lib/libcom_err.so.3 (0x00007ff578860000)
    libnghttp2.so.14 => /nix/store/wa8kywmn2aywkrj8n7kir4j90x64cjib-nghttp2-1.41.0-lib/lib/libnghttp2.so.14 (0x00007ff578834000)
    libblkid.so.1 => /nix/store/6hcqnys5sii21xi821f8yg7mvj7cx5dl-util-linux-2.36.1/lib/libblkid.so.1 (0x00007ff5787e0000)
    librt.so.1 => /nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/librt.so.1 (0x00007ff5787d5000)
    libkrb5support.so.0 => /nix/store/gi12qahl7d6h26sgj04mm89gimkanf6q-libkrb5-1.18/lib/libkrb5support.so.0 (0x00007ff5787c5000)
    libkeyutils.so.1 => /nix/store/6qqgmghykh5z6zz2apj0j94rrk3xwwdj-keyutils-1.6.3-lib/lib/libkeyutils.so.1 (0x00007ff5787be000)

Notify maintainers @ebzzry @copumpkin @vdemeester @atkinschang @Chili-Man

Metadata

justinas commented 3 years ago

The problem is likely the fact that minikube downloads a pre-built binary (docker-machine-driver-kvm2), and unpatched binaries usually don't run on NixOS out-of-the-box.

$ file docker-machine-driver-kvm2
docker-machine-driver-kvm2: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, Go BuildID=2Ky3_TJwhkFqdZ3ZeXlJ/mc6R2KhI-SnHU-Ibwb6f/FCnHVcUO9u5hsmE0hjlr/2cDp9WtsFrI4hwsChrCn, BuildID[sha1]=ed7fffed63cf1d6804a8c11c373c65f18178ed45, not stripped

The binary has the interpreter set to /lib64/ld-linux-x86-64.so.2, which does not exist under NixOS.

I imagine a true solution would be something similar to what is done for e.g. Terraform, where providers are built with Nix, rather than blobs downloaded by the tool at runtime.

You could try using patchelf, as suggested in the wiki article, on ~/.minikube/bin/docker-machine-driver-kvm2.

AtkinsChang commented 3 years ago

You should install pkgs.docker-machine-kvm2, minikube will use system binary instead of download it.

workflow commented 2 years ago

Installing pkgs.docker-machine-kvm2 didn't work for me (minikube still fetching its own unpatched binary), so I manually removed ~/.minikube/bin/docker-machine-driver-kvm2 and symlinked it to /run/current-system/sw/bin/docker-machine-driver-kvm2

 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.43, NixOS, 22.05 (Quokka), 22.05pre-git`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - channels(root): `""`
 - nixpkgs: `/nix/store/z1lxp4xlzkigxjj9vpjnfva2vim8x1zm-source`

If using home-manager, you could do:

home.file = {
      # Patch Minikube kvm2 driver, see https://github.com/NixOS/nixpkgs/issues/115878
      ".minikube/bin/docker-machine-driver-kvm2".source = "${pkgs.docker-machine-kvm2}/bin/docker-machine-driver-kvm2";
};
cyntheticfox commented 2 years ago

Would it be possible to create a wrapper for this then?