search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.6k stars 4.89k forks source link

Recently built minikube fails to start with vmware driver #16221

Closed markpeek closed 1 year ago

markpeek commented 1 year ago

What Happened?

Build both minikube and the amd64 ISO with recent changes and it fails to start with the vmware driver.

$ ./minikube-darwin-amd64 start --driver vmware --iso-url=file://$(pwd)/minikube-amd64.iso
😄  minikube v1.30.0 on Darwin 13.2.1
✨  Using the vmware driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  397.02 MiB / 397.02 MiB  100.00% 53.18 M
🔥  Creating vmware VM (CPUs=2, Memory=6000MB, Disk=20000MB) .../ ^C

From the logs this appears to be due to the docker user account being removed. Looking at commit history, it was removed via Disable plain text login for ISO by @medyagh.

Attach the log file


I0403 10:43:40.090076   20589 main.go:141] libmachine: (minikube) DBG | IP found in DHCP lease table: 192.168.111.164
I0403 10:43:40.090091   20589 main.go:141] libmachine: (minikube) DBG | Got an ip: 192.168.111.164
I0403 10:43:40.090737   20589 main.go:141] libmachine: (minikube) DBG | Creating Tar key bundle...
I0403 10:43:40.091232   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser directoryExistsInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /var/lib/boot2docker
- I0403 10:43:40.426605   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:40.429937   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser CopyFileFromHostToGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /Users/mark/.minikube/machines/minikube/userdata.tar /home/docker/userdata.tar
/ I0403 10:43:40.690751   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:40.694008   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser runScriptInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /bin/sh sudo sh -c "tar xvf /home/docker/userdata.tar -C /home/docker > /var/log/userdata.log 2>&1 && chown -R docker:staff /home/docker"
\ I0403 10:43:40.954608   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:40.957541   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser runScriptInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /bin/sh sudo /bin/mv /home/docker/userdata.tar /var/lib/boot2docker/userdata.tar
- I0403 10:43:41.219867   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
I0403 10:43:41.223214   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser enableSharedFolders /Users/mark/.minikube/machines/minikube/minikube.vmx
/ I0403 10:43:41.497243   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser addSharedFolder /Users/mark/.minikube/machines/minikube/minikube.vmx Users /Users
| I0403 10:43:41.777151   20589 main.go:141] libmachine: (minikube) DBG | executing: /Users/mark/bin/vmrun -gu docker -gp tcuser runScriptInGuest /Users/mark/.minikube/machines/minikube/minikube.vmx /bin/sh [ ! -d /hosthome ]&& sudo mkdir /hosthome; sudo mount --bind /mnt/hgfs//hosthome /hosthome || [ -f /usr/local/bin/vmhgfs-fuse ]&& sudo /usr/local/bin/vmhgfs-fuse -o allow_other .host:/Users /hosthome || sudo mount -t vmhgfs -o uid=$(id -u),gid=$(id -g) .host:/Users /hosthome
- I0403 10:43:42.041025   20589 main.go:141] libmachine: (minikube) DBG | Error: Invalid user name or password for the guest OS
- ```

### Operating System

macOS (Default)

### Driver

VMware
afbjorklund commented 1 year ago

Sorry about that, it needs to use the ssh key as provided in the image.

Error: Invalid user name or password for the guest OS

It seems to be created correctly (in the tar), in the vmware driver:

https://github.com/machine-drivers/docker-machine-driver-vmware/blob/v0.1.5/pkg/drivers/vmware/driver.go#L624

afbjorklund commented 1 year ago

The automounter will pick the information up, from the disk:

https://github.com/kubernetes/minikube/blob/master/deploy/iso/minikube-iso/package/automount/minikube-automount

After that, the ssh key is supposed to be installed and work.

/home/docker/.ssh/authorized_keys

markpeek commented 1 year ago

Thank you for the pointers and confirming the regression. Noted about needing to rewrite the vmware driver to support injecting the SSH key via cloud-init.

The use of the docker login appears to be inherited from the older Docker machine vmwarefusion driver to inject the SSH keys. As the docker login was documented in the original boot2docker project, are there any other machine drivers that rely on this same behavior that might be broken with the minikube 1.30 release?

afbjorklund commented 1 year ago

are there any other machine drivers that rely on this same behavior that might be broken with the minikube 1.30 release?

it is possible, but I couldn't find any use of "password" in the other drivers from a casual lookthrough right now (grep).

    // Register all of the drvs we know of
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/docker"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/hyperkit"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/hyperv"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/kvm2"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/none"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/parallels"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/podman"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/qemu2"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/ssh"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/virtualbox"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/vmware"
    _ "k8s.io/minikube/pkg/minikube/registry/drvs/vmwarefusion"

You are right that it was inherited from boot2docker, which in turn inherited the login from tinycore (thus the T.C.)

afbjorklund commented 1 year ago

Noted about needing to rewrite the vmware driver to support injecting the SSH key via cloud-init.

Either using cloud-init and the user-data, or through the old docker-machine approach with tar ?

There is no support for cloud-init in minikube, so I guess that would be mostly for docker-machine ?

The ubuntu cloud images should come with support, for adding the user and the key with config.

medyagh commented 1 year ago

VMWare is not our officially supported/tested driver but it seems like this can be fixed by adding it to the machine driver lib. I would accept and review any PR that would fix this issue.

till then the only solution for vmware drivers would be downgrading to an older minikube version.

marshalc commented 1 year ago

You might want to update the docs to reflect this unsupported status - https://minikube.sigs.k8s.io/docs/drivers/vmware/ . I'd just gone through the guide to use this, only to find at the cli that the unsupported message came up.

lbogdan commented 1 year ago

I'm interested in fixing this.

@afbjorklund @medyagh Please correct me if I'm wrong, but from what I understand, with only looking at the error logs in the issue description and reading the comments, the issue is with provisioning /home/docker/.ssh/authorized_keys inside a freshly created VM, where the VMware driver would use vmrun -gu docker -gp tcuser ... to run operations inside the VM, but in the latest ~boot2docker~ minikube ISO image, the password for the docker user was disabled, so running vmrun to provision authorized_keys fails.

That being said, can you please explain the approach / point me to the code of a working driver where the authorized_keys provisioning is done? Thanks!

lbogdan commented 1 year ago

I was also thinking, instead of disabling the password in the ISO, can't we keep it, and disable it first thing, as part of VM initialization?

afbjorklund commented 1 year ago

That being said, can you please explain the approach / point me to the code of a working driver where the authorized_keys provisioning is done? Thanks!

It is inserted into the disk image, like so: https://github.com/kubernetes/minikube/blob/v1.30.1/pkg/drivers/qemu/qemu.go#L643

Then it is extracted, during the automount: https://github.com/kubernetes/minikube/blob/v1.30.1/deploy/iso/minikube-iso/package/automount/minikube-automount#L32

afbjorklund commented 1 year ago

@lbogdan: I made PR to move the external driver in

That should make it faster to iterate, since it doesn't need an external docker-machine-driver-vmware

The last release from https://github.com/machine-drivers/docker-machine-driver-vmware was from 2021

afbjorklund commented 1 year ago

I think it should be possible to format and mount the vmdk, with something like FAT32 similar to a cidata.iso

Then you could extract the userdata.tar from this temporary partition, before reformatting the whole "disk"?

lbogdan commented 1 year ago

So basically the automounter expects the first 4096 bytes of the raw partition to be "boot2docker, please format-me" followed by a tar stream containing .ssh/authorized_keys[2] (any reason for the authorized_keys2?)?

Unfortunately, I don't think we can doctor a .vmdk image this way with just off the shelf Windows / macOS tooling.

afbjorklund commented 1 year ago

Unfortunately, I don't think we can doctor a .vmdk image this way with just off the shelf Windows / macOS tooling.

No, it would need a separate code path. Then again, it seemed like it already had one ? (in the automounter, that is)

afbjorklund commented 1 year ago

any reason for the authorized_keys2?

Ancient history, I think. Deprecated since 3.0 (2001)

lbogdan commented 1 year ago

Then again, it seemed like it already had one ? (in the automounter, that is)

Yeah, but that's assuming the authorized_keys provisioning is handled separately, which is done in the VMware driver here: https://github.com/machine-drivers/docker-machine-driver-vmware/blob/b30963f81e6e25743b42d469a22a5a9f5db742fb/pkg/drivers/vmware/driver.go#L262-L332 .

Note that is supports a cloud-init config drive (if d.ConfigDriveURL != "" {), but even so, it still SSHes into the VM using a password to copy over authorized_keys: https://github.com/machine-drivers/docker-machine-driver-vmware/blob/b30963f81e6e25743b42d469a22a5a9f5db742fb/pkg/drivers/vmware/driver.go#L650-L659 .

What about my other suggestion?

I was also thinking, instead of disabling the password in the ISO, can't we keep it, and disable it first thing, as part of VM initialization?

lbogdan commented 1 year ago

Hm, I might have found a way to create a .vmdk disk with the automounter expected format:

It's a bit convoluted, but it should work.

lbogdan commented 1 year ago

Did the whole process manually, attached the disk to an existing VM, and lo and behold, it works!

# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
[...]
sdb       8:16   0   10G  0 disk

# dd if=/dev/sdb bs=1 count=29
boot2docker, please format-me29+0 records in
29+0 records out
29 bytes copied, 0.000151799 s, 191 kB/s
lbogdan commented 1 year ago

@afbjorklund So where should this be fixed? Even with #16687 , it looks like the changes would still have to go to upstream https://github.com/machine-drivers/docker-machine-driver-vmware ?

afbjorklund commented 1 year ago

Well, it would have to go to a fork of that repo. If you want to go all-in (for developing), you could copy stuff:

https://github.com/kubernetes/minikube/pull/16688/commits/e4ccde5dd11f907b2f76c9affa116b2763314edb

We want to drop the dependency on docker-machine, so that we are free to evolve the "libmachine" API So there is a work-in-progress, to move to a separate entity called "minikube-machine" to offer libmachine

The code/binary in docker-machine-driver-vmware, is supposed to work with both docker-machine and minikube That would of course also be a way of testing it, to run with docker-machine create -d vmware and the binary

ErlinDez commented 1 year ago

@lbogdan how do you go about writing the tar stream to the beginning of test-flat.vmdk ?

lbogdan commented 1 year ago

I did that manually, but I guess the same as overwriting (part of) a file? Open, seek to the beginning, write bytes, close.

lbogdan commented 1 year ago

Was there any progress on this? If not, I'm planning on working on it this weekend.