dhiltgen / docker-machine-kvm

KVM driver for docker-machine
Apache License 2.0
376 stars 117 forks source link

Unexpected JSON Output #77

Open darkn3rd opened 3 years ago

darkn3rd commented 3 years ago

I tried to use the latest released version with docker-machine, and I am getting this error. I am not deeply familiar with KVM or libvirt, so I am not sure how this works. I have libvirt working fine with vagrant and mini-kube (using their embedded kvm2 docker-machine driver) without any problems. I am not sure how I could debug further.

$ docker-machine -D create -d kvm myengine1
Docker Machine Version:  0.16.2, build bd45ab13
Found binary path at /usr/local/bin/docker-machine-driver-kvm
Launching plugin server for driver kvm
Plugin server listening at address 127.0.0.1:41571
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /usr/local/bin/docker-machine-driver-kvm
Launching plugin server for driver kvm
Plugin server listening at address 127.0.0.1:45025
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(myengine1) Calling .GetMachineName
(myengine1) Calling .DriverName
(myengine1) Calling .GetCreateFlags
(myengine1) Calling .SetConfigFromFlags
(myengine1) DBG | SetConfigFromFlags called
Reading certificate data from /home/joaquin/.docker/machine/certs/ca.pem
Decoding PEM data...
Parsing certificate...
Reading certificate data from /home/joaquin/.docker/machine/certs/cert.pem
Decoding PEM data...
Parsing certificate...
Running pre-create checks...
(myengine1) Calling .PreCreateCheck
(myengine1) DBG | About to check libvirt version
(myengine1) DBG | Validating private network
(myengine1) DBG | Validating network default
(myengine1) Calling .GetConfigRaw
Creating machine...
(myengine1) Calling .Create
(myengine1) DBG | local Boot2Docker ISO version:  v19.03.12
(myengine1) Copying /home/joaquin/.docker/machine/cache/boot2docker.iso to /home/joaquin/.docker/machine/machines/myengine1/boot2docker.iso...
(myengine1) Creating SSH key...
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker/machine/machines/myengine1
(myengine1) DBG | Setting executable bit set on /home/joaquin/.docker/machine/machines/myengine1
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker/machine/machines
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker/machine
(myengine1) DBG | Verifying executable bit set on /home/joaquin/.docker
(myengine1) DBG | Verifying executable bit set on /home/joaquin
(myengine1) DBG | Verifying executable bit set on /home
(myengine1) DBG | Creating VM data disk...
(myengine1) DBG | Creating 20000 MB hard disk image...
(myengine1) DBG | Defining VM...
(myengine1) DBG | Starting VM myengine1
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 0
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 1
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 2
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 3
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 4
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 5
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 6
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 7
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 8
(myengine1) DBG | GetIP called for myengine1
(myengine1) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(myengine1) Failed to decode dnsmasq lease status: unexpected end of JSON input
(myengine1) DBG | Unable to locate IP address for MAC 52:54:00:d2:f1:a0
(myengine1) DBG | Waiting for the VM to come up... 9

There's no such file /var/lib/libvirt/dnsmasq/docker-machines.leases.

$ ls -la /var/lib/libvirt/dnsmasq
total 24
drwxr-xr-x 2 root root 4096 Oct 11 03:46 .
drwxr-xr-x 7 root root 4096 Oct 11 02:00 ..
-rw-r--r-- 1 root root    0 Oct 11 02:01 default.addnhosts
-rw------- 1 root root  619 Oct 11 02:01 default.conf
-rw-r--r-- 1 root root    0 Oct 11 02:01 default.hostsfile
-rw-r--r-- 1 root root    0 Oct 11 03:25 docker-machines.addnhosts
-rw------- 1 root root  688 Oct 11 03:25 docker-machines.conf
-rw-r--r-- 1 root root    0 Oct 11 03:25 docker-machines.hostsfile
-rw-r--r-- 1 root root   85 Oct 11 03:46 virbr0.macs
-rw-r--r-- 1 root root    0 Oct 11 01:02 virbr0.status
-rw-r--r-- 1 root root   85 Oct 11 03:46 virbr1.macs
-rw-r--r-- 1 root root    0 Oct 11 03:25 virbr1.status
afbjorklund commented 3 years ago

This is related to https://github.com/libvirt/libvirt/commit/0f87054b61d73493fb505ecb97bd16615bc53699 since libvirt doesn't use a dnsmasq file for this anymore (--leasefile-ro).

As you say, the error format is misleading. It should probably have been "file not found", not that it would have helped...

afbjorklund commented 3 years ago

Also seen in #60 and #72

minikube uses virtio now

afbjorklund commented 3 years ago

Actually it was not only using the correct code (status instead of leases), it also needs to use a different network card. The current "rtl8139" is not supported with the kernel config, it doesn't have to use virtio but can at least use "e1000".

model type rtl8139 (fails) rtl8139

model type e1000 (works) e1000

It is possible to backport code from minikube to this driver, but since it is not supported anymore there will be no releases. With enough interest it would be possible to make an version at https://github.com/machine-drivers/docker-machine-kvm

diff --git a/kvm.go b/kvm.go
index 1e6c891..c73de61 100644
--- a/kvm.go
+++ b/kvm.go
@@ -665,11 +667,22 @@ func (d *Driver) GetIP() (string, error) {
        if err != nil {
                return "", err
        }
-       /*
-        * TODO - Figure out what version of libvirt changed behavior and
-        *        be smarter about selecting which algorithm to use
-        */
-       ip, err := d.getIPByMACFromLeaseFile(mac)
+       conn, err := d.getConn()
+       if err != nil {
+               return "", err
+       }
+       defer conn.Close()
+
+       var ip string
+       libVersion, err := conn.GetLibVersion()
+       if err != nil {
+               return "", err
+       }
+
+       // Earlier versions of libvirt use a lease file instead of a status file
+       if libVersion < 1002006 {
+               ip, err = d.getIPByMACFromLeaseFile(mac)
+       }
        if ip == "" {
                ip, err = d.getIPByMacFromSettings(mac)
        }
darkn3rd commented 3 years ago

As newbie to all of this, some questions:

afbjorklund commented 3 years ago

Those were the virtual nics, the"rtl8139" is the default and I changed it to "e1000" in a local build:

--- a/kvm.go
+++ b/kvm.go
@@ -60,9 +60,11 @@ const (
     </graphics>
     <interface type='network'>
       <source network='{{.Network}}'/>
+      <model type='e1000'/>
     </interface>
     <interface type='network'>
       <source network='{{.PrivateNetwork}}'/>
+      <model type='e1000'/>
     </interface>
   </devices>
 </domain>`

The lease file used to carry the IPs allocated, but it is now in the status file (in dnsmasq) instead.

No releases: meaning there will be no more tag versions? Or meaning this is in archival mode, no more commits for master? It would be nice to add to master or docs...

See https://github.com/dhiltgen/docker-machine-kvm/issues/67

Only the VirtualBox driver was officially supported by Docker Inc., and now Machine is deprecated too.

So far there has been no volunteers to take over the development and release, in the "new" organization.

As the name implies, it was originally only for drivers:

https://github.com/machine-drivers/discussion/issues

afbjorklund commented 3 years ago

@darkn3rd : You can find a prebuilt binary here: https://github.com/afbjorklund/docker-machine-kvm

afbjorklund commented 3 years ago

The missing kernel configuration for "rtl8139cp" was reported here: https://github.com/boot2docker/boot2docker/issues/1407

jernejr commented 3 years ago

@darkn3rd : You can find a prebuilt binary here: https://github.com/afbjorklund/docker-machine-kvm

That solved this exact issue for me