rancher / k3os

Purpose-built OS for Kubernetes, fully managed by Kubernetes.
https://k3os.io
Apache License 2.0
3.5k stars 403 forks source link

qemu-guest-agent? #291

Closed cro closed 4 years ago

cro commented 4 years ago

Is there a way to get the qemu-guest-agent into k3os?

cro commented 4 years ago

thank you!!

dweomer commented 4 years ago

Hi @cro, I've added qemu-guest-agent in #292 but left it disabled by default. If you want to enable it you will need to make sure /etc/conf.d/qemu-guest-agent exists at the tail end of the boot phase. This simplest way to do this is via kernel parameter:

boot_cmd="touch /etc/conf.d/qemu-guest-qgent"

and the most robust way is via a write_files entry in config.yaml:

write_files:
- path: /etc/conf.d/qemu-guest-agent # see /etc/conf.d/qemu-guest-agent.orig
  content: |-
    # Specifies the transport method used to communicate to QEMU on the host side
    # Default: virtio-serial
    #GA_METHOD="virtio-serial"
    # Specifies the device path for the communications back to QEMU on the host
    # Default: /dev/virtio-ports/org.qemu.guest_agent.0
    #GA_PATH="/dev/virtio-ports/org.qemu.guest_agent.0"
  owner: root
  permissions: '0644'

If you are able to build your own ISO you can test this out now (because it is only on master as of yet). I want to play around with a few things first but I should be getting an RC out later this week (or early next week) for you to be able to download and run.

cro commented 4 years ago

Just a quick followup, this did get the guest agent and the extra tools (htop, etc) installed, but the virtio-serial device seems to be missing. I don't see a kernel module for virtio-serial in the image after installation. I am happy to build it, I'm just not sure how.

dweomer commented 4 years ago

@cro the necessary virtio modules are there and available. I will update the scripts/run-qemu with this sometime soon but this works for me (Qemu 4.1.1 on macOS):

qemu-system-x86_64 -m 2048 \
-machine accel=hax:kvm:hvf:tcg \
-nographic -serial mon:stdio \
-rtc base=utc,clock=rt \
-chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
-device virtio-serial \
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \
-kernel ./dist/artifacts/k3os-vmlinuz-amd64 \
-initrd ./dist/artifacts/k3os-initrd-amd64 \
-drive if=ide,media=cdrom,file=./dist/artifacts/k3os-amd64.iso \
-append 'console=ttyS0 loglevel=4 printk.devkmsg=on k3os.mode=live boot_cmd="echo GA_PATH=/dev/vport0p1>/etc/conf.d/qemu-guest-agent"'

and then from a different terminal:

echo '{"execute":"guest-get-osinfo"}' | socat unix-connect:/tmp/qga.sock stdout
cro commented 4 years ago

I'm not sure what's up then either. Here is the relevant section of the qemu entry from ps:

-chardev socket,path=/var/run/qemu-server/106.qga,server,nowait,id=qga0
-device virtio-serial,id=qga0,bus=pci.0,addr=0x8
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0

and

sudo find / -xdev -name org.qemu.guest_agent.0

doesn't return anything.

This is under Proxmox, BTW, with Qemu 4.1.1-1

dweomer commented 4 years ago

@cro please take note of the -append line below:

qemu-system-x86_64 -m 2048 \
-machine accel=hax:kvm:hvf:tcg \
-nographic -serial mon:stdio \
-rtc base=utc,clock=rt \
-chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \
-device virtio-serial \
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \
-kernel ./dist/artifacts/k3os-vmlinuz-amd64 \
-initrd ./dist/artifacts/k3os-initrd-amd64 \
-drive if=ide,media=cdrom,file=./dist/artifacts/k3os-amd64.iso \
-append 'console=ttyS0 loglevel=4 printk.devkmsg=on k3os.mode=live boot_cmd="echo GA_PATH=/dev/vport0p1>/etc/conf.d/qemu-guest-agent"'

Specifically: boot_cmd="echo GA_PATH=/dev/vport0p1>/etc/conf.d/qemu-guest-agent"

This is doing a couple of things at once:

yes, the name for the virtserialport has been passed as org.qemu.guest_agent.0 but the symlink isn't being setup (likely because we aren't yet running udev in k3OS). the actual path of the device you are creating is likely /dev/vport0p1 (assuming you haven't created other virtserialport devices for your VM) and you can see it even if qemu-ga isn't running.

so, if you are going the config.yaml route (rather than my kernel param example above) you should have a write_files that looks something like this:

write_files:
- path: /etc/conf.d/qemu-guest-agent # see /etc/conf.d/qemu-guest-agent.orig
  content: |-
    GA_PATH="/dev/vport0p1"
  owner: root
  permissions: '0644'
cro commented 4 years ago

ah, yes, so the issue turned out to be misunderstanding the name of the device node. Also for some reason in my env the node is /dev/vport2p1. After those fixes it works! Thanks for your help and patience.

pagong commented 4 years ago

I had to dig quite some time to get all of this working properly on Proxmox v6.1:

  1. enable QEMU Guest Agent in the Options tab of your VM
  2. Put the following lines into config.yaml before doing k3os install:
    write_files:
    - path: /etc/conf.d/qemu-guest-agent
    content: |-
    GA_PATH=/dev/vport2p1
    owner: root
    permissions: '0644'
  3. After installation and reboot of k3os, you'll see the guest IPs in the Summary tab of the VM

image

cubic3d commented 4 years ago

Thanks @pagong! Would like to see that as a cloud config option

sadoMasupilami commented 3 years ago

this worked for a synology nas should anyone have the same problem

write_files:
- path: /etc/conf.d/qemu-guest-agent
  content: |-
    GA_PATH=/dev/vport1p1
  owner: root
  permissions: '0644'