goneri
commented
2 years ago Hi! How do you expose the meta-data? And do you have the serial console output?
Closed jbpratt closed 2 years ago
goneri
commented
2 years ago Hi! How do you expose the meta-data? And do you have the serial console output?
jbpratt
commented
2 years ago Hi @goneri ! First off, thank you for the awesome work with the bsd images, I'm excited to get this working and as well the quick reply.
My understanding of KubeVirt internals are going to be quite shallow, but we are using cloudInitNoCloud (https://github.com/kubevirt/kubevirt/blob/099be903803556bfe3a85075a7b55f0a711d9ca7/docs/cloud-init.md) in-line like so https://github.com/ContainerCraft/qubo/blob/main/examples/vms/microk8s-br0.yaml#L68-L119 (just an example). https://github.com/kubevirt/kubevirt/blob/099be903803556bfe3a85075a7b55f0a711d9ca7/docs/cloud-init.md#nocloud-implementation-details
Yes, in https://github.com/ContainerCraft/kmi/issues/12#issue-1070652543 the first collapsed section is the serial console output, sorry that was hidden. The second is the actual VMI file.
goneri
commented
2 years ago Ok, thanks for the details, I will take a look later today.
usrbinkat
commented
2 years ago I tried a couple different ways to get the cloud config into the vm:
Cloud Init Config Drive:
Cloud Init NoCloud:
usrbinkat
commented
2 years ago This is a more complete log https://termbin.com/ekio
usrbinkat
commented
2 years ago Kubevirt launcher pod including some qemu-guest-agent connection logs: https://termbin.com/einnk
Libvirt XML: https://termbin.com/d3ix
usrbinkat
commented
2 years ago We're attaching the disks via virtio type. I also tried with efi/uefi and basic bios boot with no change.
We do come to a final login prompt via serial console but the cloud-config user credentials are not active. An ip is also acquired by the vm. SSH server does not seem to be running on the vm.
usrbinkat
commented
2 years ago A sata based cloudInitConfigDrive vm
goneri
commented
2 years ago UEFI won't work with the current image. I prepare a lab to reproduce the problem.
goneri
commented
2 years ago usrbinkat
commented
2 years ago @goneri we Tested both NoCloud and ConfigDrive with no luck. Serial console was not enabled so I only have screenshots of the vnc console. If the image has a default username / password I can login and upload the cloud-init logs and block device info to a pastebin.
Screenshot ConfigDrive Screenshot NoCloud
goneri
commented
2 years ago Can you please share your metadata disk and the libvirt domain XML.
usrbinkat
commented
2 years ago While the screen was scrolling, I saw a /dev/sd0b device not configured
goneri
commented
2 years ago I need the driver image actually. Because it's the source of the problem. It's not an ISO image exposed through a CDROM device, nor a disk with a partition. It's a mix of both.
usrbinkat
commented
2 years ago I'm checking with the Kubevirt community kubernetes slack channel for advice on how to provide that to you.
jbpratt
commented
2 years ago Hi @goneri, can you provide a password to get into the machine with? I'm not sure if there is a default password set and we are unable to set one through our usual means (virt-sysprep or cloud-init). Or if you have other recommended methods, that would be awesome. I think we can then better try to provide the data you are requesting.
Some virt-sysprep operations are supported on BSD flavors but I'm unable to inject my ssh key (or anything else) due to the disk layout it seems.
goneri
commented
2 years ago Yes, the GuestFS tools don't work properly with BSD system. @jbpratt you can try https://object-storage.public.mtl1.vexxhost.net/swift/v1/1dbafeefbd4f4c80864414a441e72dd2/bsd-cloud-image.org/openbsd-7.0-root-root.qcow2 login/pw are root/root .
We need to figure out why the meta-data disk is not mounted. It should be something like mount_cd9660 /dev/sd1a /mnt. sysctl hw.disknames will give you the list of the volume disks.
jbpratt
commented
2 years ago Thank you, here is the output
# ls /mnt
# mount_cd9660 /dev/sd1a /mnt
# sysctl hw.disknames
hw.disknames=sd0:da4d7a4ffdb53171,sd1:
# ls /mnt
meta-data network-config user-dataThanks! So my fix makes sense but I've also probably missed something else.
goneri
commented
2 years ago Can you attach the logs? /var/log/cloud-init-output.log /var/log/cloud-init.log and the content of these files (meta-data, network-onfig and user-data).
jbpratt
commented
2 years ago cloud-init.log https://termbin.com/bjxr cloud-init-output.log https://termbin.com/ax2x
# cat /mnt/network-config
version: 2
ethernets:
enp1s0:
dhcp4: true
dhcp6: true
dhcp-identifier: mac
# cat /mnt/user-data
#cloud-config
hostname: openbsd-br0
ssh_pwauth: true
disable_root: true
chpasswd:
list: |
kc2user:kc2user
expire: False
users:
- name: kc2user
shell: /bin/bash
lock_passwd: false
sudo: ['ALL=(ALL) NOPASSWD:ALL']
groups: sudo,wheel
growpart:
mode: auto
devices: ['/']
ignore_growroot_disabled: true
package_upgrade: true
packages:
- vim
- screenfetch
runcmd:
- "screenfetch"
# cat /mnt/meta-data
{"instance-id":"openbsd-br0.kargo","local-hostname":"openbsd-br0"latest PRs result in cloud-init successfully applying on boot. Thanks to @goneri for the effort.
jbpratt
commented
2 years ago This can be closed once https://github.com/canonical/cloud-init/pull/1149 is merged
goneri
commented
2 years ago FYI, I've pushed an OpenBSD 7.0 image on https://bsd-cloud-image.org/ that include the patch.
jbpratt
commented
2 years ago In that case, we can go ahead and close this. Thank you @goneri! :tada:
Playing around with packaging and booting an openbsd-6.9 CDI (no customizations). On boot, a few exceptions within cloud-init are thrown plus some concerning logs (
util.py[WARNING]: Can not apply stage final, no datasource found! Likely bad things to come!:sweat_smile: ). We should work with @goneri (sorry for the tag!) on understanding and resolving these issues. This will also allow the 'upstream' image source (https://bsd-cloud-image.org/) to be consistently tested on kubevirt within our pipeline.OpenBSD console logs
``` ❯ virtctl console -n kargo openbsd-br0 Successfully connected to openbsd-br0 console. The escape sequence is ^] cannot open hd0a:/etc/random.seed: No such file or directory booting hd0a:/bsd: 14411048+3224584+344096+0+1175552 [1009441+128+1145856+866050]=0x1528eb0 entry point at 0xffffffff81001000 [ using 3022512 bytes of bsd ELF symbol table ] Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2021 OpenBSD. All rights reserved. https://www.OpenBSD.org OpenBSD 6.9 (GENERIC.MP) #473: Mon Apr 19 10:40:28 MDT 2021 deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 3983396864 (3798MB) avail mem = 3847290880 (3669MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5a00 (11 entries) bios0: vendor SeaBIOS version "1.14.0-1.el8s" date 04/01/2014 bios0: KubeVirt None acpi0 at bios0: ACPI 3.0 acpi0: sleep states S5 acpi0: tables DSDT FACP APIC MCFG WAET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz, 664.21 MHz, 06-3c-03 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 999MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz, 750.26 MHz, 06-3c-03 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: smt 1, core 0, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xb0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) "ACPI0006" at acpi0 not configured acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 acpicmos0 at acpi0 "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "QEMU0002" at acpi0 not configured "ACPI0010" at acpi0 not configured acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) cpu0: using VERW MDS workaround pvbus0 at mainbus0: KVM pvclock0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00 vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02 wsdisplay at vga1 not configured ppb0 at pci0 dev 2 function 0 vendor "Red Hat", unknown product 0x000c rev 0x00: apic 0 int 22 pci1 at ppb0 bus 1 virtio0 at pci1 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01 vio0 at virtio0: address 02:f4:00:00:00:00 virtio0: msix shared ppb1 at pci0 dev 2 function 1 vendor "Red Hat", unknown product 0x000c rev 0x00: apic 0 int 22 pci2 at ppb1 bus 2 virtio1 at pci2 dev 0 function 0 "Qumranet Virtio 1.x SCSI" rev 0x01 vioscsi0 at virtio1: qsize 256 scsibus1 at vioscsi0: 255 targets virtio1: msix per-VQ ppb2 at pci0 dev 2 function 2 vendor "Red Hat", unknown product 0x000c rev 0x00: apic 0 int 22 pci3 at ppb2 bus 3 virtio2 at pci3 dev 0 function 0 "Qumranet Virtio 1.x Console" rev 0x01 virtio2: no matching child driver; not configured ppb3 at pci0 dev 2 function 3 vendor "Red Hat", unknown product 0x000c rev 0x00: apic 0 int 22 pci4 at ppb3 bus 4 virtio3 at pci4 dev 0 function 0 "Qumranet Virtio 1.x Storage" rev 0x01 vioblk0 at virtio3 scsibus2 at vioblk0: 1 targets sd0 at scsibus2 targ 0 lun 0:The image can be found at https://github.com/users/jbpratt/packages/container/package/openbsd
openbsd-br0.yml
``` --- # https://github.com/kubevirt/kubevirt/blob/master/docs/cloud-init.md apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: openbsd-br0 namespace: kargo labels: app: kargo spec: running: true template: spec: evictionStrategy: LiveMigrate nodeSelector: node-role.kubernetes.io/kubevirt: "" domain: clock: utc: {} timer: {} cpu: cores: 1 sockets: 1 threads: 2 model: host-passthrough dedicatedCpuPlacement: false devices: rng: {} autoattachPodInterface: false autoattachSerialConsole: true autoattachGraphicsDevice: true networkInterfaceMultiqueue: false disks: - name: containerdisk bootOrder: 1 disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - name: enp1s0 model: virtio bridge: {} machine: type: q35 resources: limits: memory: 4G requests: memory: 4G devices.kubevirt.io/kvm: "1" hostname: openbsd-br0 networks: - name: enp1s0 multus: networkName: kargo-net-attach-def-br0 terminationGracePeriodSeconds: 0 accessCredentials: - sshPublicKey: source: secret: secretName: kargo-sshpubkey-kc2user propagationMethod: qemuGuestAgent: users: - "kc2user" volumes: - name: containerdisk containerDisk: image: ghcr.io/jbpratt/openbsd:6.9-f8e31e3 imagePullPolicy: Always - name: cloudinitdisk cloudInitNoCloud: networkData: | version: 2 ethernets: enp1s0: dhcp4: true dhcp6: true dhcp-identifier: mac userData: | #cloud-config hostname: openbsd-br0 ssh_pwauth: true disable_root: true chpasswd: list: | kc2user:kc2user expire: False users: - name: kc2user shell: /bin/bash lock_passwd: false sudo: ['ALL=(ALL) NOPASSWD:ALL'] groups: sudo,wheel growpart: mode: auto devices: ['/'] ignore_growroot_disabled: true package_upgrade: true packages: - vim - screenfetch runcmd: - "screenfetch" ```