search

kairos-io / kairos

:penguin: The immutable Linux meta-distribution for edge Kubernetes.
https://kairos.io
Apache License 2.0
1.09k stars 97 forks source link

VMWare Guestinfo customization does not always run #1865

Closed sarg3nt closed 10 months ago

sarg3nt commented 1 year ago

Kairos version: Base is kairos/core-rockylinux:v2.4.0

CPU architecture, OS, and Version: VMWare VMs, X86-64, Rocky Linux 9 with a custom Dockerfile that installs some extra packages and security updates

Describe the bug I'm running the new VMWare Guest Customization and am having issues where it sometimes does not run. I've found this to be independent of the target node being created from AuroraBoot as it just happened when deploying an AuroraBoot node that is cloned from a VMware template, so that node was not booted from AuroraBoot. Undeploying and redeploying the node fixed it.

I'd estimate I'm getting this to happen about 20% of the time.

Looking for recommendations on how to troubleshoot and what logs you want to see.

cloud_init.yaml looks like this.

Note: This is a Terraform template so values with ${} are replaced during terraform apply

#cloud-config
users:
  - name: "kairos"
    passwd: ${password}
    ssh_authorized_keys:
      - ssh-rsa <redacted for brevity>
# These two files configure the target AuroraBoot server to tell it what container image to build into an ISO
# and what cloud_init.yaml file to serve to downstream target systems when they boot.
# This file is used by the data.tf cloudinit data element and is encoded in main.tf as
# extra_config.guestinfo.userdata which VMWare mounts and Kairos uses during boot to configure the AurorBoot server.
write_files:
  - encoding: b64
    content: ${container_image}
    path: /etc/systemd/system/container_image.txt
    permissions: "0600"
    owner: "root"
  - encoding: b64
    content: ${cloud_init}
    path: /etc/systemd/system/cloud_init.yaml
    permissions: "0600"
    owner: "root"
stages:
  initramfs:
    - name: "Set hostname"
      hostname: "${node_name}.${node_domain}"

To Reproduce I don't think you will be able to as it sounds like you don't have a VMWare environment to replicate this in

Itxaka commented 1 year ago

@sarg3nt is there anything on the yip/serial logs when it fails? (be sure to add rd.immucore.debug to the grub entry to see the full debug output from yip!)

Could this be related to vmware endpoints for metadata not being ready?

sarg3nt commented 11 months ago

Hey, just to update you on this. I haven't had this issue in a bit and I did upgrade to 2.4.1 which may be why?

sarg3nt commented 11 months ago

Heh, right after I leave that I ran into again on two nodes. I'm adding the debug statement and will try to replicate.

sarg3nt commented 11 months ago

Something very strange just happened. Maybe you can clarify if you think this is coming from yip or VMWare being slow to apply things??? I ran another test and node 2 didn't join the cluster, so I SSH into it and my SSH key doesn't work, so I try the nodes password, which is randomly generated and stored in our credential storage provider. That doesn't work either, so i try the password that is set before the per node cloud_init.yaml is run and that works. To clarify, we have one cloud_init.yaml that is deployed from AuroraBoot as the base that is the same for all nodes, then a second that is applied via the VMWare userdata that is different for each node. I was able to log in with the password set by AuroraBoot but not the one set for each node.

I got in and started poking around and it looked like all the settings got set from the VMWare per node cloud_init except for the password "change" and the authorized_keys. I even went to the user directory .ssh folder and confirmed there wasn't an authorized_keys file.

I find it very strange that some of the config from the secondary per node cloud_init.yaml file got applied but not all of it.

Now here's where it gets even stranger. I logged out of the SSH session then logged back in and my SSH key worked. All of a sudden the password is correct and the SSH key is there. I double and triple checked I'm on the same problematic node. After that config "finished" the node joined the k8s cluster.
This took like 5 or 10 minutes.

What could be going on here? This doesn't FEEL like a VMWare thing. It seems like either the file got there or it didn't and I confirmed that some of it had been applied.

I added the debug statement as you suggested but I can't find the yip/serial file. Where do I look for that? Or is that a section in /run/immucore/rootfs_stage.log?

Also, I added 

install:
  grub_options:
    extra_cmdline: "rd.immucore.debug"

To the root cloud_init.yaml file served by AuroraBoot but not to the secondary one being sent per server via VMWare, does it need to be in both? I assumed the root one would do the job.

In the rootfs_stage.log I think I see where it is creating the /oem/userdata file and that file has the per node, from VMWare settings that it should, including the password change and public key.

FYI, I'm setting the kairos users password in the AuroraBoot cloud_init.yaml like this

users:
  # The kairos user is configured in the target nodes terraform
  - name: "kairos"
    passwd: ${password}

NOTE: That file is a Terraform template so the ${password} is injected in during the terraform apply.

Then the per node cloud_init.yaml has this

users:
  - name: "kairos"
    passwd: "${password}"
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA<cut>

The ${password} being injected here is the randomly generated per node password. We do it this way so if the secondary change doesn't work I can use the password set by AuroraBoot.

This usually works, but if you forsee an issue with it let me know.

Here's the logs for rootfs_stage.log

[root@lpul-vault-k8s-server-2 immucore]# cat rootfs_stage.log
time="2023-10-31T23:49:24Z" level=info msg="Running stage: rootfs.before\n"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Datasource handling.Pull data from provider'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Datasource handling.Sentinel file for userdata'"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_datasource.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Datasource handling.Pull data from provider'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="If statement result "
time="2023-10-31T23:49:24Z" level=info msg="Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )"
time="2023-10-31T23:49:24Z" level=debug msg="Stage: {\"Commands\":null,\"Files\":null,\"Downloads\":null,\"Directories\":null,\"If\":\"[ ! -f /oem/userdata ]\",\"EnsureEntities\":null,\"DeleteEntities\":null,\"Dns\":{\"Nameservers\":null,\"DnsSearch\":null,\"DnsOptions\":null,\"Path\":\"\"},\"Hostname\":\"\",\"Name\":\"Pull data from provider\",\"Sysctl\":null,\"SSHKeys\":null,\"Node\":\"\",\"Users\":null,\"Modules\":null,\"Systemctl\":{\"Enable\":null,\"Disable\":null,\"Start\":null,\"Mask\":null},\"Environment\":null,\"EnvironmentFile\":\"\",\"After\":null,\"DataSources\":{\"Providers\":[\"cdrom\",\"gcp\",\"openstack\",\"aws\",\"azure\",\"hetzner\",\"packet\",\"vultr\",\"digitalocean\",\"metaldata\",\"vmware\"],\"Path\":\"/oem\"},\"Layout\":{\"Device\":null,\"Expand\":null,\"Parts\":null},\"SystemdFirstBoot\":null,\"TimeSyncd\":null,\"Git\":{\"Auth\":{\"Username\":\"\",\"Password\":\"\",\"PrivateKey\":\"\",\"Insecure\":false,\"PublicKey\":\"\"},\"URL\":\"\",\"Path\":\"\",\"Branch\":\"\",\"BranchOnly\":false}}"
time="2023-10-31T23:49:24Z" level=debug msg="Creating directory /run/config"
time="2023-10-31T23:49:24Z" level=debug msg="Full provider list: [GCP Openstack AWS Azure Hetzner Packet Vultr DigitalOcean Metaldata VMWARE]"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider GCP"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider Openstack"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider AWS"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider Azure"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider Hetzner"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider Packet"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider Vultr"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider DigitalOcean"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider Metaldata"
time="2023-10-31T23:49:24Z" level=debug msg="Starting provider VMWARE"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from Hetzner"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from Vultr"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from Openstack"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from AWS"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from Azure"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from Packet"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from GCP"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from DigitalOcean"
time="2023-10-31T23:49:24Z" level=debug msg="Found userdata from VMWARE"
time="2023-10-31T23:49:24Z" level=debug msg="Didnt found userdata from Metaldata"
time="2023-10-31T23:49:24Z" level=debug msg="Creating file /oem/userdata"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="Creating file /oem/userdata.yaml"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="did not find any device in layout"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_datasource.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Datasource handling.Sentinel file for userdata'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ ! -f /oem/userdata ]: exit status 1)' stage name: Sentinel file for userdata"
time="2023-10-31T23:49:24Z" level=info msg="Done executing stage 'rootfs.before'\n"
time="2023-10-31T23:49:24Z" level=info msg="Running stage: rootfs\n"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings.Mount BPF on Alpine systems'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings.Layout configuration for active/passive mode'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings.Layout configuration for recovery mode'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings.Layout configuration for booting local node from livecd'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings for UKI.Layout configuration for UKI boot'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings for UKI.Layout configuration for UKI installer'"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings.Mount BPF on Alpine systems'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ ! -e \"/sbin/systemctl\" ] && [ ! -e \"/usr/bin/systemctl\" ] && [ ! -e \"/usr/sbin/systemctl\" ] && [ ! -e \"/usr/bin/systemctl\" ]: exit status 1)' stage name: Mount BPF on Alpine systems"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings.Layout configuration for active/passive mode'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="If statement result "
time="2023-10-31T23:49:24Z" level=info msg="Processing stage step 'Layout configuration for active/passive mode'. ( commands: 0, files: 0, ... )"
time="2023-10-31T23:49:24Z" level=debug msg="Stage: {\"Commands\":null,\"Files\":null,\"Downloads\":null,\"Directories\":null,\"If\":\"[ ! -f \\\"/run/cos/recovery_mode\\\" ] \\u0026\\u0026 [ ! -e \\\"/run/cos/uki_boot_mode\\\" ] \\u0026\\u0026 [ ! -e \\\"/run/cos/uki_install_mode\\\" ]\",\"EnsureEntities\":null,\"DeleteEntities\":null,\"Dns\":{\"Nameservers\":null,\"DnsSearch\":null,\"DnsOptions\":null,\"Path\":\"\"},\"Hostname\":\"\",\"Name\":\"Layout configuration for active/passive mode\",\"Sysctl\":null,\"SSHKeys\":null,\"Node\":\"\",\"Users\":null,\"Modules\":null,\"Systemctl\":{\"Enable\":null,\"Disable\":null,\"Start\":null,\"Mask\":null},\"Environment\":{\"OVERLAY\":\"tmpfs:25%\",\"PERSISTENT_STATE_BIND\":\"true\",\"PERSISTENT_STATE_PATHS\":\"/etc/systemd /etc/modprobe.d /etc/rancher /etc/sysconfig /etc/runlevels /etc/ssh /etc/ssl/certs /etc/iscsi /etc/zfs /etc/cni /etc/kubernetes /home /opt /root /snap /var/snap /usr/libexec /var/log /var/lib/containerd /var/lib/extensions /var/lib/rancher /var/lib/kubelet /var/lib/snapd /var/lib/wicked /var/lib/longhorn /var/lib/cni /var/lib/dbus /usr/share/pki/trust /usr/share/pki/trust/anchors /var/lib/ca-certificates\",\"RW_PATHS\":\"/var /etc /srv\",\"VOLUMES\":\"LABEL=COS_PERSISTENT:/usr/local\"},\"EnvironmentFile\":\"/run/cos/cos-layout.env\",\"After\":null,\"DataSources\":{\"Providers\":null,\"Path\":\"\"},\"Layout\":{\"Device\":null,\"Expand\":null,\"Parts\":null},\"SystemdFirstBoot\":null,\"TimeSyncd\":null,\"Git\":{\"Auth\":{\"Username\":\"\",\"Password\":\"\",\"PrivateKey\":\"\",\"Insecure\":false,\"PublicKey\":\"\"},\"URL\":\"\",\"Path\":\"\",\"Branch\":\"\",\"BranchOnly\":false}}"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="did not find any device in layout"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings.Layout configuration for recovery mode'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ -f \"/run/cos/recovery_mode\" ]: exit status 1)' stage name: Layout configuration for recovery mode"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings.Layout configuration for booting local node from livecd'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run grep -q \"kairos.boot_live_mode\" /proc/cmdline: exit status 1)' stage name: Layout configuration for booting local node from livecd"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs_uki.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings for UKI.Layout configuration for UKI boot'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ -e \"/run/cos/uki_boot_mode\" ]: exit status 1)' stage name: Layout configuration for UKI boot"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs_uki.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings for UKI.Layout configuration for UKI installer'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ -e \"/run/cos/uki_install_mode\" ]: exit status 1)' stage name: Layout configuration for UKI installer"
time="2023-10-31T23:49:24Z" level=info msg="Done executing stage 'rootfs'\n"
time="2023-10-31T23:49:24Z" level=info msg="Running stage: rootfs.after\n"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings.add custom bind and ephemeral mounts to /run/cos/cos-layout.env'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage 'Rootfs Layout Settings.Grow persistent'"
time="2023-10-31T23:49:24Z" level=debug msg="Generating op for stage '/system/oem/11_persistent.yaml.Grow persistent'"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/11_persistent.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage '/system/oem/11_persistent.yaml.Grow persistent'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings.add custom bind and ephemeral mounts to /run/cos/cos-layout.env'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ ! -f /run/cos/recovery_mode ] && [ ! -f /run/cos/live_mode ] && [ -f \"/sys/firmware/devicetree/base/model\" ] && grep -i \"Raspberry Pi 4\" \"/sys/firmware/devicetree/base/model\": exit status 1)' stage name: Grow persistent"
time="2023-10-31T23:49:24Z" level=warning msg="(conditional) Skip 'Skipping stage (if statement error: failed to run [ -r /run/cos/custom-layout.env ] && [ ! -f \"/run/cos/recovery_mode\" ] && [ ! -f /run/cos/live_mode ]: exit status 1)' stage name: add custom bind and ephemeral mounts to /run/cos/cos-layout.env"
time="2023-10-31T23:49:24Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
time="2023-10-31T23:49:24Z" level=debug msg="Executing stage 'Rootfs Layout Settings.Grow persistent'"
time="2023-10-31T23:49:24Z" level=debug msg="{\"sysinfo\":{\"version\":\"0.9.5\",\"timestamp\":\"2023-10-31T23:49:21.432409069Z\"},\"node\":{\"hostname\":\"localhost\",\"machineid\":\"c67ed855b8dd411ba041ab261995c387\",\"hypervisor\":\"vmware\"},\"os\":{\"name\":\"Rocky Linux 9.2 (Blue Onyx) dracut-057-21.git20230214.el9 (Initramfs)\",\"vendor\":\"rocky\",\"version\":\"9.2\",\"architecture\":\"amd64\"},\"kernel\":{\"release\":\"5.14.0-284.30.1.el9_2.x86_64\",\"version\":\"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023\",\"architecture\":\"x86_64\"},\"product\":{\"name\":\"VMware Virtual Platform\",\"vendor\":\"VMware, Inc.\",\"version\":\"None\",\"serial\":\"VMware-42 36 2b 50 6c ad 47 6d-1e b7 74 65 d5 a0 91 3b\"},\"board\":{\"name\":\"440BX Desktop Reference Platform\",\"vendor\":\"Intel Corporation\",\"version\":\"None\",\"serial\":\"None\"},\"chassis\":{\"type\":1,\"vendor\":\"No Enclosure\",\"version\":\"N/A\",\"serial\":\"None\",\"assettag\":\"No Asset Tag\"},\"bios\":{\"vendor\":\"Phoenix Technologies LTD\",\"version\":\"6.00\",\"date\":\"11/12/2020\"},\"cpu\":{\"vendor\":\"AuthenticAMD\",\"model\":\"AMD EPYC 9374F 32-Core Processor\",\"speed\":3867,\"cache\":1024,\"threads\":4},\"memory\":{\"type\":\"DRAM\",\"size\":16384},\"storage\":[{\"name\":\"sda\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":85},{\"name\":\"sdb\",\"driver\":\"sd\",\"vendor\":\"VMware\",\"model\":\"Virtual disk\",\"size\":42}],\"network\":[{\"name\":\"ens192\",\"driver\":\"vmxnet3\",\"macaddress\":\"00:50:56:b6:21:18\",\"port\":\"tp\",\"speed\":10000}]}"
time="2023-10-31T23:49:24Z" level=debug msg="If statement result "
time="2023-10-31T23:49:24Z" level=info msg="Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... )"
time="2023-10-31T23:49:24Z" level=debug msg="Stage: {\"Commands\":null,\"Files\":null,\"Downloads\":null,\"Directories\":null,\"If\":\"[ ! -f /run/cos/recovery_mode ] \\u0026\\u0026 [ ! -f /run/cos/live_mode ]\",\"EnsureEntities\":null,\"DeleteEntities\":null,\"Dns\":{\"Nameservers\":null,\"DnsSearch\":null,\"DnsOptions\":null,\"Path\":\"\"},\"Hostname\":\"\",\"Name\":\"Grow persistent\",\"Sysctl\":null,\"SSHKeys\":null,\"Node\":\"\",\"Users\":null,\"Modules\":null,\"Systemctl\":{\"Enable\":null,\"Disable\":null,\"Start\":null,\"Mask\":null},\"Environment\":null,\"EnvironmentFile\":\"\",\"After\":null,\"DataSources\":{\"Providers\":null,\"Path\":\"\"},\"Layout\":{\"Device\":{\"Label\":\"COS_PERSISTENT\",\"Path\":\"\"},\"Expand\":{\"Size\":0},\"Parts\":null},\"SystemdFirstBoot\":null,\"TimeSyncd\":null,\"Git\":{\"Auth\":{\"Username\":\"\",\"Password\":\"\",\"PrivateKey\":\"\",\"Insecure\":false,\"PublicKey\":\"\"},\"URL\":\"\",\"Path\":\"\",\"Branch\":\"\",\"BranchOnly\":false}}"
time="2023-10-31T23:49:24Z" level=debug msg="Using label COS_PERSISTENT for layout expansion"
time="2023-10-31T23:49:24Z" level=debug msg="Output of udevadm settle: "
time="2023-10-31T23:49:24Z" level=debug msg="Got device /dev/sda for label COS_PERSISTENT"
time="2023-10-31T23:49:24Z" level=error msg="Failed analyzing disk: failed to run sgdisk -p /dev/sda: exit status 127\n"
time="2023-10-31T23:49:24Z" level=warning msg="Could not calculate disk free space: failed to run sgdisk -p /dev/sda: exit status 127"
time="2023-10-31T23:49:24Z" level=warning msg="Not enough unpartitioned space in disk to operate"
time="2023-10-31T23:49:24Z" level=info msg="Done executing stage 'rootfs.after'\n"
time="2023-10-31T23:49:24Z" level=info msg="Running stage: rootfs.before\n"
time="2023-10-31T23:49:24Z" level=info msg="Done executing stage 'rootfs.before'\n"
time="2023-10-31T23:49:24Z" level=info msg="Running stage: rootfs\n"
time="2023-10-31T23:49:24Z" level=info msg="Done executing stage 'rootfs'\n"
time="2023-10-31T23:49:24Z" level=info msg="Running stage: rootfs.after\n"
time="2023-10-31T23:49:24Z" level=info msg="Done executing stage 'rootfs.after'\n"

According to the log time stamp it completed well before I tried to log in. So could it not have applied everything in the resultant oem file for a while?

Is there another log for yip I need to look at to see how the file is being applied?

sarg3nt commented 11 months ago

In /oem there are two files, userdata and a userdata.yaml both of which have the same contents. This is before the node reboots. Found it on one of the nodes that got stuck unmounting volumes

And on another node that has rebooted but exhibiting the partially ran userdata issue the /oem folder is showing the following: image

The last few lines of /run/immucore/rootfs_stage.log are

time="2023-11-01T20:03:39Z" level=debug msg="Using label COS_PERSISTENT for layout expansion"
time="2023-11-01T20:03:39Z" level=debug msg="Output of udevadm settle: "
time="2023-11-01T20:03:39Z" level=debug msg="Got device /dev/sda for label COS_PERSISTENT"
time="2023-11-01T20:03:39Z" level=error msg="Failed analyzing disk: failed to run sgdisk -p /dev/sda: exit status 127\n"
time="2023-11-01T20:03:39Z" level=warning msg="Could not calculate disk free space: failed to run sgdisk -p /dev/sda: exit status 127"
time="2023-11-01T20:03:39Z" level=warning msg=c
time="2023-11-01T20:03:39Z" level=info msg="Done executing stage 'rootfs.after'\n"
time="2023-11-01T20:03:39Z" level=info msg="Running stage: rootfs.before\n"
time="2023-11-01T20:03:39Z" level=info msg="Done executing stage 'rootfs.before'\n"
time="2023-11-01T20:03:39Z" level=info msg="Running stage: rootfs\n"
time="2023-11-01T20:03:39Z" level=info msg="Done executing stage 'rootfs'\n"
time="2023-11-01T20:03:39Z" level=info msg="Running stage: rootfs.after\n"
time="2023-11-01T20:03:39Z" level=info msg="Done executing stage 'rootfs.after'\n"

I checked and sgdisk is "not found"

[kairos@lpul-vault-k8s-server-1 ~]$ sgdisk -p /dev/sda
-bash: sgdisk: command not found

I installed gdisk in the root image but am getting the same error though I can run it fine on the node.

time="2023-11-01T23:12:37Z" level=debug msg="Using label COS_PERSISTENT for layout expansion"
time="2023-11-01T23:12:37Z" level=debug msg="Output of udevadm settle: "
time="2023-11-01T23:12:37Z" level=debug msg="Got device /dev/sda for label COS_PERSISTENT"
time="2023-11-01T23:12:37Z" level=error msg="Failed analyzing disk: failed to run sgdisk -p /dev/sda: exit status 127\n"
time="2023-11-01T23:12:37Z" level=warning msg="Could not calculate disk free space: failed to run sgdisk -p /dev/sda: exit status 127"
time="2023-11-01T23:12:37Z" level=warning msg="Not enough unpartitioned space in disk to operate"
time="2023-11-01T23:12:37Z" level=info msg="Done executing stage 'rootfs.after'\n"
time="2023-11-01T23:12:37Z" level=info msg="Running stage: rootfs.before\n"
time="2023-11-01T23:12:37Z" level=info msg="Done executing stage 'rootfs.before'\n"
time="2023-11-01T23:12:37Z" level=info msg="Running stage: rootfs\n"
time="2023-11-01T23:12:37Z" level=info msg="Done executing stage 'rootfs'\n"
time="2023-11-01T23:12:37Z" level=info msg="Running stage: rootfs.after\n"
time="2023-11-01T23:12:37Z" level=info msg="Done executing stage 'rootfs.after'\n"
[kairos@lpul-vault-k8s-server-1 immucore]$ sudo sgdisk -p /dev/sda
Disk /dev/sda: 167772160 sectors, 80.0 GiB
Model: Virtual disk
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 7241A51E-F8C5-4898-9A1F-4A2FCA6DC7F7
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 167772126
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  bios
   2            4096          135167   64.0 MiB    8300  oem
   3          135168         7888895   3.7 GiB     8300  recovery
   4         7888896        20953087   6.2 GiB     8300  state
   5        20953088       167770111   70.0 GiB    8300  persistent
[kairos@lpul-vault-k8s-server-1 immucore]$

And after about 15 minutes the userdata file finished applying, I can now log in with my private key and the servers added themselves to the cluster immediately after.
This is very strange and it doesn't always do it.

I've also just had this do this on the deployment of the AuroraBoot server. This server is deployed via TerraForm from a VSphere template and is also using the userdata through VMWare.

To recap:

  1. There are two userdata files
  2. Sometimes the userdata file is not applying all at once. Part of it is running, then about 15 minutes later the rest of it.
  3. There are sgdisk errors in /run/immucore/rootfs_stage.log
  4. I don't know where else to look for yip progress and errors.
jimmykarily commented 10 months ago

I'll reply as thing come to me:

  1. I think I've seen this before. I don't think it's a bug. I think non-yaml files are ignored (based on extension). I would need to check, I'm not sure.

  2. yip/immucore will try to run stages in parallel when that's possible afaik. This could explain why you see some things being applied while some things not yet. The question is why does it take so long for some of the things to apply. Especially the user configuration. This doesn't involve any network activity, it's fully local. I don't have a suggestion here.

  3. You need to have sgdisk available in the initramfs (initrd). This happens here for regular Kairos images: https://github.com/kairos-io/immucore/blob/f075442ea527c68306185074c5bf43389dadf240/dracut/28immucore/module-setup.sh#L28 . If you do a similar step like this one: https://github.com/kairos-io/kairos/blob/a658a3fa5f294b14377631dedfa0031d3551f2b2/Earthfile#L343 to install the generic framework image, you should have those files in place and sgdisk should be there in initramfs. You said you have a custom image. How exactly do you build it? Are you taking a final, fully cooked Kairos image and build on top of it or are you using the Dockerfiles in the Kairos repo? If you do the latter, you are missing some steps that still live in the Earthfile (but will be moved to the dockerfiles soon).

  4. The files you look at is what Ixaka meant: /run/immucore/* and you seem to have enabled debugging output correctly since there is level=debug in the output.

sarg3nt commented 10 months ago

It looks like this was fixed in https://github.com/kairos-io/kairos-sdk/pull/58 as I have not seen this happen since switching to the master build.

mauromorales commented 10 months ago

Awesome, thanks for the feedback @sarg3nt