systemvm stuck in "Starting" and missing /usr/local/cloud/ files

[mierea]

ISSUE TYPE

• Bug Report

COMPONENT NAME

SSVM

CLOUDSTACK VERSION

4.18.1.0

CONFIGURATION

Advanced networking (one vlan for public/guest/management, one vlan for storage) Ceph with rbd for primary storage, ceph nfs for secondary storage (after zone is enabled I can see the systemvm template downloaded on secondary nfs)

OS / ENVIRONMENT

Ubuntu 22.04 LTS, KVM hosts

SUMMARY

After I create and enable zone the system vm's are stuck in "Starting" status. I can see the template was downloaded on secondary storage and I can manually mount it from s-1-VM and from the cloudstack vm.

STEPS TO REPRODUCE

Fresh Cloudstack install with kvm nodes, all on Ubuntu 22.04 LTS.
Primary and secondary storage on Ceph (rbd and nfs)
Cloudstack vm is hosted on first kvm node

in systemvm s-1-VM, cloud.log reads: https://pastebin.com/s4CXu8Gk

EXPECTED RESULTS

system vm's start 

ACTUAL RESULTS

system vm's are stuck starting and are missing files in /usr/local/cloud

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

@mierea can you test the following command on the kvm host ?

scp -i /root/.ssh/id_rsa.cloud -P 3922 /usr/share/cloudstack-common/vms/cloud-scripts.tgz 169.254.xx.yy:/root/

[mierea]

@weizhouapache thank you! For some reason cloud0 interface was down and so this SCP command was not working. As soon as I brought it back up and re-created the systemvm's it worked like a charm!

[weizhouapache]

@weizhouapache thank you! For some reason cloud0 interface was down and so this SCP command was not working. As soon as I brought it back up and re-created the systemvm's it worked like a charm!

@mierea good to know it. I remember I faced the same issue before, no idea what caused it.