Secured Libvirt/Qemu console does not work on aarch64 host

I couldn't confirm before upgrading, but post upgrading & renewing the host certificates, vm instances wouldn't deploy on an aarch64 host due to libvirt/vnc certificate errors (I had checked the pem file existed & had a certificate content). After comment the tls/vnc config in /etc/libvirt/qemu.conf I could get unsecured console to work in the web/novnc popup. I didn't hit this kind of issue on x86 host, so likely this could be just an aarch64 env/config issue.

The following was seen in the cloudstack-agent logs on the aarch64 host:

2024-11-27 12:47:59,353 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:[]) (logid:bd7b5a1e) Creating volume 54eeeb93-bccc-47f4-bc81-e14db979c9fa from template 92105cb5-2728-40b8-ae28-64a621f493e0 in pool 63229d40-f348-4636-8707-74ab46ab28d0 (Filesystem) with size (100.00 GB) 107374182400
2024-11-27 12:47:59,354 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:[]) (logid:bd7b5a1e) Attempting to create volume 54eeeb93-bccc-47f4-bc81-e14db979c9fa (Filesystem) in pool 63229d40-f348-4636-8707-74ab46ab28d0 with size (3.50 GB) 3758096384
2024-11-27 12:47:59,597 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Trying to fetch storage pool 63229d40-f348-4636-8707-74ab46ab28d0 from libvirt
2024-11-27 12:47:59,625 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Trying to fetch storage pool 63229d40-f348-4636-8707-74ab46ab28d0 from libvirt
2024-11-27 12:47:59,947 WARN  [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2024-11-27 12:47:59,947 WARN  [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Groovy scripting engine is not initialized. Data transformation skipped.
2024-11-27 12:48:00,892 WARN  [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) LibvirtException org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: 2024-11-27T07:18:00.671827Z qemu-system-aarch64: Cannot load CA certificate '/etc/pki/libvirt-vnc/ca-cert.pem': Error while reading file.
    at org.libvirt.ErrorHandler.processError(Unknown Source)
    at org.libvirt.ErrorHandler.processError(Unknown Source)
    at org.libvirt.Connect.domainCreateXML(Unknown Source)
    at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1909)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:87)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:48)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
    at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
    at com.cloud.agent.Agent.processRequest(Agent.java:686)
    at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
    at com.cloud.utils.nio.Task.call(Task.java:83)
    at com.cloud.utils.nio.Task.call(Task.java:29)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)

ISSUE TYPE

Bug Report

COMPONENT NAME

KVM/aarch64

CLOUDSTACK VERSION

4.20 RC3

CONFIGURATION

Adv zone without security groups, with two clusters - one of x86 with three x86 hosts and one aarch64 cluster with a single arm64 host.

apache / cloudstack