cannot add new host to cluster

[shulemmosko]

• Bug Report

Cloudstack-agent

version 4.18.1.0

Im using shapeblue repository. servers are running latest version of Ubuntu.

Im trying to add a new host but it fails, on the management server there is not error or failure. in the agent.log I got the following.

2023-12-05 16:03:32,884 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-12-05 16:03:32,888 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.1.0 2023-12-05 16:03:32,890 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-12-05 16:03:32,942 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2023-12-05 16:03:32,942 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2023-12-05 16:03:32,994 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2023-12-05 16:03:32,995 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection 2023-12-05 16:03:33,094 INFO [cloud.agent.Agent] (main:null) (logid:) id is 2023-12-05 16:03:33,108 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties. 2023-12-05 16:03:33,165 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks 2023-12-05 16:03:33,170 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one 2023-12-05 16:03:33,299 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: com.cloud.utils.exception.CloudRuntimeException: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1076) at com.cloud.agent.Agent.(Agent.java:190) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:452) at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431) at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415) at com.cloud.agent.AgentShell.start(AgentShell.java:511) at com.cloud.agent.AgentShell.main(AgentShell.java:541)

EXPECTED RESULTS

the host to be added to the cluster.

ACTUAL RESULTS

in the management GUI when im trying to add the host its not giving an error nor does it give a confirmation.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[rajujith]

@shulemmosko have you verified all the dependencies for KVM are installed and libvirtd is running? Which distribution and version you are using?

[shulemmosko]

@rajujith

Yes, as far as I know, all the dependencies are running.

Ubuntu Focal fully up-to-date and using the latest release of Cloudstack 4.18.1.0

The following line of the log looks to me as the issue, but I'm not sure. [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks

Thank you!

[shulemmosko]

So for now i ended up downgrading the agent to 4.17.2.0

so I'm running the management on 4.18.1.0 and agents on 4.17.2.0

this combination is working fine now.

[rajujith]

@shulemmosko mismatching versions of the management server and agent could give you other issues.

[shulemmosko]

@rajujith the issue is that i cant downgrade the management without restoring the previous database that was running under 4.17

[weizhouapache]

[kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks

@shulemmosko this is just a warning message, no issues found caused by it.

[weizhouapache]

@shulemmosko it looks libvirt does not work

2023-12-05 16:03:33,299 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

can you check if virsh list command works ?

[shulemmosko]

yes virsh list works fine, and im actually running vm's on it.

[shulemmosko]

@weizhouapache @rajujith

i just tried upgrading one host again to 4.18 and it does not connect. this time I'm not getting the libvirt error

2023-12-08 14:27:20,667 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-12-08 14:27:20,671 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.1.0 2023-12-08 14:27:20,673 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-12-08 14:27:20,724 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2023-12-08 14:27:20,725 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2023-12-08 14:27:20,775 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2023-12-08 14:27:20,776 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection 2023-12-08 14:27:20,871 INFO [cloud.agent.Agent] (main:null) (logid:) id is 1220 2023-12-08 14:27:20,885 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties. 2023-12-08 14:27:20,940 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks 2023-12-08 14:27:20,944 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one 2023-12-08 14:27:21,211 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) IO uring driver for Qemu: disabled 2023-12-08 14:27:21,296 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor 2023-12-08 14:27:21,300 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: com.cloud.utils.exception.CloudRuntimeException: java.lang.reflect.InvocationTargetException at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.(KVMStoragePoolManager.java:119) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1151) at com.cloud.agent.Agent.(Agent.java:190) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:452) at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431) at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415) at com.cloud.agent.AgentShell.start(AgentShell.java:511) at com.cloud.agent.AgentShell.main(AgentShell.java:541)

only the "failed to find passphrase for keystore"

[shulemmosko]

@weizhouapache @rajujith

The Passphrase is in the agent.properties file and the cloud.jks is in /etc/cloudstack/agent/

[weizhouapache]

@weizhouapache @rajujith

The Passphrase is in the agent.properties file and the cloud.jks is in /etc/cloudstack/agent/

@shulemmosko this is not the root cause, you can ignore it. it has been fixed in #7923 in upcoming 4.19, but there is no issue caused by it.

[shulemmosko]

I see, so how can i troubleshoot the connection between the agent and management?

this is how it looks in the UI.

Thanks!

[weizhouapache]

@weizhouapache @rajujith

i just tried upgrading one host again to 4.18 and it does not connect. this time I'm not getting the libvirt error

2023-12-08 14:27:20,667 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-12-08 14:27:20,671 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.1.0 2023-12-08 14:27:20,673 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-12-08 14:27:20,724 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2023-12-08 14:27:20,725 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2023-12-08 14:27:20,775 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2023-12-08 14:27:20,776 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection 2023-12-08 14:27:20,871 INFO [cloud.agent.Agent] (main:null) (logid:) id is 1220 2023-12-08 14:27:20,885 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties. 2023-12-08 14:27:20,940 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks 2023-12-08 14:27:20,944 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one 2023-12-08 14:27:21,211 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) IO uring driver for Qemu: disabled 2023-12-08 14:27:21,296 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor 2023-12-08 14:27:21,300 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: com.cloud.utils.exception.CloudRuntimeException: java.lang.reflect.InvocationTargetException at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.(KVMStoragePoolManager.java:119) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1151) at com.cloud.agent.Agent.(Agent.java:190) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:452) at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431) at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415) at com.cloud.agent.AgentShell.start(AgentShell.java:511) at com.cloud.agent.AgentShell.main(AgentShell.java:541)

only the "failed to find passphrase for keystore"

@shulemmosko can you list the files below

cd /usr/share/cloudstack-agent/lib/
ls -l cloud-*

[shulemmosko]

@weizhouapache @rajujith

[weizhouapache]

@shulemmosko can you check if there are some settings in agent.properties ?

ideally it should look like

cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.network.device=cloudbr1
keystore.passphrase=xxxxxxx
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr1
local.storage.uuid=30613d55-0d90-4d6a-a370-f4b28cb7bf8b
host=x.x.x.x@static
guid=e414520b-96c0-37a2-bef3-ebe439432765
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

[shulemmosko]

@weizhouapache

this is the content of my agent.properties

Thanks for your time!

[shulemmosko]

@weizhouapache @rajujith

Any ideas where else I can look for clues? Why the agent is not connecting after I upgrade from 4.17 to 4.18.

Thanks!

[shulemmosko]

@weizhouapache @rajujith

Ok here we go. I was wondering why i see the following plugin added since I'm not using linstor

2023-12-08 14:27:21,296 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor

So I renamed the plugin file in /usr/share/cloudstack-agent/lib/cloud-plugin-storage-volume-linstorl-4.18.1.0.jar to cloud-plugin-storage-volume-linstor-4.18.1.0.jar.OLD

And, walla, everything came up again.

So I wonder if this is a bug or something unique with my setup.

Im still having issues using consoleproxy with SSL but ill get back to troubleshoot tomorrow.

[weizhouapache]

@weizhouapache @rajujith

Ok here we go. I was wondering why i see the following plugin added since I'm not using linstor

2023-12-08 14:27:21,296 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor

So I renamed the plugin file in /usr/share/cloudstack-agent/lib/cloud-plugin-storage-volume-linstorl-4.18.1.0.jar to cloud-plugin-storage-volume-linstor-4.18.1.0.jar.OLD

hmm. that is what exactly I was planning to suggest you: remove unnecessary jars for plugins. good it works for you. @shulemmosko

maybe linstore guys can have a look. @rp-

And, walla, everything came up again.

So I wonder if this is a bug or something unique with my setup.

Im still having issues using consoleproxy with SSL but ill get back to troubleshoot tomorrow.

[rp-]

From my experience, this would mean that some class file couldn't be loaded(duplicate or missing), but if this would been a general problem non of the CloudStack smoke tests would work on the 4.18 branch. As the linstor plugin is loaded on every installation.

[weizhouapache]

From my experience, this would mean that some class file couldn't be loaded(duplicate or missing), but if this would been a general problem non of the CloudStack smoke tests would work on the 4.18 branch. As the linstor plugin is loaded on every installation.

thanks @rp-

might it be related to java version ? which openjdk version do you use ? @shulemmosko

[weizhouapache]

same issue #6794 #6797

[shulemmosko]

@weizhouapache @rp-

This is the Java version used on the hosts

openjdk version "11.0.21" 2023-10-17 OpenJDK Runtime Environment (build 11.0.21+9-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.21+9-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

[shulemmosko]

@weizhouapache

I'm also having issues with console proxy. Its only works if SSL is disabled; I don't know if it's related to this or if I should open a new issue.

[weizhouapache]

@weizhouapache

I'm also having issues with console proxy. Its only works if SSL is disabled; I don't know if it's related to this or if I should open a new issue.

Please refer to https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/

[shulemmosko]

@weizhouapache never mind, I just had to reset my browser. There was some stuck cookie issue, and the console proxy is working as expected now.

[shulemmosko]

@weizhouapache @rp- Should i close this issue or you guys still want to investigate further?

[weizhouapache]

@weizhouapache @rp- Should i close this issue or you guys still want to investigate further?

@shulemmosko let's keep it open for some days

[weizhouapache]

@shulemmosko can you list all files under /usr/share/cloudstack-* ?

ls -l /usr/share/cloudstack-*/*/

[shulemmosko]

@weizhouapache

Here you go.

root@PS-NODE4:~# ls -l /usr/share/cloudstack-// /usr/share/cloudstack-agent/lib/: total 100696 -rwxr-xr-x 1 root root 69409 Sep 14 09:46 activation-1.1.1.jar -rwxr-xr-x 1 root root 17536 Sep 14 09:46 annotations-13.0.jar -rwxr-xr-x 1 root root 90662 Sep 14 09:46 antisamy-1.5.3.jar -rwxr-xr-x 1 root root 14768 Sep 14 09:46 aopalliance-repackaged-2.5.0-b42.jar -rwxr-xr-x 1 root root 448794 Sep 14 09:46 apache-log4j-extras-1.2.17.jar -rwxr-xr-x 1 root root 121836 Sep 14 09:46 asm-7.3.1.jar -rwxr-xr-x 1 root root 33446 Sep 14 09:46 asm-analysis-7.3.1.jar -rwxr-xr-x 1 root root 71548 Sep 14 09:46 asm-commons-7.3.1.jar -rwxr-xr-x 1 root root 52826 Sep 14 09:46 asm-tree-7.3.1.jar -rwxr-xr-x 1 root root 84817 Sep 14 09:46 asm-util-7.3.1.jar -rwxr-xr-x 1 root root 1930381 Sep 14 09:46 aspectjweaver-1.8.13.jar -rwxr-xr-x 1 root root 1053038 Sep 14 09:46 aws-java-sdk-core-1.12.439.jar -rwxr-xr-x 1 root root 702186 Sep 14 09:46 aws-java-sdk-kms-1.12.439.jar -rwxr-xr-x 1 root root 1261041 Sep 14 09:46 aws-java-sdk-s3-1.12.439.jar -rwxr-xr-x 1 root root 8433 Sep 14 09:46 batik-constants-1.14.jar -rwxr-xr-x 1 root root 330318 Sep 14 09:46 batik-css-1.14.jar -rwxr-xr-x 1 root root 11466 Sep 14 09:46 batik-i18n-1.14.jar -rwxr-xr-x 1 root root 6663 Sep 14 09:46 batik-shared-resources-1.14.jar -rwxr-xr-x 1 root root 127477 Sep 14 09:46 batik-util-1.14.jar -rwxr-xr-x 1 root root 963713 Sep 14 09:46 bcpkix-jdk15on-1.70.jar -rwxr-xr-x 1 root root 5867298 Sep 14 09:46 bcprov-jdk15on-1.70.jar -rwxr-xr-x 1 root root 1457398 Sep 14 09:46 bctls-jdk15on-1.70.jar -rwxr-xr-x 1 root root 482530 Sep 14 09:46 bcutil-jdk15on-1.70.jar -rwxr-xr-x 1 root root 143992 Sep 14 09:46 bsh-core-2.0b4.jar -rwxr-xr-x 1 root root 415348 Sep 14 09:46 cglib-nodep-3.3.0.jar -rwxr-xr-x 1 root root 208835 Sep 14 09:46 checker-qual-3.12.0.jar -rwxr-xr-x 1 root root 86322 Sep 14 09:46 cloud-agent-4.18.1.0.jar -rwxr-xr-x 1 root root 2329401 Sep 14 09:46 cloud-api-4.18.1.0.jar -rwxr-xr-x 1 root root 610505 Sep 14 09:46 cloud-core-4.18.1.0.jar -rwxr-xr-x 1 root root 104795 Sep 14 09:46 cloud-engine-api-4.18.1.0.jar -rwxr-xr-x 1 root root 78370 Sep 14 09:46 cloud-engine-components-api-4.18.1.0.jar -rwxr-xr-x 1 root root 1965234 Sep 14 09:46 cloud-engine-schema-4.18.1.0.jar -rwxr-xr-x 1 root root 161354 Sep 14 09:46 cloud-engine-storage-4.18.1.0.jar -rwxr-xr-x 1 root root 17662 Sep 14 09:46 cloud-engine-storage-configdrive-4.18.1.0.jar -rwxr-xr-x 1 root root 65097 Sep 14 09:46 cloud-engine-storage-image-4.18.1.0.jar -rwxr-xr-x 1 root root 92848 Sep 14 09:46 cloud-engine-storage-volume-4.18.1.0.jar -rwxr-xr-x 1 root root 6666 Sep 14 09:46 cloud-framework-agent-lb-4.18.1.0.jar -rwxr-xr-x 1 root root 7624 Sep 14 09:46 cloud-framework-ca-4.18.1.0.jar -rwxr-xr-x 1 root root 78078 Sep 14 09:46 cloud-framework-cluster-4.18.1.0.jar -rwxr-xr-x 1 root root 43812 Sep 14 09:46 cloud-framework-config-4.18.1.0.jar -rwxr-xr-x 1 root root 180911 Sep 14 09:46 cloud-framework-db-4.18.1.0.jar -rwxr-xr-x 1 root root 10469 Sep 14 09:46 cloud-framework-direct-download-4.18.1.0.jar -rwxr-xr-x 1 root root 12494 Sep 14 09:46 cloud-framework-events-4.18.1.0.jar -rwxr-xr-x 1 root root 92014 Sep 14 09:46 cloud-framework-ipc-4.18.1.0.jar -rwxr-xr-x 1 root root 120906 Sep 14 09:46 cloud-framework-jobs-4.18.1.0.jar -rwxr-xr-x 1 root root 18094 Sep 14 09:46 cloud-framework-managed-context-4.18.1.0.jar -rwxr-xr-x 1 root root 12364 Sep 14 09:46 cloud-framework-rest-4.18.1.0.jar -rwxr-xr-x 1 root root 24863 Sep 14 09:46 cloud-framework-security-4.18.1.0.jar -rwxr-xr-x 1 root root 633473 Sep 14 09:46 cloud-plugin-hypervisor-kvm-4.18.1.0.jar -rwxr-xr-x 1 root root 84739 Sep 14 09:46 cloud-plugin-network-ovs-4.18.1.0.jar -rwxr-xr-x 1 root root 417741 Sep 14 09:46 cloud-plugin-network-tungsten-4.18.1.0.jar -rwxr-xr-x 1 root root 52870 Sep 14 09:46 cloud-plugin-storage-volume-linstor-4.18.1.0.jar.old -rwxr-xr-x 1 root root 89377 Sep 14 09:46 cloud-plugin-storage-volume-scaleio-4.18.1.0.jar -rwxr-xr-x 1 root root 171602 Sep 14 09:46 cloud-plugin-storage-volume-storpool-4.18.1.0.jar -rwxr-xr-x 1 root root 125260 Sep 14 09:46 cloud-secondary-storage-4.18.1.0.jar -rwxr-xr-x 1 root root 3546706 Sep 14 09:46 cloud-server-4.18.1.0.jar -rwxr-xr-x 1 root root 5120961 Sep 14 09:46 cloud-utils-4.18.1.0.jar -rwxr-xr-x 1 root root 233859 Sep 14 09:46 commons-beanutils-1.9.2.jar -rwxr-xr-x 1 root root 58284 Sep 14 09:46 commons-cli-1.5.0.jar -rwxr-xr-x 1 root root 353793 Sep 14 09:46 commons-codec-1.15.jar -rwxr-xr-x 1 root root 575389 Sep 14 09:46 commons-collections-3.2.1.jar -rwxr-xr-x 1 root root 1018316 Sep 14 09:46 commons-compress-1.21.jar -rwxr-xr-x 1 root root 362679 Sep 14 09:46 commons-configuration-1.10.jar -rwxr-xr-x 1 root root 25765 Sep 14 09:46 commons-daemon-1.3.3.jar -rwxr-xr-x 1 root root 211301 Sep 14 09:46 commons-dbcp2-2.9.0.jar -rwxr-xr-x 1 root root 146108 Sep 14 09:46 commons-digester-1.8.1.jar -rwxr-xr-x 1 root root 54495 Sep 14 09:46 commons-email-1.5.jar -rwxr-xr-x 1 root root 72446 Sep 14 09:46 commons-fileupload-1.4.jar -rwxr-xr-x 1 root root 305001 Sep 14 09:46 commons-httpclient-3.1.jar -rwxr-xr-x 1 root root 285424 Sep 14 09:46 commons-io-2.8.0.jar -rwxr-xr-x 1 root root 284220 Sep 14 09:46 commons-lang-2.6.jar -rwxr-xr-x 1 root root 577742 Sep 14 09:46 commons-lang3-3.11.jar -rwxr-xr-x 1 root root 61829 Sep 14 09:46 commons-logging-1.2.jar -rwxr-xr-x 1 root root 2213560 Sep 14 09:46 commons-math3-3.6.1.jar -rwxr-xr-x 1 root root 307003 Sep 14 09:46 commons-net-3.7.2.jar -rwxr-xr-x 1 root root 134858 Sep 14 09:46 commons-pool2-2.9.0.jar -rwxr-xr-x 1 root root 186077 Sep 14 09:46 commons-validator-1.6.jar -rwxr-xr-x 1 root root 5340 Sep 14 09:46 converter-moshi-2.9.0.jar -rwxr-xr-x 1 root root 544403 Sep 14 09:46 core-3.2.1.jar -rwxr-xr-x 1 root root 1409265 Sep 14 09:46 cxf-core-3.2.14.jar -rwxr-xr-x 1 root root 688071 Sep 14 09:46 cxf-rt-frontend-jaxrs-3.2.14.jar -rwxr-xr-x 1 root root 376299 Sep 14 09:46 cxf-rt-transports-http-3.2.14.jar -rwxr-xr-x 1 root root 63292 Sep 14 09:46 eddsa-0.3.0.jar -rwxr-xr-x 1 root root 1347501 Sep 14 09:46 ehcache-core-2.6.11.jar -rwxr-xr-x 1 root root 15979 Sep 14 09:46 error_prone_annotations-2.11.0.jar -rwxr-xr-x 1 root root 395859 Sep 14 09:46 esapi-2.1.0.1.jar -rwxr-xr-x 1 root root 4617 Sep 14 09:46 failureaccess-1.0.1.jar -rwxr-xr-x 1 root root 7202725 Sep 14 09:46 groovy-all-2.4.17.jar -rwxr-xr-x 1 root root 173608 Sep 14 09:46 gson-1.7.2.jar -rwxr-xr-x 1 root root 2959479 Sep 14 09:46 guava-31.1-jre.jar -rwxr-xr-x 1 root root 748221 Sep 14 09:46 guava-testlib-18.0.jar -rwxr-xr-x 1 root root 186763 Sep 14 09:46 hk2-api-2.5.0-b42.jar -rwxr-xr-x 1 root root 189454 Sep 14 09:46 hk2-locator-2.5.0-b42.jar -rwxr-xr-x 1 root root 135317 Sep 14 09:46 hk2-utils-2.5.0-b42.jar -rwxr-xr-x 1 root root 785639 Sep 14 09:46 httpclient-4.5.14.jar -rwxr-xr-x 1 root root 327891 Sep 14 09:46 httpcore-4.4.16.jar -rwxr-xr-x 1 root root 163254 Sep 14 09:46 influxdb-java-2.22.jar -rwxr-xr-x 1 root root 102220 Sep 14 09:46 ini4j-0.5.4.jar -rwxr-xr-x 1 root root 565410 Sep 14 09:46 ion-java-1.0.2.jar -rwxr-xr-x 1 root root 8781 Sep 14 09:46 j2objc-annotations-1.3.jar -rwxr-xr-x 1 root root 75704 Sep 14 09:46 jackson-annotations-2.12.5.jar -rwxr-xr-x 1 root root 365536 Sep 14 09:46 jackson-core-2.12.5.jar -rwxr-xr-x 1 root root 1515991 Sep 14 09:46 jackson-databind-2.12.5.jar -rwxr-xr-x 1 root root 61503 Sep 14 09:46 jackson-dataformat-cbor-2.12.6.jar -rwxr-xr-x 1 root root 120362 Sep 14 09:46 jackson-datatype-jsr310-2.12.5.jar -rwxr-xr-x 1 root root 31147 Sep 14 09:46 jackson-jaxrs-base-2.13.3.jar -rwxr-xr-x 1 root root 15964 Sep 14 09:46 jackson-jaxrs-json-provider-2.13.3.jar -rwxr-xr-x 1 root root 34589 Sep 14 09:46 jackson-module-jaxb-annotations-2.8.4.jar -rwxr-xr-x 1 root root 142255 Sep 14 09:46 jasypt-1.9.3.jar -rwxr-xr-x 1 root root 33304 Sep 14 09:46 java-ipv6-0.17.jar -rwxr-xr-x 1 root root 379481 Sep 14 09:46 java-linstor-0.3.0.jar -rwxr-xr-x 1 root root 38104 Sep 14 09:46 javase-3.2.1.jar -rwxr-xr-x 1 root root 782774 Sep 14 09:46 javassist-3.27.0-GA.jar -rwxr-xr-x 1 root root 26586 Sep 14 09:46 javax.annotation-api-1.3.2.jar -rwxr-xr-x 1 root root 2497 Sep 14 09:46 javax.inject-1.jar -rwxr-xr-x 1 root root 5951 Sep 14 09:46 javax.inject-2.5.0-b42.jar -rwxr-xr-x 1 root root 608084 Sep 14 09:46 javax.mail-1.5.6.jar -rwxr-xr-x 1 root root 203426 Sep 14 09:46 javax.persistence-2.2.1.jar -rwxr-xr-x 1 root root 95505 Sep 14 09:46 javax.servlet-api-4.0.1.jar -rwxr-xr-x 1 root root 127509 Sep 14 09:46 javax.ws.rs-api-2.1.jar -rwxr-xr-x 1 root root 125632 Sep 14 09:46 jaxb-api-2.3.0.jar -rwxr-xr-x 1 root root 255502 Sep 14 09:46 jaxb-core-2.3.0.jar -rwxr-xr-x 1 root root 963660 Sep 14 09:46 jaxb-impl-2.3.0.jar -rwxr-xr-x 1 root root 17075 Sep 14 09:46 jbcrypt-1.0.0.jar -rwxr-xr-x 1 root root 63504 Sep 14 09:46 jcommander-1.48.jar -rwxr-xr-x 1 root root 181496 Sep 14 09:46 jersey-client-2.26.jar -rwxr-xr-x 1 root root 1125788 Sep 14 09:46 jersey-common-2.26.jar -rwxr-xr-x 1 root root 69689 Sep 14 09:46 jersey-entity-filtering-2.26.jar -rwxr-xr-x 1 root root 62474 Sep 14 09:46 jersey-hk2-2.26.jar -rwxr-xr-x 1 root root 60934 Sep 14 09:46 jersey-media-json-jackson-2.26.jar -rwxr-xr-x 1 root root 66241 Sep 14 09:46 jersey-media-multipart-2.26.jar -rwxr-xr-x 1 root root 27590 Sep 14 09:46 jmespath-java-1.12.439.jar -rwxr-xr-x 1 root root 1506993 Sep 14 09:46 jna-5.5.0.jar -rwxr-xr-x 1 root root 2681931 Sep 14 09:46 jna-platform-5.5.0.jar -rwxr-xr-x 1 root root 637993 Sep 14 09:46 joda-time-2.12.5.jar -rwxr-xr-x 1 root root 282591 Sep 14 09:46 jsch-0.1.55.jar -rwxr-xr-x 1 root root 33015 Sep 14 09:46 jsr305-1.3.9.jar -rwxr-xr-x 1 root root 475561 Sep 14 09:46 juniper-tungsten-api-2.0.jar -rwxr-xr-x 1 root root 1487085 Sep 14 09:46 kotlin-stdlib-1.4.10.jar -rwxr-xr-x 1 root root 191212 Sep 14 09:46 kotlin-stdlib-common-1.4.0.jar -rwxr-xr-x 1 root root 3985 Sep 14 09:46 kotlin-stdlib-jdk7-1.4.10.jar -rwxr-xr-x 1 root root 16230 Sep 14 09:46 kotlin-stdlib-jdk8-1.4.10.jar -rwxr-xr-x 1 root root 148040 Sep 14 09:46 libvirt-0.5.3.jar -rwxr-xr-x 1 root root 5922 Sep 14 09:46 libvirtqemuhook -rwxr-xr-x 1 root root 2199 Sep 14 09:46 listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar -rwxr-xr-x 1 root root 15680 Sep 14 09:46 logging-interceptor-4.9.1.jar -rwxr-xr-x 1 root root 519087 Sep 14 09:46 mail-1.5.0-b01.jar -rwxr-xr-x 1 root root 85448 Sep 14 09:46 metrics-core-3.0.2.jar -rwxr-xr-x 1 root root 30865 Sep 14 09:46 metrics-jvm-3.0.2.jar -rwxr-xr-x 1 root root 63977 Sep 14 09:46 mimepull-1.9.6.jar -rwxr-xr-x 1 root root 138602 Sep 14 09:46 moshi-1.8.0.jar -rwxr-xr-x 1 root root 125069 Sep 14 09:46 msgpack-core-0.9.0.jar -rwxr-xr-x 1 root root 29680 Sep 14 09:46 mxparser-1.2.2.jar -rwxr-xr-x 1 root root 2167288 Sep 14 09:46 nashorn-core-15.3.jar -rwxr-xr-x 1 root root 124106 Sep 14 09:46 nekohtml-1.9.16.jar -rwxr-xr-x 1 root root 4080249 Sep 14 09:46 netty-all-4.1.42.Final.jar -rwxr-xr-x 1 root root 258278 Sep 14 09:46 not-yet-commons-ssl-0.3.9.jar -rwxr-xr-x 1 root root 791390 Sep 14 09:46 okhttp-4.9.1.jar -rwxr-xr-x 1 root root 243179 Sep 14 09:46 okio-2.8.0.jar -rwxr-xr-x 1 root root 1413987 Sep 14 09:46 opensaml-2.6.6.jar -rwxr-xr-x 1 root root 574762 Sep 14 09:46 openws-1.5.6.jar -rwxr-xr-x 1 root root 20235 Sep 14 09:46 osgi-resource-locator-1.0.1.jar -rwxr-xr-x 1 root root 1681866 Sep 14 09:46 protobuf-java-3.19.3.jar -rwxr-xr-x 1 root root 105909 Sep 14 09:46 rados-0.6.0.jar -rwxr-xr-x 1 root root 105848 Sep 14 09:46 reflections-0.9.12.jar -rwxr-xr-x 1 root root 333325 Sep 14 09:46 reload4j-1.2.25.jar -rwxr-xr-x 1 root root 125435 Sep 14 09:46 retrofit-2.9.0.jar -rwxr-xr-x 1 root root 3710 Sep 14 09:46 rolling-maintenance -rwxr-xr-x 1 root root 26084 Sep 14 09:46 slf4j-api-1.7.5.jar -rwxr-xr-x 1 root root 383374 Sep 14 09:46 spring-aop-5.3.26.jar -rwxr-xr-x 1 root root 703417 Sep 14 09:46 spring-beans-5.3.26.jar -rwxr-xr-x 1 root root 1275199 Sep 14 09:46 spring-context-5.3.26.jar -rwxr-xr-x 1 root root 1487015 Sep 14 09:46 spring-core-5.3.26.jar -rwxr-xr-x 1 root root 290661 Sep 14 09:46 spring-expression-5.3.26.jar -rwxr-xr-x 1 root root 24438 Sep 14 09:46 spring-jcl-5.3.26.jar -rwxr-xr-x 1 root root 1642463 Sep 14 09:46 spring-web-5.3.26.jar -rwxr-xr-x 1 root root 161867 Sep 14 09:46 stax2-api-3.1.4.jar -rwxr-xr-x 1 root root 34431 Sep 14 09:46 swagger-annotations-2.0.0.jar -rwxr-xr-x 1 root root 1814861 Sep 14 09:46 tink-1.7.0.jar -rwxr-xr-x 1 root root 4957 Sep 14 09:46 totp-1.0.jar -rwxr-xr-x 1 root root 343549 Sep 14 09:46 trilead-ssh2-build-217-jenkins-27.jar -rwxr-xr-x 1 root root 85203 Sep 14 09:46 truth-0.23.jar -rwxr-xr-x 1 root root 449505 Sep 14 09:46 velocity-1.7.jar -rwxr-xr-x 1 root root 512742 Sep 14 09:46 woodstox-core-5.0.3.jar -rwxr-xr-x 1 root root 2730442 Sep 14 09:46 xalan-2.7.0.jar -rwxr-xr-x 1 root root 1207073 Sep 14 09:46 xercesImpl-2.8.0.jar -rwxr-xr-x 1 root root 674607 Sep 14 09:46 xmlgraphics-commons-2.6.jar -rwxr-xr-x 1 root root 7188 Sep 14 09:46 xmlpull-1.1.3.1.jar -rwxr-xr-x 1 root root 173293 Sep 14 09:46 xmlschema-core-2.2.5.jar -rwxr-xr-x 1 root root 655992 Sep 14 09:46 xmlsec-1.5.7.jar -rwxr-xr-x 1 root root 673001 Sep 14 09:46 xmltooling-1.4.6.jar -rwxr-xr-x 1 root root 308863 Sep 14 09:46 xom-1.2.5.jar -rwxr-xr-x 1 root root 644649 Sep 14 09:46 xstream-1.4.20.jar

/usr/share/cloudstack-agent/plugins/: total 0

/usr/share/cloudstack-agent/tmp/: total 0

/usr/share/cloudstack-common/lib/: total 5144 -rwxr-xr-x 1 root root 5120961 Sep 14 09:52 cloudstack-utils.jar -rwxr-xr-x 1 root root 142255 Sep 14 09:52 jasypt-1.9.3.jar

/usr/share/cloudstack-common/scripts/: total 20 drwxr-xr-x 2 root root 4096 Dec 9 23:23 installer drwxr-xr-x 6 root root 4096 Apr 29 2022 network drwxr-xr-x 4 root root 4096 Dec 9 23:21 storage drwxr-xr-x 2 root root 4096 Dec 9 23:22 util drwxr-xr-x 5 root root 4096 Dec 9 23:21 vm

/usr/share/cloudstack-common/vms/: total 105536 -rwxr-xr-x 1 root root 107921128 Sep 14 09:52 agent.zip -rwxr-xr-x 1 root root 139181 Sep 14 09:52 cloud-scripts.tgz -rwxr-xr-x 1 root root 4603 Sep 14 09:52 patch-sysvms.sh

[weizhouapache]

@shulemmosko the files are identical on my host.

Do you use a non-root user ? can you run command /usr/bin/hostname as the user ?

[weizhouapache]

@shulemmosko the files are identical on my host.

Do you use a non-root user ? can you run command /usr/bin/hostname as the user ?

PING @shulemmosko

[weizhouapache]

@rp- there are some users facing the issue below

2023-12-08 14:27:21,296 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor
2023-12-08 14:27:21,300 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: java.lang.reflect.InvocationTargetException
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.(KVMStoragePoolManager.java:119)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1151)

which seems to be caused by the change in #6682

there is no issue before this change. the workaround is (1) remove linstor jar ; or (2) create symblink for /usr/bin/hostname

@rp- can you please have a look ?

[rp-]

I'm not sure if there was a specific reason to provide the full path here. But it also seems all modern distributions have hostname at /usr/bin/hostname now. Anyway I can change it back to "hostname"

[weizhouapache]

I'm not sure if there was a specific reason to provide the full path here. But it also seems all modern distributions have hostname at /usr/bin/hostname now. Anyway I can change it back to "hostname"

@rp- thanks I agree with that most OS have the file, but this issue has been reported by users 2-3 times already. it looks the file is missing in some version/release of hostname package.

[rohityadavcloud]

@shulemmosko were you able to consider and try some of the suggestions?

[weizhouapache]

fixed by #8633