search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
1.98k stars 1.09k forks source link

can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host #7823

Open gangaswamy opened 1 year ago

gangaswamy commented 1 year ago

Similar bugs were reported for previous versions like #6716, but tagged as fixed however it is not working in 4.18

ISSUE TYPE

UI,

CLOUDSTACK VERSION

4.18

CONFIGURATION

Basic Networking

OS / ENVIRONMENT

$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal

SUMMARY

Error 530 Could not add host at [http://192.168.122.1] with zone [2], pod [2] and cluster [2] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.122.1 - Failed to setup keystore on the KVM host: 192.168.122.1].

STEPS TO REPRODUCE

Fresh install on Ubuntu 20.04 Cloudstack Management server installed successfully and I can add Zone, Pod, Cluster, Network etc but not able to add host on KVM host. I am using the sudo user and password

EXPECTED RESULTS

Suppose to add host easily

ACTUAL RESULTS

Not able to add KVM host

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template!

rohityadavcloud commented 1 year ago

Hi @gangaswamy can you share your management server and KVM agent logs in DEBUG more (set DEBUG instead of INFO in the log4j xml configuration). Does it happen only on 20.04, have you tried 22.04?

gangaswamy commented 1 year ago

I am trying on Ubuntu 20.04,

$ sudo systemctl status cloudstack-agent

$ sudo systemctl status cloudstack-management

Log Management-server.log tail -100 management-server.log 2023-08-06 16:46:52,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5334a76b) (logid:409817d1) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:46:52,715 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-fdcea4b5) (logid:9801384b) HA health check task is running... 2023-08-06 16:46:53,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f231adf6) (logid:54cc4f4c) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:46:53,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f231adf6) (logid:54cc4f4c) Trying to connect to 69.30.206.114 2023-08-06 16:46:53,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f231adf6) (logid:54cc4f4c) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:46:54,036 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-b0695b7a) (logid:30f15c20) AutoScaling Monitor is running... 2023-08-06 16:46:54,037 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-b0695b7a) (logid:30f15c20) Skipping AutoScaling Monitor 2023-08-06 16:46:54,040 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-0cac04e8) (logid:ee24c5bf) HostStatsCollector is running... 2023-08-06 16:46:54,074 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-dd217e8b) (logid:6b972f22) StorageCollector is running... 2023-08-06 16:46:55,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-db5c792a) (logid:6605d38f) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:46:55,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-db5c792a) (logid:6605d38f) Trying to connect to 69.30.206.114 2023-08-06 16:46:55,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-db5c792a) (logid:6605d38f) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:46:56,717 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-cf2e0db4) (logid:709a1cfe) HA health check task is running... 2023-08-06 16:46:56,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b06d0a88) (logid:4812af10) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:46:56,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b06d0a88) (logid:4812af10) Trying to connect to 69.30.206.114 2023-08-06 16:46:56,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b06d0a88) (logid:4812af10) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:46:58,270 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee4f712d) (logid:f9e98b08) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:46:58,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee4f712d) (logid:f9e98b08) Trying to connect to 69.30.206.114 2023-08-06 16:46:58,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ee4f712d) (logid:f9e98b08) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:46:59,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-23b9a1c4) (logid:884d82e3) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:46:59,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-23b9a1c4) (logid:884d82e3) Trying to connect to 69.30.206.114 2023-08-06 16:46:59,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-23b9a1c4) (logid:884d82e3) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:00,719 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-05d2fb9b) (logid:60d3c258) HA health check task is running... 2023-08-06 16:47:00,785 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-0273aabb) (logid:5b5fb62c) Enabled non-edge zones available for scan: 2023-08-06 16:47:00,806 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-94682ae5) (logid:5084e107) Skip capacity scan as there is no Primary Storage in 'Up' state 2023-08-06 16:47:01,273 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b6a0f5a1) (logid:cd10354d) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:01,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b6a0f5a1) (logid:cd10354d) Trying to connect to 69.30.206.114 2023-08-06 16:47:01,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b6a0f5a1) (logid:cd10354d) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:02,773 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-cf8d8cf1) (logid:c401c62f) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:02,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-cf8d8cf1) (logid:c401c62f) Trying to connect to 69.30.206.114 2023-08-06 16:47:02,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-cf8d8cf1) (logid:c401c62f) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:04,268 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-168c5ddc) (logid:c4e688e1) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:04,268 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-168c5ddc) (logid:c4e688e1) Trying to connect to 69.30.206.114 2023-08-06 16:47:04,268 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-168c5ddc) (logid:c4e688e1) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:04,721 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-c1821802) (logid:755d25d1) HA health check task is running... 2023-08-06 16:47:05,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c5bd5732) (logid:e62a765c) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:05,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c5bd5732) (logid:e62a765c) Trying to connect to 69.30.206.114 2023-08-06 16:47:05,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c5bd5732) (logid:e62a765c) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:07,273 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c123f860) (logid:a751340d) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:07,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c123f860) (logid:a751340d) Trying to connect to 69.30.206.114 2023-08-06 16:47:07,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c123f860) (logid:a751340d) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:08,723 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-6afb8f78) (logid:3850adc9) HA health check task is running... 2023-08-06 16:47:08,773 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-21553adb) (logid:d0c0db0a) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:08,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-21553adb) (logid:d0c0db0a) Trying to connect to 69.30.206.114 2023-08-06 16:47:08,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-21553adb) (logid:d0c0db0a) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:09,029 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-944279a5) (logid:11574509) Found 0 routers to update status. 2023-08-06 16:47:09,031 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-944279a5) (logid:11574509) Found 0 VPC's to update Redundant State. 2023-08-06 16:47:09,032 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-944279a5) (logid:11574509) Found 0 networks to update RvR status. 2023-08-06 16:47:10,270 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-017a1332) (logid:7617ae3a) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:10,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-017a1332) (logid:7617ae3a) Trying to connect to 69.30.206.114 2023-08-06 16:47:10,270 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-017a1332) (logid:7617ae3a) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:11,769 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9013f0) (logid:97fc5253) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:11,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9013f0) (logid:97fc5253) Trying to connect to 69.30.206.114 2023-08-06 16:47:11,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9013f0) (logid:97fc5253) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:12,725 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-66d5a4dd) (logid:332442fe) HA health check task is running... 2023-08-06 16:47:13,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5a2c65d0) (logid:fdde8df1) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:13,272 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5a2c65d0) (logid:fdde8df1) Trying to connect to 69.30.206.114 2023-08-06 16:47:13,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5a2c65d0) (logid:fdde8df1) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:14,769 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4a67c36b) (logid:06ce6c7e) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:14,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4a67c36b) (logid:06ce6c7e) Trying to connect to 69.30.206.114 2023-08-06 16:47:14,770 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4a67c36b) (logid:06ce6c7e) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:16,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-04577350) (logid:af02a3f6) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:16,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-04577350) (logid:af02a3f6) Trying to connect to 69.30.206.114 2023-08-06 16:47:16,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-04577350) (logid:af02a3f6) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:16,726 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-7e7be91a) (logid:0798b84f) HA health check task is running... 2023-08-06 16:47:17,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2406167b) (logid:fec38bb1) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:17,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2406167b) (logid:fec38bb1) Trying to connect to 69.30.206.114 2023-08-06 16:47:17,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-2406167b) (logid:fec38bb1) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:19,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9387f5d7) (logid:0f921bbc) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:19,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9387f5d7) (logid:0f921bbc) Trying to connect to 69.30.206.114 2023-08-06 16:47:19,273 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9387f5d7) (logid:0f921bbc) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:20,728 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-29a0a8d8) (logid:fb74a13e) HA health check task is running... 2023-08-06 16:47:20,770 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-fc5918ec) (logid:95cd156a) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:20,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-fc5918ec) (logid:95cd156a) Trying to connect to 69.30.206.114 2023-08-06 16:47:20,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-fc5918ec) (logid:95cd156a) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:22,277 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a4caa824) (logid:29e84059) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:22,277 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a4caa824) (logid:29e84059) Trying to connect to 69.30.206.114 2023-08-06 16:47:22,277 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a4caa824) (logid:29e84059) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:23,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ff3826e7) (logid:74ac5079) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:23,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ff3826e7) (logid:74ac5079) Trying to connect to 69.30.206.114 2023-08-06 16:47:23,772 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ff3826e7) (logid:74ac5079) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:24,730 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-ed0a4a4c) (logid:43ada46b) HA health check task is running... 2023-08-06 16:47:25,273 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f2b387ba) (logid:b0989592) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:25,274 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f2b387ba) (logid:b0989592) Trying to connect to 69.30.206.114 2023-08-06 16:47:25,274 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-f2b387ba) (logid:b0989592) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:26,772 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9baaad) (logid:338ccc7b) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:26,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9baaad) (logid:338ccc7b) Trying to connect to 69.30.206.114 2023-08-06 16:47:26,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-ac9baaad) (logid:338ccc7b) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:28,269 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9e28f3e1) (logid:b3e8b611) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:28,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9e28f3e1) (logid:b3e8b611) Trying to connect to 69.30.206.114 2023-08-06 16:47:28,269 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9e28f3e1) (logid:b3e8b611) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:28,732 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-b17d4c43) (logid:678c3cee) HA health check task is running... 2023-08-06 16:47:29,773 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3f3dc80) (logid:f78e7eca) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:29,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3f3dc80) (logid:f78e7eca) Trying to connect to 69.30.206.114 2023-08-06 16:47:29,773 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-b3f3dc80) (logid:f78e7eca) Management node 4 is detected inactive by timestamp but is pingable 2023-08-06 16:47:30,785 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-a6a43229) (logid:d983f3ba) Enabled non-edge zones available for scan: 2023-08-06 16:47:30,806 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a1a52827) (logid:d3350bcf) Skip capacity scan as there is no Primary Storage in 'Up' state 2023-08-06 16:47:31,272 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-528b7c2b) (logid:a5d42a5b) Detected management node left, id:4, nodeIP:69.30.206.114 2023-08-06 16:47:31,272 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-528b7c2b) (logid:a5d42a5b) Trying to connect to 69.30.206.114 2023-08-06 16:47:31,272 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-528b7c2b) (logid:a5d42a5b) Management node 4 is detected inactive by timestamp but is pingabl

Agent log $ tail -100 agent.log 2023-08-06 16:43:19,029 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:43:29,767 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:43:29,772 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:43:29,773 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:43:40,469 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:43:40,474 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:43:40,476 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:43:51,034 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:43:51,039 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:43:51,041 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:44:01,675 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:44:01,680 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:44:01,681 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:44:12,273 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:44:12,278 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:44:12,279 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:44:23,016 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:44:23,021 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:44:23,023 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:44:33,770 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:44:33,775 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:44:33,777 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:44:44,528 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:44:44,533 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:44:44,535 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:44:55,272 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:44:55,277 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:44:55,279 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:45:06,019 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:45:06,024 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:45:06,025 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:45:16,768 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:45:16,772 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:45:16,774 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:45:27,521 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:45:27,526 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:45:27,527 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:45:38,269 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:45:38,274 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:45:38,277 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:45:49,019 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:45:49,024 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:45:49,026 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:45:59,760 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:45:59,764 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:45:59,766 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:46:10,524 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:46:10,529 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:46:10,530 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:46:21,269 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:46:21,274 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:46:21,276 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:46:32,022 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:46:32,027 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:46:32,028 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:46:42,772 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:46:42,777 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:46:42,778 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:46:53,511 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:46:53,516 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:46:53,517 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:47:04,280 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:47:04,285 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:47:04,287 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:47:15,021 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:47:15,025 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:47:15,027 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:47:25,757 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:47:25,763 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:47:25,765 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:47:36,518 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:47:36,523 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:47:36,525 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:47:47,270 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:47:47,275 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:47:47,277 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:47:58,022 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:47:58,027 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:47:58,029 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:48:08,781 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:48:08,785 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:48:08,787 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:48:19,517 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:48:19,521 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:48:19,523 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:48:30,273 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:48:30,278 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:48:30,279 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:48:41,008 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:48:41,013 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:48:41,015 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:48:51,773 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:48:51,778 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:48:51,780 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:49:02,523 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:49:02,528 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:49:02,530 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2023-08-06 16:49:13,269 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2023-08-06 16:49:13,274 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.0.0 2023-08-06 16:49:13,276 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties

rajujith commented 1 year ago

@gangaswamy could you check whether package tzdata-java is installed on the KVM host if not retry the agent setup after installing this package?

gangaswamy commented 1 year ago

$ dpkg -l | grep tzdata ii tzdata 2023c-0ubuntu0.20.04.2 all time zone and daylight-saving time data

it is installed and its latest version. tzdata-java package is not available for Ubuntu

xuanyuanaosheng commented 1 year ago

@gangaswamy Was this server added to CloudStack during zone initialization? If so, please modify the server's IP address to another unused IP address in the same VLAN, and re-add the server.

weizhouapache commented 1 year ago

@gangaswamy it looks cloudstack-agent service is stuck at activating

can you run commands systemctl status cloudstack-agent and journalctl -xeu cloudstack-agent ? they might give more information

please also check the status and log of libvirtd service.

axeldaiber commented 9 months ago

Any news regarding this issue? I do have the same issue with Ubuntu 22.04 jammy and Cloudstack 4.18.1.0: Failed to setup keystore on the KVM host:

weizhouapache commented 9 months ago

Any news regarding this issue? I do have the same issue with Ubuntu 22.04 jammy and Cloudstack 4.18.1.0: Failed to setup keystore on the KVM host:

@axeldaiber although the log says "Failed to setup keystore", the issue could be caused by something else, for example the network config.

do you use netplan ? have you configured cloudbr0 and/or cloudbr1 ?

hoogenm commented 7 months ago

I got this error message when the automated ssh-login (from setup to the host) without a tty was unsuccessful. For diagnostics, I tried again with root, and that worked. Hope this may help others. Ubuntu Jammy & Cloudstack 4.18

rohityadavcloud commented 4 months ago

I'm not sure if using the sudo user and password is supported. The docs suggest to use root user with password, or with ssh-public key auth of the management server (cloud user's ssh key can ssh the KVM host via root user)

rudolfjs commented 3 weeks ago

I'm not sure if using the sudo user and password is supported. The docs suggest to use root user with password, or with ssh-public key auth of the management server (cloud user's ssh key can ssh the KVM host via root user)

I temporarily allowed root password login via SSH (revered after deployment) and it worked. Setting root as the user for the target host fixed the keystore problem.

tampler commented 1 week ago

I see this error on ACS 4.20.0:

can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.1.1 - Failed to setup keystore on the K
VM host: 192.168.1.1

This happens due to the following:

2024-09-10T11:22:15,279 DEBUG [c.c.u.s.SSHCmdHelper] (qtp481511146-370:[ctx-23c36759, ctx-2d609a42]) (logid:136b20d9) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/                                      
SSH command output:                                                                                                                               
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper            
sudo: a password is required

So there's an invalid ssh auth on the ACS mgmt side when adding a host with the user/pass. Should be done with the "-S" option as per error above

tampler commented 1 week ago

Actually, keystore-setup script doesn't work:

> # /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying
weizhouapache commented 1 week ago

Actually, keystore-setup script doesn't work:

> # /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying

this is not the full command , if you copied from management server log.

tampler commented 1 week ago

this is not the full command , if you copied from management server log.

Yes, the command above was lifted from the log. I executed it in the command line and pasted the output

2024-09-10T11:52:05,946 DEBUG [c.c.u.s.SSHCmdHelper] (qtp481511146-33:[ctx-b9629dea, ctx-60272547]) (logid:c352422a) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/

SSH command output:                                                                                                                               
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required
weizhouapache commented 1 week ago

this is not the full command , if you copied from management server log.

Yes, the command above was lifted from the log. I executed it in the command line and pasted the output

2024-09-10T11:52:05,946 DEBUG [c.c.u.s.SSHCmdHelper] (qtp481511146-33:[ctx-b9629dea, ctx-60272547]) (logid:c352422a) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/

SSH command output:                                                                                                                               
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

Can you enable NOPASSWD in sudoers for the user ?

tampler commented 1 week ago

Can you enable NOPASSWD in sudoers for the user ?

@weizhouapache Thanks, that fixed my problem! I've been using ACS since 2019 and each time I reinstall it I figure out yet another issue/trouble :laughing: