search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.11k stars 1.11k forks source link

Can't add host server to cloudstack management server #9705

Closed subincb30 closed 2 months ago

subincb30 commented 2 months ago
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
I am using 4.19 version on cloudstack management and cloudstack host server. But, I am unable to add the host server to management server. Both are Ubuntu 22.04 server.
 Getting the following error:

---
Error 530
Unable to add the host: Cannot find the server resources at http://192.168.200.238
---

Able to get ping response from host and management server:

---
From management server

root@ubuntu:~# ping 192.168.200.238
PING 192.168.200.238 (192.168.200.238) 56(84) bytes of data.
64 bytes from 192.168.200.238: icmp_seq=1 ttl=64 time=2.00 ms
64 bytes from 192.168.200.238: icmp_seq=2 ttl=64 time=1.42 ms

From the hots server

root@host:~# ping 192.168.200.215
PING 192.168.200.215 (192.168.200.215) 56(84) bytes of data.
64 bytes from 192.168.200.215: icmp_seq=1 ttl=64 time=1.98 ms
64 bytes from 192.168.200.215: icmp_seq=2 ttl=64 time=1.98 ms
64 bytes from 192.168.200.215: icmp_seq=3 ttl=64 time=2.00 ms
----
boring-cyborg[bot] commented 2 months ago

Thanks for opening your first issue here! Be sure to follow the issue template!

kiranchavala commented 2 months ago

@subincb30

Could you please take look at the /var/log/cloudstack/management/management-server.log for any other exceptions

and can you provide the log to investigate further

subincb30 commented 2 months ago

Here is the recent logs:

root@ubuntu:~# tail -f /var/log/cloudstack/management/management-server.log 2024-09-19 08:09:17,523 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:ctx-8f2e2288) (logid:3fc927f1) Starting to remove all volume_stats rows older than [Wed Sep 18 20:09:17 UTC 2024]. 2024-09-19 08:09:17,525 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:ctx-fe5244f9) (logid:40b04762) Removed a total of [0] vm_stats rows older than [Thu Sep 19 08:08:17 UTC 2024]. 2024-09-19 08:09:17,525 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:ctx-8f2e2288) (logid:3fc927f1) Removed a total of [0] volume_stats rows older than [Wed Sep 18 20:09:17 UTC 2024]. 2024-09-19 08:09:19,255 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-8f547b9c) (logid:38edd7cf) HA health check task is running... 2024-09-19 08:09:23,258 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-765d761d) (logid:bd6be1a8) HA health check task is running... 2024-09-19 08:09:25,242 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-23bae9a7) (logid:1d353076) Skip capacity scan as there is no Primary Storage in 'Up' state 2024-09-19 08:09:25,256 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Enabled non-edge zones available for scan: 1 2024-09-19 08:09:25,257 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Zone 1 has no host available which is enabled and in Up state 2024-09-19 08:09:25,257 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Zone [1] is not ready to launch secondary storage VM. 2024-09-19 08:09:27,260 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-3:ctx-243e29db) (logid:feaac53e) HA health check task is running... 2024-09-19 08:09:31,263 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-ad46d590) (logid:232a2af8) HA health check task is running... 2024-09-19 08:09:31,848 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 routers to update status. 2024-09-19 08:09:31,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 VPC's to update Redundant State. 2024-09-19 08:09:31,852 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 networks to update RvR status.

Also, I checked for the specific error for host server IP: 192.168.200.238:

2024-09-19 07:24:47,528 INFO [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Trying to add a new host at http://192.168.200.238 in data center 1 2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted. 2024-09-19 07:30:00,375 WARN [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Cannot find the server resources at http://192.168.200.238 com.cloud.exception.DiscoveryException: Unable to add the host: Cannot find the server resources at http://192.168.200.238 2024-09-19 07:30:00,380 INFO [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Unable to add the host: Cannot find the server resources at http://192.168.200.238 root@ubuntu:~#

Please check and let me know any further details needed.

subincb30 commented 2 months ago

Additionally, I found that the port 8250 is not listening in my cloudstack host server.

weizhouapache commented 2 months ago

the logs show

2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted.

there should be some errors in log like CloudStack Agent setup through command ... failed

subincb30 commented 2 months ago

@weizhouapache Sorry I didn't get your point. Also, I couldn't find any errors related to the cloud stack agent on the cloud stack management-server.log.

weizhouapache commented 2 months ago

@weizhouapache Sorry I didn't get your point. Also, I couldn't find any errors related to the cloud stack agent on the cloud stack management-server.log.

the logs shows the mgmt server has connected to the host and configured security (for cloudstack-agent and libvirt). so it is not a connectivity issue.

please upload the full log by keyword ctx-6dbc68fe if possible.

subincb30 commented 2 months ago

@weizhouapache Here is the full log with keyword: ctx-6dbc68fe:

root@ubuntu:~# cat /var/log/cloudstack/management/management-server.log | grep ctx-6dbc68fe 2024-09-19 07:24:47,504 DEBUG [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"3bc44629-66bf-11ef-9691-70b5e860d7ec"}]' is allowed to perform API calls: 0.0.0.0/0,::/0 2024-09-19 07:24:47,513 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2024-09-19 07:24:47,515 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) API rate limiting is disabled. We will not use ApiRateLimitService. 2024-09-19 07:24:47,520 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Received unknown parameters for command addHost. Unknown parameters : clustertype 2024-09-19 07:24:47,528 INFO [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Trying to add a new host at http://192.168.200.238 in data center 1 2024-09-19 07:24:47,749 WARN [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Failed to authenticate with ssh key 2024-09-19 07:24:47,750 INFO [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Failed to authenticate with ssh key, retrying with password 2024-09-19 07:24:47,784 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: ls /dev/kvm 2024-09-19 07:24:49,149 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: ls /dev/kvm 2024-09-19 07:24:49,160 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ 2024-09-19 07:24:50,831 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ 2024-09-19 07:24:50,831 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ 2024-09-19 07:24:52,675 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ 2024-09-19 07:24:52,725 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties ykT2GOj8BCqvXvQR /etc/cloudstack/agent/ 2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted. 2024-09-19 07:24:55,082 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: cloudstack-setup-agent -m 192.168.200.215 -z 1 -p 1 -c 1 -g 50638cf5-4698-39d1-b55c-3b7b1bb9647a -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm 2024-09-19 07:25:00,349 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: cloudstack-setup-agent -m 192.168.200.215 -z 1 -p 1 -c 1 -g 50638cf5-4698-39d1-b55c-3b7b1bb9647a -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm 2024-09-19 07:30:00,373 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Timeout, to wait for the host connecting to mgt svr, assuming it is failed 2024-09-19 07:30:00,375 WARN [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Cannot find the server resources at http://192.168.200.238 2024-09-19 07:30:00,375 INFO [c.c.u.e.CSExceptionErrorCode] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions 2024-09-19 07:30:00,378 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Exception: 2024-09-19 07:30:00,380 INFO [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Unable to add the host: Cannot find the server resources at http://192.168.200.238 2024-09-19 07:30:00,381 DEBUG [c.c.a.ApiServlet] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) ===END=== 192.168.232.15 -- POST command=addHost&response=json

weizhouapache commented 2 months ago

@subincb30 can you upload the file /var/log/cloudstack/agent/setup.log on the kvm host ?

subincb30 commented 2 months ago

@weizhouapache Please find the log below:

root@host:~# tail -f /var/log/cloudstack/agent/setup.log DEBUG:root:execute:sudo /usr/sbin/service network-manager start DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found. DEBUG:root:execute:sudo /usr/sbin/service network-manager status DEBUG:root:Failed to execute:Unit network-manager.service could not be found. DEBUG:root:execute:sudo /usr/sbin/service network-manager start DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found. DEBUG:root:execute:/etc/init.d/networking stop DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found DEBUG:root:execute:/etc/init.d/networking start DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found

subincb30 commented 2 months ago

Someone please help me to resolve the issue.

weizhouapache commented 2 months ago

@subincb30 have you configured cloudbr0 on the kvm host ?

subincb30 commented 2 months ago

Can you give me complete guide for that? I got some doc for for that but it was not clear. But, I found one new network interface in my server.

5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:fe:08:e5 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever

Can you help me setting up cloudbr0?

weizhouapache commented 2 months ago

@subincb30 please refer to https://docs.cloudstack.apache.org/en/4.19.1.1/installguide/hypervisor/kvm.html

subincb30 commented 2 months ago

@weizhouapache I already checked this earlier. But, I am little bit confussed on it. I will paste my current netplan settings below.

root@host:~# cat /etc/netplan/00-installer-config.yaml

This is the network config written by 'subiquity'

network: renderer: networkd ethernets: enp1s0: dhcp4: true enp2s0: addresses:

Can you help me adding the additional parameters in this file.

weizhouapache commented 2 months ago

@subincb30 the doc has provided an example of netplan

subincb30 commented 2 months ago

@weizhouapache Yes I saw that. But some parameters are confusing for me. Can you check my current settings which I shared earlier and give an example accordingly?