search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.05k stars 1.1k forks source link

Incorrect Network setup on agent (ACS 4.19.1 ; Ubuntu Server 24.04.1) #9777

Closed ong228 closed 3 days ago

ong228 commented 3 days ago
ISSUE TYPE
COMPONENT NAME
Initial setup adding KVM host to ACS MGMT
CLOUDSTACK VERSION
4.19.1
CONFIGURATION

Picked Core -> Advanced Networking ; Has basic Zone, Pod and Cluster

OS / ENVIRONMENT

Ubuntu Server 24.04.1 LTS

SUMMARY
STEPS TO REPRODUCE
Ubuntu Server KVM Host Netplan

network:
  version: 2
  renderer: networkd
  ethernets:
      ens3f0:
        dhcp4: no
      ens3f1:
        dhcp4: no
      ens3f2:
        dhcp4: no
      ens3f3:
        dhcp4: no
      ens3f4:
        dhcp4: no
      ens3f5:
        dhcp4: no
      ens3f6:
        dhcp4: no
      ens3f7:
        dhcp4: no
  bonds:
    MgmtBond:
      interfaces: [ens3f0,ens3f1]
      parameters:
        mode: 802.3ad
      mtu: 9000
    NFSBond:
      interfaces: [ens3f2,ens3f3]
      parameters:
        mode: 802.3ad
      mtu: 9000
    TunnelsBond:
      interfaces: [ens3f4,ens3f5]
      parameters:
        mode: 802.3ad
      mtu: 9000
    NotUsedBond:
      interfaces: [ens3f6,ens3f7]
      parameters:
        mode: 802.3ad
      mtu: 9000
  bridges:
    MgmtNetVLAN600:
      addresses: [x.x.x.x/23]
      dhcp4: no
      routes:
        - to: default
          via: x.x.x.x
      nameservers:
        addresses: [xxxxxxxxxxxxx]
        search: [xxxxxxxxxxxx]
      interfaces:
        - MgmtBond
    NFSNetVLAN600:
      addresses: [x.x.x.x]
      dhcp4: no
      nameservers:
        addresses: [xxxxxxxx]
        search: [xxxx]
      interfaces:
        - NFSBond
    GuestNet:
      dhcp4: no
      interfaces:
       - TunnelsBond

brctl show

bridge name     bridge id               STP enabled     interfaces
GuestNet                8000.829f60077655       no              TunnelsBond
MgmtNetVLAN600          8000.b678ceb71e00       no              MgmtBond
NFSNetVLAN600           8000.6a708247dfd8       no              NFSBond
cloud0          8000.929cbacc7081       no
EXPECTED RESULTS
Adding host works i suppose. It's my first time setup.

Note : 
It worked when i create only bonding in Netplan and use MgmtNetVLAN600 as name with IP/GW/DNS etc (no bridges created in Netplan at all) but subsequent setup with System VMs and Guest VMs did not work hence I am trying this route with creating bonds, and then bridges too
ACTUAL RESULTS

ACS Management log ('error cannot find network MgmtNetVLAN600')

2024-10-08 03:57:55,584 INFO  [c.c.r.ResourceManagerImpl] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Trying to add a new host at http://<KVM-Host-FQDN> in data center 2
2024-10-08 03:57:55,705 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Executing cmd: ls /dev/kvm
2024-10-08 03:57:56,580 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-26714a4f) (logid:622d07be) HA health check task is running...
2024-10-08 03:57:57,046 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) SSH command: ls /dev/kvm
SSH command output:/dev/kvm

2024-10-08 03:57:57,052 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-10-08 03:57:58,024 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-198c6ffd) (logid:cadd9377) Skip capacity scan as there is no Primary Storage in 'Up' state
2024-10-08 03:57:58,192 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-5000b4f8) (logid:010e37e4) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:57:58,192 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-5000b4f8) (logid:010e37e4) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:57:58,196 DEBUG [c.c.a.ApiServer] (qtp1374754488-20:ctx-5000b4f8 ctx-e2bb36a6) (logid:010e37e4) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:57:58,198 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-20:ctx-5000b4f8 ctx-e2bb36a6) (logid:010e37e4) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:57:58,199 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-20:ctx-5000b4f8 ctx-e2bb36a6) (logid:010e37e4) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:57:58,200 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-5000b4f8 ctx-e2bb36a6) (logid:010e37e4) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:57:58,226 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-a306536e) (logid:89796b84) Enabled non-edge zones available for scan:
2024-10-08 03:57:58,755 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying

2024-10-08 03:57:58,755 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-10-08 03:58:00,581 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-02918256) (logid:f9c1d55b) HA health check task is running...
2024-10-08 03:58:00,899 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:

2024-10-08 03:58:00,924 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties qVyN7RKCCM6x48Gf /etc/cloudstack/agent/
2024-10-08 03:58:03,193 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-6b786ed9) (logid:2b4b154c) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:03,193 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-6b786ed9) (logid:2b4b154c) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:03,197 DEBUG [c.c.a.ApiServer] (qtp1374754488-20:ctx-6b786ed9 ctx-ad59e666) (logid:2b4b154c) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:03,200 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-20:ctx-6b786ed9 ctx-ad59e666) (logid:2b4b154c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:03,200 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-20:ctx-6b786ed9 ctx-ad59e666) (logid:2b4b154c) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:03,201 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-6b786ed9 ctx-ad59e666) (logid:2b4b154c) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:03,252 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Succeeded to import certificate in the keystore for agent on the KVM host: <KVM-Host-IP>. Agent secured and trusted.
2024-10-08 03:58:03,255 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) Executing cmd: sudo cloudstack-setup-agent  -m <MGMT-IP> -z 2 -p 2 -c 1 -g e47f5b77-147f-36d8-8619-1e67f220d5e1 -a -s  --pubNic=GuestNet --prvNic=MgmtNetVLAN600 --guestNic=GuestNet --hypervisor=kvm
2024-10-08 03:58:04,582 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-6316354a) (logid:dadb8ac5) HA health check task is running...
2024-10-08 03:58:05,696 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-9cfb730e) (logid:c92d1c32) Found 0 routers to update status.
2024-10-08 03:58:05,697 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-9cfb730e) (logid:c92d1c32) Found 0 VPC's to update Redundant State.
2024-10-08 03:58:05,698 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-9cfb730e) (logid:c92d1c32) Found 0 networks to update RvR status.
2024-10-08 03:58:05,728 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-488064dd) (logid:0ad81c25) DbCollector is running...
2024-10-08 03:58:06,726 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) ManagementServerCollector is running...
2024-10-08 03:58:06,727 DEBUG [c.c.u.d.DbProperties] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) DB properties were already loaded
2024-10-08 03:58:06,727 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c systemctl status cloudstack-usage | grep "  Loaded:" ].
2024-10-08 03:58:06,734 WARN  [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Execution of process [56039] for command [/bin/bash -c systemctl status cloudstack-usage | grep "  Loaded:" ] failed.
2024-10-08 03:58:06,734 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Exit value of process [56039] for command [/bin/bash -c systemctl status cloudstack-usage | grep "  Loaded:" ] is [1].
2024-10-08 03:58:06,734 WARN  [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Process [56039] for command [/bin/bash -c systemctl status cloudstack-usage | grep "  Loaded:" ] encountered the error: [Unit cloudstack-usage.service could not be found.].
2024-10-08 03:58:06,735 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) usage install: null
2024-10-08 03:58:06,735 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Sessions active: 1
2024-10-08 03:58:06,736 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ].
2024-10-08 03:58:06,739 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56042] for command [/bin/bash -c cat /etc/os-release | grep PRETTY_NAME | cut -f2 -d '=' | tr -d '"' ].
2024-10-08 03:58:06,739 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c uname -r ].
2024-10-08 03:58:06,741 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56047] for command [/bin/bash -c uname -r ].
2024-10-08 03:58:06,742 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ].
2024-10-08 03:58:06,744 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56048] for command [/bin/bash -c cat /proc/meminfo | grep MemTotal | cut -f 2 -d ':' | tr -d 'a-zA-z ' ].
2024-10-08 03:58:06,744 INFO  [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) system memory from /proc: 8326586368
2024-10-08 03:58:06,745 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ].
2024-10-08 03:58:06,747 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56053] for command [/bin/bash -c cat /proc/meminfo | grep MemFree | cut -f 2 -d ':' | tr -d 'a-zA-z ' ].
2024-10-08 03:58:06,747 INFO  [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) free memory from /proc: 4363476992
2024-10-08 03:58:06,747 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c ps -o rss= 1175 ].
2024-10-08 03:58:06,755 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56058] for command [/bin/bash -c ps -o rss= 1175 ].
2024-10-08 03:58:06,755 INFO  [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) used memory from /proc: 816200
2024-10-08 03:58:06,755 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c uptime -s ].
2024-10-08 03:58:06,757 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56059] for command [/bin/bash -c uptime -s ].
2024-10-08 03:58:06,758 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c ps -o vsz= 1175 ].
2024-10-08 03:58:06,764 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56060] for command [/bin/bash -c ps -o vsz= 1175 ].
2024-10-08 03:58:06,765 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ].
2024-10-08 03:58:06,768 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56061] for command [/bin/bash -c cat /proc/cpuinfo | grep "cpu MHz" | grep "cpu MHz" | cut -f 2 -d : | tr -d ' '| tr '\n' " " ].
2024-10-08 03:58:06,768 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c cat /proc/loadavg ].
2024-10-08 03:58:06,770 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56068] for command [/bin/bash -c cat /proc/loadavg ].
2024-10-08 03:58:06,770 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ].
2024-10-08 03:58:06,773 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56069] for command [/bin/bash -c cat /proc/stat | grep "cpu " | tr -d "cpu" ].
2024-10-08 03:58:06,773 DEBUG [c.c.u.LogUtils] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) file for FILE : /var/log/cloudstack/management/management-server.log
2024-10-08 03:58:06,773 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c du -sh /var/log/cloudstack/management/management-server.log | cut -f '1' ].
2024-10-08 03:58:06,775 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56073] for command [/bin/bash -c du -sh /var/log/cloudstack/management/management-server.log | cut -f '1' ].
2024-10-08 03:58:06,776 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Executing command [/bin/bash -c df -h /var/log/cloudstack/management/management-server.log | grep -v Filesystem | awk '{print "on disk " $1 " mounted on " $6 " (" $5 " full)"}' ].
2024-10-08 03:58:06,779 DEBUG [c.c.u.s.Script] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Successfully executed process [56076] for command [/bin/bash -c df -h /var/log/cloudstack/management/management-server.log | grep -v Filesystem | awk '{print "on disk " $1 " mounted on " $6 " (" $5 " full)"}' ].
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail buffersdirect.capacity, 242660
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail buffersdirect.count, 20
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail buffersdirect.used, 242660
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail buffersmapped.capacity, 0
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail buffersmapped.count, 0
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail buffersmapped.used, 0
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail gcPS-MarkSweep.count, 6
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail gcPS-MarkSweep.time, 1180
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail gcPS-Scavenge.count, 321
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail gcPS-Scavenge.time, 1995
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail jvmname, 1175@<MGMT-FQDN>
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail jvmuptime, 73349034
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail jvmvendor, Ubuntu OpenJDK 64-Bit Server VM 11.0.24+8-post-Ubuntu-1ubuntu324.04.1 (11)
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memoryheap.committed, 346030080
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memoryheap.init, 132120576
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memoryheap.usage, 0.10579723933344849
2024-10-08 03:58:06,780 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorynon-heap.committed, 221929472
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorynon-heap.init, 7667712
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorynon-heap.max, -1
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorynon-heap.usage, -2.14037608E8
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorynon-heap.used, 214037608
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.3162087429775281
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.17123975273260464
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.36812579145561186
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.Compressed-Class-Space.usage, 0.011766009032726288
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.Metaspace.usage, 0.9737033644206021
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.PS-Eden-Space.usage, 0.011914277759919177
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.PS-Old-Gen.usage, 0.13457664732791616
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorypools.PS-Survivor-Space.usage, 0.7812652587890625
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorytotal.committed, 567959552
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorytotal.init, 139788288
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorytotal.max, 1908932607
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail memorytotal.used, 416075200
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail threadsdeadlocks, []
2024-10-08 03:58:06,781 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail threadsnew.count, 0
2024-10-08 03:58:06,782 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) not storing detail threadstimed_waiting.count, 80
2024-10-08 03:58:06,783 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-5:ctx-0579e6a1) (logid:ed27ba51) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"79a10d06-c488-41cf-969d-3f31b75435f9","collectionTime":"Oct 8, 2024, 3:58:06 AM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":346030080,"freeJvmMemoryBytes":145056344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":73348988,"jvmStartTime":1728286537893,"availableProcessors":4,"loadAverage":0.01708984375,"totalInit":139788288,"totalUsed":415014000,"totalCommitted":567959552,"pid":1175,"jvmName":"1175@<MGMT-FQDN>","jvmVendor":"Ubuntu","jvmVersion":"11.0.24+8-post-Ubuntu-1ubuntu324.04.1","osDistribution":"Ubuntu 24.04.1 LTS","agentCount":0,"heapMemoryUsed":201959800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":15,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":129,"systemMemoryTotal":8326586368,"systemMemoryFree":4363476992,"systemMemoryUsed":816200,"systemMemoryVirtualSize":6234066944,"logInfo":"/var/log/cloudstack/management/management-server.log using: 6.2M\non disk /dev/mapper/ubuntu--vg-ubuntu--lv mounted on / (5% full)","systemTotalCpuCycles":9577.496,"systemLoadAverages":[0.02,0.01,0.0],"systemCyclesUsage":[79069,40903,29083240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Oct 7, 2024, 9:35:30 AM","kernelVersion":"6.8.0-45-generic"} to 345051037460
2024-10-08 03:58:06,783 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:ctx-f8aab4de) (logid:39e2a891) Cluster PDU 345051037460 -> 345051037460. agent: 0, pdu seq: 2445, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"79a10d06-c488-41cf-969d-3f31b75435f9","collectionTime":"Oct 8, 2024, 3:58:06 AM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":346030080,"freeJvmMemoryBytes":145056344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":73348988,"jvmStartTime":1728286537893,"availableProcessors":4,"loadAverage":0.01708984375,"totalInit":139788288,"totalUsed":415014000,"totalCommitted":567959552,"pid":1175,"jvmName":"1175@<MGMT-FQDN>","jvmVendor":"Ubuntu","jvmVersion":"11.0.24+8-post-Ubuntu-1ubuntu324.04.1","osDistribution":"Ubuntu 24.04.1 LTS","agentCount":0,"heapMemoryUsed":201959800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":15,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":129,"systemMemoryTotal":8326586368,"systemMemoryFree":4363476992,"systemMemoryUsed":816200,"systemMemoryVirtualSize":6234066944,"logInfo":"/var/log/cloudstack/management/management-server.log using: 6.2M\non disk /dev/mapper/ubuntu--vg-ubuntu--lv mounted on / (5% full)","systemTotalCpuCycles":9577.496,"systemLoadAverages":[0.02,0.01,0.0],"systemCyclesUsage":[79069,40903,29083240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Oct 7, 2024, 9:35:30 AM","kernelVersion":"6.8.0-45-generic"}
2024-10-08 03:58:06,783 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:ctx-f8aab4de) (logid:39e2a891) Executing ClusterServicePdu with service URL: https://<MGMT-IP>:9090/clusterservice
2024-10-08 03:58:06,787 DEBUG [c.c.s.StatsCollector] (Cluster-Worker-547:ctx-9bcae50b) (logid:b56f5716) StatusUpdate from 345051037460, json: {"managementServerHostId":1,"managementServerHostUuid":"79a10d06-c488-41cf-969d-3f31b75435f9","collectionTime":"Oct 8, 2024, 3:58:06 AM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":346030080,"freeJvmMemoryBytes":145056344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":73348988,"jvmStartTime":1728286537893,"availableProcessors":4,"loadAverage":0.01708984375,"totalInit":139788288,"totalUsed":415014000,"totalCommitted":567959552,"pid":1175,"jvmName":"1175@<MGMT-FQDN>","jvmVendor":"Ubuntu","jvmVersion":"11.0.24+8-post-Ubuntu-1ubuntu324.04.1","osDistribution":"Ubuntu 24.04.1 LTS","agentCount":0,"heapMemoryUsed":201959800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":15,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":129,"systemMemoryTotal":8326586368,"systemMemoryFree":4363476992,"systemMemoryUsed":816200,"systemMemoryVirtualSize":6234066944,"logInfo":"/var/log/cloudstack/management/management-server.log using: 6.2M\non disk /dev/mapper/ubuntu--vg-ubuntu--lv mounted on / (5% full)","systemTotalCpuCycles":9577.496,"systemLoadAverages":[0.02,0.01,0.0],"systemCyclesUsage":[79069,40903,29083240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Oct 7, 2024, 9:35:30 AM","kernelVersion":"6.8.0-45-generic"}
2024-10-08 03:58:06,788 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:ctx-f8aab4de) (logid:39e2a891) POST https://<MGMT-IP>:9090/clusterservice response :true, responding time: 4 ms
2024-10-08 03:58:06,788 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-4:ctx-f8aab4de) (logid:39e2a891) Cluster PDU 345051037460 -> 345051037460 completed. time: 4ms. agent: 0, pdu seq: 2445, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"79a10d06-c488-41cf-969d-3f31b75435f9","collectionTime":"Oct 8, 2024, 3:58:06 AM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":346030080,"freeJvmMemoryBytes":145056344,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":73348988,"jvmStartTime":1728286537893,"availableProcessors":4,"loadAverage":0.01708984375,"totalInit":139788288,"totalUsed":415014000,"totalCommitted":567959552,"pid":1175,"jvmName":"1175@<MGMT-FQDN>","jvmVendor":"Ubuntu","jvmVersion":"11.0.24+8-post-Ubuntu-1ubuntu324.04.1","osDistribution":"Ubuntu 24.04.1 LTS","agentCount":0,"heapMemoryUsed":201959800,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":15,"threadsRunnableCount":9,"threadsTerminatedCount":0,"threadsTotalCount":218,"threadsWaitingCount":129,"systemMemoryTotal":8326586368,"systemMemoryFree":4363476992,"systemMemoryUsed":816200,"systemMemoryVirtualSize":6234066944,"logInfo":"/var/log/cloudstack/management/management-server.log using: 6.2M\non disk /dev/mapper/ubuntu--vg-ubuntu--lv mounted on / (5% full)","systemTotalCpuCycles":9577.496,"systemLoadAverages":[0.02,0.01,0.0],"systemCyclesUsage":[79069,40903,29083240],"dbLocal":true,"usageLocal":false,"systemBootTime":"Oct 7, 2024, 9:35:30 AM","kernelVersion":"6.8.0-45-generic"}
2024-10-08 03:58:08,194 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-11ca45a7) (logid:5bcd6d11) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:08,194 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-11ca45a7) (logid:5bcd6d11) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:08,199 DEBUG [c.c.a.ApiServer] (qtp1374754488-16:ctx-11ca45a7 ctx-2589f2f9) (logid:5bcd6d11) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:08,201 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-16:ctx-11ca45a7 ctx-2589f2f9) (logid:5bcd6d11) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:08,201 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-16:ctx-11ca45a7 ctx-2589f2f9) (logid:5bcd6d11) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:08,203 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-11ca45a7 ctx-2589f2f9) (logid:5bcd6d11) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:08,583 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-fb5dfdb9) (logid:ec45238d) HA health check task is running...
2024-10-08 03:58:10,507 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-26a3c9e8) (logid:fb4d56c5) VM scheduler.poll is being called at 2024-10-08 03:58:00 GMT
2024-10-08 03:58:10,509 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-26a3c9e8) (logid:fb4d56c5) Got 0 scheduled jobs to be executed at 2024-10-08 03:58:00 GMT
2024-10-08 03:58:10,510 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-9bcf7c1d) (logid:37e7ebad) VM Operation Thread Running
2024-10-08 03:58:10,510 INFO  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-26a3c9e8) (logid:fb4d56c5) Cleaned up 0 VM scheduled job entries
2024-10-08 03:58:10,864 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-c78b7db0) (logid:b6dd136a) ClusterDRS.poll is being called at 2024-10-08 03:58:00 GMT
2024-10-08 03:58:10,869 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-c78b7db0) (logid:b6dd136a) Removed 0 old drs migration plans
2024-10-08 03:58:12,584 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-049d2459) (logid:fa89dc81) HA health check task is running...
2024-10-08 03:58:13,198 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-38558ebf) (logid:d3629985) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:13,198 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-38558ebf) (logid:d3629985) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:13,202 DEBUG [c.c.a.ApiServer] (qtp1374754488-20:ctx-38558ebf ctx-473196ab) (logid:d3629985) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:13,205 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-20:ctx-38558ebf ctx-473196ab) (logid:d3629985) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:13,205 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-20:ctx-38558ebf ctx-473196ab) (logid:d3629985) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:13,206 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-38558ebf ctx-473196ab) (logid:d3629985) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:16,585 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-3e103b21) (logid:ff5eb29b) HA health check task is running...
2024-10-08 03:58:18,185 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-8ca4f6cb) (logid:7f57b394) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:18,186 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-8ca4f6cb) (logid:7f57b394) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:18,189 DEBUG [c.c.a.ApiServer] (qtp1374754488-20:ctx-8ca4f6cb ctx-27cdc0e8) (logid:7f57b394) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:18,191 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-20:ctx-8ca4f6cb ctx-27cdc0e8) (logid:7f57b394) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:18,191 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-20:ctx-8ca4f6cb ctx-27cdc0e8) (logid:7f57b394) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:18,192 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-8ca4f6cb ctx-27cdc0e8) (logid:7f57b394) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:20,586 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-6e808a10) (logid:54d5deba) HA health check task is running...
2024-10-08 03:58:22,362 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-f5aa7a82) (logid:1e3291d5) VmStatsCollector is running to process VMs across 0 UP hosts
2024-10-08 03:58:22,852 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-a5163f35) (logid:bc57c537) AutoScaling Monitor is running...
2024-10-08 03:58:22,852 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-c9dbe77c) (logid:569e7a8b) StorageCollector is running...
2024-10-08 03:58:22,853 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-037ca908) (logid:4226b488) HostStatsCollector is running to process 0 UP hosts
2024-10-08 03:58:23,186 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-43402c73) (logid:f6c7d99c) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:23,186 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-43402c73) (logid:f6c7d99c) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:23,190 DEBUG [c.c.a.ApiServer] (qtp1374754488-16:ctx-43402c73 ctx-628d3fa6) (logid:f6c7d99c) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:23,192 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-16:ctx-43402c73 ctx-628d3fa6) (logid:f6c7d99c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:23,192 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-16:ctx-43402c73 ctx-628d3fa6) (logid:f6c7d99c) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:23,193 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-43402c73 ctx-628d3fa6) (logid:f6c7d99c) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:23,474 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:ctx-cce905a0) (logid:d71b10bd) Starting to remove all vm_stats rows older than [Mon Oct 07 15:58:23 UTC 2024].
2024-10-08 03:58:23,474 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:ctx-45a5d6e5) (logid:2dcaa3bb) Starting to remove all volume_stats rows older than [Mon Oct 07 15:58:23 UTC 2024].
2024-10-08 03:58:23,474 INFO  [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:ctx-cce905a0) (logid:d71b10bd) Removed a total of [0] vm_stats rows older than [Mon Oct 07 15:58:23 UTC 2024].
2024-10-08 03:58:23,474 INFO  [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-2:ctx-45a5d6e5) (logid:2dcaa3bb) Removed a total of [0] volume_stats rows older than [Mon Oct 07 15:58:23 UTC 2024].
2024-10-08 03:58:24,587 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-94d873b3) (logid:3fd55037) HA health check task is running...
2024-10-08 03:58:28,024 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-e26978a2) (logid:21221808) Skip capacity scan as there is no Primary Storage in 'Up' state
2024-10-08 03:58:28,188 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-fce647a5) (logid:b0c9e2c9) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:28,188 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-fce647a5) (logid:b0c9e2c9) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:28,192 DEBUG [c.c.a.ApiServer] (qtp1374754488-20:ctx-fce647a5 ctx-ed35d142) (logid:b0c9e2c9) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:28,194 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-20:ctx-fce647a5 ctx-ed35d142) (logid:b0c9e2c9) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:28,194 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-20:ctx-fce647a5 ctx-ed35d142) (logid:b0c9e2c9) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:28,195 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-fce647a5 ctx-ed35d142) (logid:b0c9e2c9) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:28,226 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-a2b4b642) (logid:5e54b63d) Enabled non-edge zones available for scan:
2024-10-08 03:58:28,587 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-4ea857f0) (logid:c3de6d33) HA health check task is running...
2024-10-08 03:58:32,588 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-ec6a2c91) (logid:7c6d84e2) HA health check task is running...
2024-10-08 03:58:33,195 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-aadf2ecb) (logid:a0b7bb76) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:33,195 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-aadf2ecb) (logid:a0b7bb76) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:33,199 DEBUG [c.c.a.ApiServer] (qtp1374754488-16:ctx-aadf2ecb ctx-03b1b2a0) (logid:a0b7bb76) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:33,201 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-16:ctx-aadf2ecb ctx-03b1b2a0) (logid:a0b7bb76) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:33,202 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-16:ctx-aadf2ecb ctx-03b1b2a0) (logid:a0b7bb76) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:33,203 DEBUG [c.c.a.ApiServlet] (qtp1374754488-16:ctx-aadf2ecb ctx-03b1b2a0) (logid:a0b7bb76) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:35,697 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-cf97d28b) (logid:4a97aae5) Found 0 routers to update status.
2024-10-08 03:58:35,697 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-cf97d28b) (logid:4a97aae5) Found 0 VPC's to update Redundant State.
2024-10-08 03:58:35,698 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-cf97d28b) (logid:4a97aae5) Found 0 networks to update RvR status.
2024-10-08 03:58:36,589 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-bce0b9da) (logid:7177fb9f) HA health check task is running...
2024-10-08 03:58:37,647 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1374754488-18:ctx-98530a47 ctx-f1ee4153) (logid:af1051eb) SSH command: sudo cloudstack-setup-agent  -m <MGMT-IP> -z 2 -p 2 -c 1 -g e47f5b77-147f-36d8-8619-1e67f220d5e1 -a -s  --pubNic=GuestNet --prvNic=MgmtNetVLAN600 --guestNic=GuestNet --hypervisor=kvm
SSH command output:Starting to configure your system:
Configure Host ...            [OK]
Configure Apparmor ...        [OK]
Configure Network ...         [OK]
Configure Libvirt ...         [OK]
Configure Firewall ...        [OK]
Configure Nfs ...             [OK]
Configure cloudAgent ...      [OK]
CloudStack Agent setup is done!

2024-10-08 03:58:38,192 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-11e6b25e) (logid:1e13b99b) ===START===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:38,192 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-11e6b25e) (logid:1e13b99b) Two factor authentication is already verified for the user 2, so skipping
2024-10-08 03:58:38,196 DEBUG [c.c.a.ApiServer] (qtp1374754488-20:ctx-11e6b25e ctx-f122db45) (logid:1e13b99b) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"9995c13e-808d-11ef-8d51-005056a4ff14"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-10-08 03:58:38,198 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1374754488-20:ctx-11e6b25e ctx-f122db45) (logid:1e13b99b) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-10-08 03:58:38,198 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1374754488-20:ctx-11e6b25e ctx-f122db45) (logid:1e13b99b) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-10-08 03:58:38,200 DEBUG [c.c.a.ApiServlet] (qtp1374754488-20:ctx-11e6b25e ctx-f122db45) (logid:1e13b99b) ===END===  x.x.x.x -- GET  command=readyForShutdown&response=json
2024-10-08 03:58:38,842 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-462-thread-1:null) (logid:) A client/agent attempting connection from address=<KVM-Host-IP> has presented these certificate(s):
Certificate [1] :
 Serial: 79f82e8eb188419f
  Not Before:Mon Oct 07 15:58:00 UTC 2024
  Not After:Wed Oct 08 03:58:00 UTC 2025
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=<KVM-Host-FQDN>
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:[[7, x.x.x.x], [7, <KVM-Host-IP>], [2, <KVM-Host-IP>], [2, <KVM-Host-FQDN>]]
Certificate [2] :
 Serial: 1466753dbfeb19c6
  Not Before:Tue Oct 01 19:11:59 UTC 2024
  Not After:Fri Sep 25 07:11:59 UTC 2054
  Signature Algorithm:SHA256withRSA
  Version:3
  Subject DN:CN=ca.cloudstack.apache.org
  Issuer DN:CN=ca.cloudstack.apache.org
  Alternative Names:null
2024-10-08 03:58:38,844 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-462-thread-1:null) (logid:) Client/agent connection from ip=<KVM-Host-IP> has been validated and trusted.
2024-10-08 03:58:39,067 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq -1-0: Scheduling the first command  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"4","cpus":"96","speed":"2400","memory":"810159267840","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"6.8.0-45-generic","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"24.04","host.volume.encryption":"true","host.instance.conversion":"true","secured":"false","Host.OS":"Ubuntu"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"2","pod":"2","cluster":"1","guid":"e47f5b77-147f-36d8-8619-1e67f220d5e1-LibvirtComputingResource","name":"<KVM-Host-FQDN>","version":"4.19.1.1","iqn":"iqn.2004-10.com.ubuntu:01:cf82f14213f1","privateIpAddress":"<KVM-Host-IP>","privateMacAddress":"b6:78:ce:b7:1e:00","privateNetmask":"255.255.254.0","storageIpAddress":"<KVM-Host-IP>","storageNetmask":"255.255.254.0","storageMacAddress":"b6:78:ce:b7:1e:00","resourceName":"LibvirtComputingResource","gatewayIpAddress":"x.x.x.x","msHostList":"<MGMT-IP>","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"1e6b4fd8-7ce4-4237-8b37-f18962e18afc","host":"<KVM-Host-IP>","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(436.01 GB) 468157972480","availableBytes":"(423.06 GB) 454255984640"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"2","pod":"2","guid":"e47f5b77-147f-36d8-8619-1e67f220d5e1-LibvirtComputingResource","name":"<KVM-Host-FQDN>","version":"4.19.1.1","resourceName":"LibvirtComputingResource","msHostList":"<MGMT-IP>","wait":"0","bypassHostMaintenance":"false"}}] }
2024-10-08 03:58:39,069 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Seq -1-0: Processing the first command  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":"4","cpus":"96","speed":"2400","memory":"810159267840","dom0MinMemory":"1073741824","poolSync":"false","supportsClonedVolumes":"false","caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"6.8.0-45-generic","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"24.04","host.volume.encryption":"true","host.instance.conversion":"true","secured":"false","Host.OS":"Ubuntu"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"2","pod":"2","cluster":"1","guid":"e47f5b77-147f-36d8-8619-1e67f220d5e1-LibvirtComputingResource","name":"<KVM-Host-FQDN>","version":"4.19.1.1","iqn":"iqn.2004-10.com.ubuntu:01:cf82f14213f1","privateIpAddress":"<KVM-Host-IP>","privateMacAddress":"b6:78:ce:b7:1e:00","privateNetmask":"255.255.254.0","storageIpAddress":"<KVM-Host-IP>","storageNetmask":"255.255.254.0","storageMacAddress":"b6:78:ce:b7:1e:00","resourceName":"LibvirtComputingResource","gatewayIpAddress":"x.x.x.x","msHostList":"<MGMT-IP>","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":"(0 bytes) 0","poolInfo":{"uuid":"1e6b4fd8-7ce4-4237-8b37-f18962e18afc","host":"<KVM-Host-IP>","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":"(436.01 GB) 468157972480","availableBytes":"(423.06 GB) 454255984640"},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"2","pod":"2","guid":"e47f5b77-147f-36d8-8619-1e67f220d5e1-LibvirtComputingResource","name":"<KVM-Host-FQDN>","version":"4.19.1.1","resourceName":"LibvirtComputingResource","msHostList":"<MGMT-IP>","wait":"0","bypassHostMaintenance":"false"}}] }
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to CiscoVnmcElement
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BareMetalDiscoverer
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BrocadeVcsElement
2024-10-08 03:58:39,071 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer
2024-10-08 03:58:39,071 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) createHostVOForConnectedAgent: Host {"id":0,"name":"<KVM-Host-FQDN>","type":null,"uuid":"d5022935-187b-4714-8067-4d39cdf0345b"}
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to LxcServerDiscoverer
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovs
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to VmwareServerDiscoverer
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to OvmDiscoverer
2024-10-08 03:58:39,072 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
2024-10-08 03:58:39,088 DEBUG [c.c.r.ResourceState] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Resource state update: [id = 15; name = <KVM-Host-FQDN>; old state = Creating; event = InternalCreated; new state = Enabled]
2024-10-08 03:58:39,088 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Transition:[Resource state = Enabled, Agent event = AgentConnected, Host id = 15, name = <KVM-Host-FQDN>]
2024-10-08 03:58:39,096 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Agent's management server host list is not up to date, sending list update:[<MGMT-IP>]
2024-10-08 03:58:39,096 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) create ClusteredAgentAttache for 15
2024-10-08 03:58:39,096 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: XcpServerDiscoverer
2024-10-08 03:58:39,096 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Not XenServer so moving on.
2024-10-08 03:58:39,097 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: HypervServerDiscoverer
2024-10-08 03:58:39,097 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Not Hyper-V hypervisor, so moving on.
2024-10-08 03:58:39,097 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: TungstenElement
2024-10-08 03:58:39,097 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: SecondaryStorageListener
2024-10-08 03:58:39,097 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: SecurityGroupListener
2024-10-08 03:58:39,097 INFO  [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Received a host startup notification
2024-10-08 03:58:39,098 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Seq 15-5367727805872209921: Sending  { Cmd , MgmtId: 345051037460, via: 15(<KVM-Host-FQDN>), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":"2529","wait":"0","bypassHostMaintenance":"false"}}] }
2024-10-08 03:58:39,098 INFO  [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Scheduled network rules cleanup, interval=2529
2024-10-08 03:58:39,098 INFO  [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Received a host startup notification
2024-10-08 03:58:39,098 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: StoragePoolMonitor
2024-10-08 03:58:39,101 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending Connect to listener: NetworkOrchestrator
2024-10-08 03:58:39,102 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Host's hypervisorType is: KVM
2024-10-08 03:58:39,106 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Sending CheckNetworkCommand to check the Network is setup correctly on Agent
2024-10-08 03:58:39,108 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Seq 15-5367727805872209922: Sending  { Cmd , MgmtId: 345051037460, via: 15(<KVM-Host-FQDN>), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.CheckNetworkCommand":{"networkInfoList":[{"physicalNetworkId":"203","privateNetworkName":"MgmtNetVLAN600"},{"physicalNetworkId":"204","storageNetworkName":"NFSNetVLAN602"},{"physicalNetworkId":"210","publicNetworkName":"GuestNet","guestNetworkName":"GuestNet"}],"wait":"0","bypassHostMaintenance":"false"}}] }
2024-10-08 03:58:39,109 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:null) (logid:) Seq 15-5367727805872209921: Processing:  { Ans: , MgmtId: 345051037460, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2024-10-08 03:58:39,220 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Ping from Routing host 15(<KVM-Host-FQDN>)
2024-10-08 03:58:39,220 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) (logid:) Seq 15-5367727805872209922: Processing:  { Ans: , MgmtId: 345051037460, via: 15, Ver: v1, Flags: 110, [{"com.cloud.agent.api.CheckNetworkAnswer":{"_reconnect":"false","result":"false","details":"Can not find network: MgmtNetVLAN600","wait":"0","bypassHostMaintenance":"false"}}] }
2024-10-08 03:58:39,220 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Seq 15-5367727805872209922: Received:  { Ans: , MgmtId: 345051037460, via: 15(<KVM-Host-FQDN>), Ver: v1, Flags: 110, { CheckNetworkAnswer } }
2024-10-08 03:58:39,221 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) (logid:) Seq 15-5367727805872209922: No more commands found
2024-10-08 03:58:39,221 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Process host VM state report from ping process. host: 15
2024-10-08 03:58:39,221 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Details from executing class com.cloud.agent.api.CheckNetworkCommand: Can not find network: MgmtNetVLAN600
2024-10-08 03:58:39,221 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Process VM state report. host: 15, number of records in report: 0
2024-10-08 03:58:39,221 WARN  [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Unable to setup agent 15 due to Can not find network: MgmtNetVLAN600
2024-10-08 03:58:39,221 WARN  [c.c.a.AlertManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) alertType=[7] dataCenterId=[2] podId=[2] clusterId=[null] message=[Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: MgmtNetVLAN600].
2024-10-08 03:58:39,222 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Done with process of VM state report. host: 15
2024-10-08 03:58:39,223 INFO  [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) PingMap for agent: 15 will not be updated because agent is no longer in the PingMap
2024-10-08 03:58:39,226 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Not processing PingRoutingCommand for agent id=0; can't find the host in the DB
2024-10-08 03:58:39,228 WARN  [c.c.a.AlertManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: MgmtNetVLAN600] and content [Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: MgmtNetVLAN600].
2024-10-08 03:58:39,228 INFO  [c.c.u.e.CSExceptionErrorCode] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
2024-10-08 03:58:39,228 WARN  [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-456:ctx-ac8f76a7) (logid:ee7c4c43) Monitor NetworkOrchestrator says there is an error in the connect process for 15 due to Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: MgmtNetVLAN600
boring-cyborg[bot] commented 3 days ago

Thanks for opening your first issue here! Be sure to follow the issue template!

rajujith commented 3 days ago

@ong228

Ensure that the interfaces names to be used for configuring the bridges match one of the following patterns: ‘eth’, ‘bond’, ‘team’, ‘vlan’, ‘em’, ‘pp’, ‘ens’, ‘eno’, ‘enp’, ‘enx*’.

Otherwise, the KVM agent will not be able to configure the bridges properly.

Please change the bond name to bond* https://docs.cloudstack.apache.org/en/4.19.1.1/installguide/hypervisor/kvm.html#configuring-the-networking

You can find more specific error in the agent.log if you have DEBUG logging enabled.

ong228 commented 3 days ago

Thanks a lot @rajujith ! It's working now