search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
1.98k stars 1.09k forks source link

CloudStack agent is not working on Debian 11.5.0 #6944

Open Notespeak opened 1 year ago

Notespeak commented 1 year ago
ISSUE TYPE
COMPONENT NAME
cloudstack agent
CLOUDSTACK VERSION
Cloudstack version 4.17
CONFIGURATION

Networking config `auto eno1 iface eno1 inet manual auto eno2 iface eno2 inet manual auto br0 iface br0 inet static bridge_ports eno1 address 185.185.58.82/26 gateway 185.185.58.65 dns-nameservers 1.1.1.1 bridge_fd 15 bridge_hello 2 bridge_maxage 20 bridge_stp off

auto cloudbrvm0 iface cloudbrvm0 inet manual bridge_ports eno2 bridge_fd 15 bridge_hello 2 bridge_maxage 20 bridge_stp off

Host specs Dell R620 CPU E5-2695v2 Ram 64GB DDR3 ECC RAM Networking 4 x 1gb/s interfaces VT extentions support - Yes Nested Virtualization support - Yes

Vm host specs MacBook pro 2019 CPU I9-9900 RAM 32GB DDR4 Networking 1 bridged wifi interface VT extentions support - Yes Nested Virtualization support - Yes

OS / ENVIRONMENT

Debian 11.5.0

Java -v openjdk version "11.0.16" 2022-07-19 OpenJDK Runtime Environment (build 11.0.16+8-post-Debian-1deb11u1) OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Debian-1deb11u1, mixed mode, sharing)

Python -v Python 3.9.2

Libvirt version version: 7.0.0, package: 3

SUMMARY

I am getting repeatable errors when installing cloudstack-agent on a fresh Debian 11 installation. For some reason I am getting libvirtd errors, when I is installing as a dependancy of cloudstack-agent. I tried removing the pid and also purging libvirt and cloudstack agent, but unfortunately this didn't yield any results. I was also able to reproduce this behaviour on a vm.

STEPS TO REPRODUCE
Install a new OS ( Debian 11.5.0)
Install openjdk 11.0.16+8-1~deb11u1  
Install python3 connector
Setup Database
Install Cloudstack-management
Install Cloustack-agent 
Setup networking (networking config)
Configure uuid in cloudstack-agent
Observe agent failing
Check agent log ( log snippet 1)
Check libvirtd status ( libvirt deamon status)
Send a sample call to libvrtd service ( log snippet 2)
Enable debug level logs for libvirt
**
PS: shortened line by line console output are available in  "commands".

**log snippet 1**
`2022-12-03 10:08:35,792 WARN  [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) Duplicate StorageAdaptor type PowerFlex, not loading com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2022-12-03 10:08:35,793 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2022-12-03 10:08:35,799 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to get private nic name
2022-12-03 10:08:35,799 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: Failed to get private nic name
`
`● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-12-03 10:08:49 CST; 17min ago
TriggeredBy: ● libvirtd-ro.socket
             ● libvirtd.socket
             ● libvirtd-admin.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
   Main PID: 4210 (libvirtd)
      Tasks: 19 (limit: 32768)
     Memory: 11.2M
        CPU: 257ms
     CGroup: /system.slice/libvirtd.service
             └─4210 /usr/sbin/libvirtd

`
`libvirtd -v (server output)
2022-12-03 16:27:52.629+0000: 5169: info : libvirt version: 7.0.0, package: 3 (Andrea Bolognani <eof@kiyuko.org> Fri, 26 Feb 2021 16:46:34 +0100)
2022-12-03 16:27:52.629+0000: 5169: info : hostname: mskn101
2022-12-03 16:27:52.629+0000: 5169: info : virObjectNew:259 : OBJECT_NEW: obj=0x55b6a17dc5f0 classname=virAccessManager
2022-12-03 16:27:52.629+0000: 5169: info : virObjectNew:259 : OBJECT_NEW: obj=0x55b6a17dc650 classname=virAccessManager
2022-12-03 16:27:52.629+0000: 5169: info : virObjectRef:403 : OBJECT_REF: obj=0x55b6a17dc5f0
2022-12-03 16:27:52.629+0000: 5169: info : virObjectUnref:381 : OBJECT_UNREF: obj=0x55b6a17dc5f0
2022-12-03 16:27:52.629+0000: 5169: error : virPidFileAcquirePath:367 : Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2022-12-03 16:27:52.629+0000: 5169: info : virNetlinkEventServiceStopAll:956 : stopping all netlink event services
2022-12-03 16:27:52.629+0000: 5169: info : virNetlinkEventServiceStop:919 : stopping netlink event service

`cloudstack-agent logs from a test vm
2022-12-03 12:24:13,837 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2022-12-03 12:24:13,843 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.17.1.0
2022-12-03 12:24:13,845 INFO  [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2022-12-03 12:24:13,855 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2022-12-03 12:24:13,856 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2022-12-03 12:24:13,875 INFO  [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2022-12-03 12:24:13,875 INFO  [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2022-12-03 12:24:13,991 INFO  [cloud.agent.Agent] (main:null) (logid:) id is 
2022-12-03 12:24:13,995 WARN  [cloud.resource.ServerResourceBase] (main:null) (logid:) Nics are not specified in properties file/db, will try to autodiscover
2022-12-03 12:24:13,998 INFO  [cloud.resource.ServerResourceBase] (main:null) (logid:) Designating private to be nic ens33
2022-12-03 12:24:14,004 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2022-12-03 12:24:14,053 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find keystore file: cloud.jks
2022-12-03 12:24:14,054 INFO  [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2022-12-03 12:24:14,248 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) IO uring driver for Qemu: enabled
2022-12-03 12:24:14,334 WARN  [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) Duplicate StorageAdaptor type PowerFlex, not loading com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2022-12-03 12:24:14,335 INFO  [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2022-12-03 12:24:14,342 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to get private nic name
2022-12-03 12:24:14,342 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: Failed to get private nic name
`

`
Line by line install commands

apt-get update && apt-get upgrade -y
apt-get install sudo gnupg gnupg2 software-properties-common curl ebtables bridge-utils python3-dev openjdk11-jre -y
sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system  virtinst libvirt-daemon
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BDF0E176584DF93F
nano /etc/apt/sources.list.d/cloudstack.list
apt-get install  python3-dnspython python3-netaddr python3-requests-toolbelt

apt-get install cloudstack-agent

`
EXPECTED RESULTS

A clean installation of cloudstack-agent


##### ACTUAL RESULTS
<!-- What actually happened? -->
Issues with libvirt and non operational cloudstack-agent.
boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template!

weizhouapache commented 1 year ago

It seems the cloudstack-agent is not setup correctly, please check /etc/cloudstack/agent/agent.properties. When kvm host is added to a cloudstack zone, cloudstack-agent will be setup automatically.

Notespeak commented 1 year ago

Greetings, weizhouapache. The only thing that I changed in agent.properties are uuid keys and I also tried to specify a public interface right in the config https://pastebin.com/WmxvXjvJ. The agent is exiting before I am able to add it to the cloudstack zone. The management server is located right on this test vm, so don't mind me using localhost.)

weizhouapache commented 1 year ago

Greetings, weizhouapache. The only thing that I changed in agent.properties are uuid keys and I also tried to specify a public interface right in the config https://pastebin.com/WmxvXjvJ. The agent is exiting before I am able to add it to the cloudstack zone. The management server is located right on this test vm, so don't mind me using localhost.)

@Notespeak based on what you described, you need to set public.network.device=br0 private.network.device=cloudbrvm0 guest.network.device=cloudbrvm0

Anyway, I suggest you to add the host in cloudstack management server UI, instead of manually configuring agent.properties.

rohityadavcloud commented 1 year ago

Hi @Notespeak thanks for the issue, we don't have Debian 11 as a support KVM distro. You can refer to the compatibility matrix here: https://docs.cloudstack.apache.org/en/latest/releasenotes/compat.html

You may use one of the supported KVM distros until we can support Debian 11.