search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.12k stars 1.11k forks source link

The cloudstack-agent can not work 4.18.1.0 #8223

Open clustspace opened 1 year ago

clustspace commented 1 year ago

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

  1. Cloudstack management (fresh install)
  1. Cloudstack agent (fresh install)

SUMMARY

After installing cloudstack-agent it won't start.

Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Stopped CloudStack Agent.
░░ Subject: A stop job for unit cloudstack-agent.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit cloudstack-agent.service has finished.
░░
░░ The job identifier is 28655 and the job result is done.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.244s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Started CloudStack Agent.
░░ Subject: A start job for unit cloudstack-agent.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit cloudstack-agent.service has finished successfully.
░░
░░ The job identifier is 28655.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell).
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN Please initialize the log4j system properly.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Agent started
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Implementation Version is 4.18.1.0
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: Unable to find the guid
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service has successfully entered the 'dead' state.
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.239s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.

Error log from cloudstack-management server

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:823)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 00:34:31,482 INFO  [c.c.a.ApiServer] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template!

rajujith commented 1 year ago

@clustspace could you verify that the KVM host has the required dependencies installed like libvirt ,KVM etc? If thats good, is this KVM added to Cloudstack? try adding it.

clustspace commented 1 year ago 
kvm --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.15)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

libvirt-daemon-system is already the newest version (8.0.0-1ubuntu7.7).

virsh # version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0
weizhouapache commented 1 year ago

@clustspace this could be caused by some configurations (e.g. network) on the kvm host. can you upload

clustspace commented 1 year ago

@weizhouapache there is no setup.log on kvm host

Log from management-server.log in first post

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: **Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:** 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
weizhouapache commented 1 year ago

@weizhouapache there is no setup.log on kvm host

Log from management-server.log in first post

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: **Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:** 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].

@clustspace can you enable DEBUG log level on management server and retry to add the host, it will give more details ?

sed -i "s/INFO/DEBUG/g" /etc/cloudstack/management/log4j-cloud.xml

You need to restart management server to make it effective.

clustspace commented 1 year ago

@weizhouapache same error

2023-11-13 12:19:34,915 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
2023-11-13 12:19:34,915 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy199.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-11-13 12:19:34,918 DEBUG [c.c.r.ResourceManagerImpl] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy199.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 12:19:34,920 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2023-11-13 12:19:34,921 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:823)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy199.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 12:19:34,924 INFO  [c.c.a.ApiServer] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
2023-11-13 12:19:34,925 DEBUG [c.c.a.ApiServlet] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) ===END===  185.122.140.1 -- POST  command=addHost&response=json
2023-11-13 12:19:35,132 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Resetting hosts suitable for reconnect
2023-11-13 12:19:35,135 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Completed resetting hosts suitable for reconnect
2023-11-13 12:19:35,135 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Acquiring hosts for clusters already owned by this management server
2023-11-13 12:19:35,137 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Completed acquiring hosts for clusters already owned by this management server
2023-11-13 12:19:35,137 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Acquiring hosts for clusters not owned by any management server
2023-11-13 12:19:35,138 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Completed acquiring hosts for clusters not owned by any management server
2023-11-13 12:19:37,354 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-658f94c5) (logid:9f33176f) HA health check task is running...
2023-11-13 12:19:40,490 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-eaafffc6) (logid:f40a1484) Enabled non-edge zones available for scan: 1
2023-11-13 12:19:40,494 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-eaafffc6) (logid:f40a1484) Zone 1 has no host available which is enabled and in Up state
2023-11-13 12:19:40,494 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-eaafffc6) (logid:f40a1484) Zone [1] is not ready to launch secondary storage VM.
2023-11-13 12:19:40,590 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4e5a015e) (logid:2ce1a8da) Skip capacity scan as there is no Primary Storage in 'Up' state
2023-11-13 12:19:41,356 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-7597a4b3) (logid:a8227727) HA health check task is running...
2023-11-13 12:19:45,359 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-6fedacea) (logid:2d831233) HA health check task is running...
2023-11-13 12:19:49,361 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-9e6e5d21) (logid:8bb8ea01) HA health check task is running...

First of all, we can't start cloudstack-agent on host server

tampler commented 1 year ago

Having exactly the same error on the same setup. ACS 4.18.1.0 on Ubuntu 22.04.3.

For me this happens for a non-root user (admin). Here's my setup for a visudo list:

Defaults:cloud !requiretty
Defaults:cloudstack !requiretty
cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.0.20                                                           
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)                                                  
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)                                                                
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)

PS: Seems like an agent miscofing on my side. That's what I see in the agent log:

Caused by: java.io.IOException: SSL Handshake failed while connecting to host: acs port: 8250
        at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
tampler commented 1 year ago

Looks like #6716 is back

weizhouapache commented 1 year ago

Looks like #6716 is back

@tampler It looks they are different. Although the error logs show that there are exception "Failed to setup keystore", the root cause seems not to be related to keystore, but others like network configurations, privileges, etc.

It would be good to grant the non-root user stronger privileges, if you do not use root when add a host.

tampler commented 1 year ago

There are few caveats in installing KVM on Ubuntu 22.04. Those are not yet covered in the project documentation. One is an ability for an agent host to register itself within the mgmt host, which is done somewhat counter intuitively. To resolve your issue, do this:

  1. Update libvirt configs, following the doc
  2. On the agent host: sudo nano /etc/cloudstack/agent/agent.properties and set proper values for UUID, host and bridges
  3. Copy cloudstack generated ssh key to the authorized_keys on a mgmt host: 
    cat id_rsa.pub.cloud >> /root/.ssh/authorized_keys
  4. Restart libvirt and agent and make sure they started properly: 
    sudo systemctl restart libvirtd.service
sudo systemctl restart cloudstack-agent.service
  5. If everything is ok, switch to the mgmt service UI config. When you get to adding the host, specify for root user and click the SSH Public Key Access to login with the autogenerated key.

NOTE: You'll probably face another errors, which you would report here or open another issue :smirk::smirk::smirk:

tampler commented 1 year ago

That's a config issue for Ubuntu 22.04. Related to #8218

weizhouapache commented 1 year ago

There are few caveats in installing KVM on Ubuntu 22.04. Those are not yet covered in the project documentation. One is an ability for an agent host to register itself within the mgmt host, which is done somewhat counter intuitively. To resolve your issue, do this:

  1. Update libvirt configs, following the doc
  2. On the agent host: sudo nano /etc/cloudstack/agent/agent.properties and set proper values for UUID, host and bridges
  3. Copy cloudstack generated ssh key to the authorized_keys on a mgmt host:
cat id_rsa.pub.cloud >> /root/.ssh/authorized_keys
  1. Restart libvirt and agent and make sure they started properly:
sudo systemctl restart libvirtd.service
sudo systemctl restart cloudstack-agent.service
  1. If everything is ok, switch to the mgmt service UI config. When you get to adding the host, specify for root user and click the SSH Public Key Access to login with the autogenerated key.

NOTE: You'll probably face another errors, which you would report here or open another issue 😏😏😏

yes @tampler If you do not want to use root password, ssh key is a good alternative. it just works.

If you use a non-root user, /etc/sudoers should be configured. I suspect the following configuration is correct for non-root users.

cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent
Defaults:cloudstack !requiretty

https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#install-and-configure-the-agent

tampler commented 1 year ago

Another issue with ssh is that I can't config an alternative port for it. We have a policy NOT to use default port 22 for SSH. @weizhouapache should I open a new issue for this?

weizhouapache commented 1 year ago

Another issue with ssh is that I can't config an alternative port for it. We have a policy NOT to use default port 22 for SSH. @weizhouapache should I open a new issue for this?

yes, please @tampler it looks like a valid improvement .

rohityadavcloud commented 9 months ago

@clustspace were you able to follow some suggestions and resolve your issue?

clustspace commented 9 months ago

@clustspace were you able to follow some suggestions and resolve your issue?

No, without results....we install 4.17 version

weizhouapache commented 9 months ago

I wonder if the issue is same as #8310 and related to #8633 @clustspace can you check if /usr/bin/hostname exists ?

rohityadavcloud commented 7 months ago

@clustspace can you try again with 4.18.2.0 or 4.19.0.1 release ? Here's some notes for install on Ubuntu https://rohityadav.cloud/blog/cloudstack-kvm/