search

artemiscloud / activemq-artemis-operator

Apache License 2.0
65 stars 62 forks source link

scale down test can fail intermittently if dns entry for ss-1 is missing #663

Closed gtully closed 1 year ago

gtully commented 1 year ago

Verification of scale down is part of the problem, the exit code is 0 in error. see: https://github.com/artemiscloud/activemq-artemis-broker-kubernetes-image/issues/52 however the DNS entries should be fresh, there is potentially some issue with the probes that result in not found for the ss-0 entry in some case.

`[drain.sh] scale down target is: scale-ss-0.scale-hdls-svc.default.svc.cluster.local Starting artemis-service artemis-service is now running (143) 2023-07-24 20:03:06,226 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server version 2.29.0 2023-07-24 20:03:06,275 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/opt/scale/data/journal,bindingsDirectory=/opt/scale/data/bindings,largeMessagesDirectory=/opt/scale/data/large-messages,pagingDirectory=/opt/scale/data/paging) 2023-07-24 20:03:06,307 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal 2023-07-24 20:03:06,354 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 8396996608 2023-07-24 20:03:06,570 INFO [org.jgroups.JChannel] local_addr: 47e14544-3de6-491d-8864-14329e53f1cb, name: scale-ss-1-44267 2023-07-24 20:03:06,595 INFO [org.jgroups.protocols.FD_SOCK2] server listening on *.7900 2023-07-24 20:03:06,658 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 2023-07-24 20:03:06,659 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 2023-07-24 20:03:06,659 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 2023-07-24 20:03:06,660 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 2023-07-24 20:03:06,660 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE 2023-07-24 20:03:06,660 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 2023-07-24 20:03:06,710 INFO [org.apache.activemq.artemis.core.server] AMQ221034: Waiting indefinitely to obtain live lock 2023-07-24 20:03:06,710 INFO [org.apache.activemq.artemis.core.server] AMQ221035: Live Server Obtained live lock 2023-07-24 20:03:06,962 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST] 2023-07-24 20:03:06,963 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ 2023-07-24 20:03:06,967 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST] 2023-07-24 20:03:06,967 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue 2023-07-24 20:03:07,110 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at scale-ss-1:61616 for protocols [CORE] 2023-07-24 20:03:07,114 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live 2023-07-24 20:03:07,114 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.29.0 [amq-broker, nodeID=159909b9-2a5d-11ee-afdf-4e1a7de39b51] 2023-07-24 20:03:07,119 INFO [org.apache.activemq.artemis] AMQ241003: Starting embedded web server 2023-07-24 20:03:07,310 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin 2023-07-24 20:03:07,345 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin 2023-07-24 20:03:07,388 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?] at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?] at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?] at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?] at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?] at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406) ~[netty-transport-classes-epoll-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.29.0.jar:?] 2023-07-24 20:03:07,388 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?] at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?] at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?] at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]

2023-07-24 20:03:07,388 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?] at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?] at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?] at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?] at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?] at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] ... 2023-07-24 20:03:07,402 WARN [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge@1184b917 [name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2d316791 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1184b917 [name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2d316791 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=scale-ss-0-scale-hdls-svc-default-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1259037900[nodeUUID=159909b9-2a5d-11ee-afdf-4e1a7de39b51, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=scale-ss-1, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=scale-ss-0-scale-hdls-svc-default-svc-cluster-local], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying 2023-07-24 20:03:07,558 INFO [io.hawt.HawtioContextListener] Initialising hawtio services 2023-07-24 20:03:07,560 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties 2023-07-24 20:03:07,561 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.15.0 2023-07-24 20:03:07,565 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "admin" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal" 2023-07-24 20:03:07,569 INFO [io.hawt.web.auth.LoginRedirectFilter] Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout 2023-07-24 20:03:07,575 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled 2023-07-24 20:03:07,578 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/home/jboss/amq-broker/etc/jolokia-access.xml] 2023-07-24 20:03:07,633 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://scale-ss-1:8161 2023-07-24 20:03:07,633 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://scale-ss-1:8161/console/jolokia 2023-07-24 20:03:07,633 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://scale-ss-1:8161/console 2023-07-24 20:03:08,405 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?] at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?] at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?] .. 2023-07-24 20:03:08,405 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known .. 2023-07-24 20:03:09,405 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?] .. 2023-07-24 20:03:09,406 WARN [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge@1184b917 [name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2d316791 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1184b917 [name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@2d316791 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=scale-ss-0-scale-hdls-svc-default-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1259037900[nodeUUID=159909b9-2a5d-11ee-afdf-4e1a7de39b51, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=scale-ss-1, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=scale-ss-0-scale-hdls-svc-default-svc-cluster-local], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying 2023-07-24 20:03:09,408 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known .. 2023-07-24 20:03:09,409 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]

2023-07-24 20:03:10,414 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known 2023-07-24 20:03:10,415 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 02023-07-24 20:03:11,417 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known 2023-07-24 20:03:11,420 INFO [org.apache.activemq.artemis.core.server] AMQ221029: stopped bridge $.artemis.internal.sf.my-cluster.116ee7c7-2a5d-11ee-b31d-a6d658714b63 2023-07-24 20:03:11,423 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]

20:03:12,523 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator is closed (maybe it was garbage collected) java.lang.IllegalStateException: Server locator is closed (maybe it was garbage collected) at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1869) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:669) ~ 100 195 0 195 0 0 82 0 --:--:-- 0:00:02 --:--:-- 82 [drain.sh] curl return code 200 [drain.sh] scaledown is successful 2023-07-24 20:03:13,426 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection java.net.UnknownHostException: scale-ss-0.scale-hdls-svc.default.svc.cluster.local: Name or service not known at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?] at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?] at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?] at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?] at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?] at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?] at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?] at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[netty-resolver-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406) ~[netty-transport-classes-epoll-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.93.Final.jar:4.1.93.Final] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.29.0.jar:?] 2023-07-24 20:03:13,427 WARN [org.apache.activemq.artemis.core.client] AMQ212025: did not connect the cluster connection to other nodes java.lang.Exception: null at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.(ServerLocatorImpl.java:326) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.(ServerLocatorImpl.java:378) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.api.core.client.ActiveMQClient.createServerLocatorWithHA(ActiveMQClient.java:372) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.cluster.ha.ScaleDownPolicy.getScaleDownConnector(ScaleDownPolicy.java:104) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.connectToScaleDownTarget(LiveOnlyActivation.java:142) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.freezeConnections(LiveOnlyActivation.java:116) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.freezeConnections(ActiveMQServerImpl.java:1550) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1324) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1243) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.fail(ActiveMQServerImpl.java:1222) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.management.impl.ActiveMQServerControlImpl.scaleDown(ActiveMQServerControlImpl.java:4172) ~[artemis-server-2.29.0.jar:2.29.0] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262) ~[?:?] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) ~[?:?] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) ~[?:?] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) ~[?:?] at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) ~[?:?] at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) ~[?:?] at javax.management.StandardMBean.invoke(StandardMBean.java:405) ~[?:?] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:814) ~[?:?] at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:802) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerBuilder$MBeanInvocationHandler.invoke(ArtemisMBeanServerBuilder.java:96) ~[artemis-server-2.29.0.jar:2.29.0] at jdk.proxy2.$Proxy31.invoke(Unknown Source) ~[?:?] at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:429) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.HttpRequestHandler.handleGetRequest(HttpRequestHandler.java:86) ~[jolokia-core-1.7.0.jar:?] .. 2.15.0.jar:2.15.0] at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?] at javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?] at io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.web.auth.AuthenticationFilter.lambda$doFilter$0(AuthenticationFilter.java:80) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.system.Authenticator.authenticate(Authenticator.java:158) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:79) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.auth.SessionExpiryFilter.process(SessionExpiryFilter.java:107) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.web.auth.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.apache.activemq.artemis.component.JolokiaFilter.doFilter(JolokiaFilter.java:50) ~[artemis-web-2.29.0.jar:2.29.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) ~[jetty-security-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.15.jar:10.0.15] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.15.jar:10.0.15] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) [jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) [jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) [jetty-util-10.0.15.jar:10.0.15] at java.lang.Thread.run(Thread.java:833) [?:?] 2023-07-24 20:03:13,430 WARN [org.apache.activemq.artemis.core.server] AMQ222181: Unable to scaleDown messages org.apache.activemq.artemis.api.core.ActiveMQNotConnectedException: AMQ219009: Failed to connect to any static connectors at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1790) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:561) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) ~[artemis-core-client-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.LiveNodeLocator.connectToCluster(LiveNodeLocator.java:85) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.connectToScaleDownTarget(LiveOnlyActivation.java:148) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.freezeConnections(LiveOnlyActivation.java:116) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.freezeConnections(ActiveMQServerImpl.java:1550) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1324) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1243) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.fail(ActiveMQServerImpl.java:1222) ~[artemis-server-2.29.0.jar:2.29.0] at org.apache.activemq.artemis.core.management.impl.ActiveMQServerControlImpl.scaleDown(ActiveMQServerControlImpl.java:4172) ~[artemis-server-2.29.0.jar:2.29.0] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:262) ~[?:?] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) ~[?:?] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) ~[?:?] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) ~[?:?] at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) ~[?:?] at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) ~[?:?] at javax.management.StandardMBean.invoke(StandardMBean.java:405) ~[?:?] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:814) ~[?:?] at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:802) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.apache.activemq.artemis.core.server.management.ArtemisMBeanServerBuilder$MBeanInvocationHandler.invoke(ArtemisMBeanServerBuilder.java:96) ~[artemis-server-2.29.0.jar:2.29.0] at jdk.proxy2.$Proxy31.invoke(Unknown Source) ~[?:?] at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:429) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.HttpRequestHandler.handleGetRequest(HttpRequestHandler.java:86) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.AgentServlet$4.handleRequest(AgentServlet.java:503) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:383) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.AgentServlet.handle(AgentServlet.java:354) ~[jolokia-core-1.7.0.jar:?] at org.jolokia.http.AgentServlet.doGet(AgentServlet.java:310) ~[jolokia-core-1.7.0.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:503) ~[jetty-servlet-api-4.0.6.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.FlightRecordingDownloadFacade.doFilter(FlightRecordingDownloadFacade.java:68) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.auth.LoginRedirectFilter.doFilter(LoginRedirectFilter.java:63) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.auth.AuthenticationFilter.lambda$executeAs$1(AuthenticationFilter.java:105) ~[hawtio-system-2.15.0.jar:2.15.0] at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?] at javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?] at io.hawt.web.auth.AuthenticationFilter.executeAs(AuthenticationFilter.java:104) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.web.auth.AuthenticationFilter.lambda$doFilter$0(AuthenticationFilter.java:80) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.system.Authenticator.authenticate(Authenticator.java:158) ~[hawtio-system-2.15.0.jar:2.15.0] at io.hawt.web.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:79) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at io.hawt.web.filters.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:43) ~[hawtio-system-2.15.0.jar:2.15.0] .. at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) ~[jetty-security-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) ~[jetty-server-10.0.15.jar:10.0.15] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.15.jar:10.0.15] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.15.jar:10.0.15] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194) ~[jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) [jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) [jetty-util-10.0.15.jar:10.0.15] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) [jetty-util-10.0.15.jar:10.0.15] at java.lang.Thread.run(Thread.java:833) [?:?] 2023-07-24 20:03:13,580 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.29.0 [159909b9-2a5d-11ee-afdf-4e1a7de39b51] stopped, uptime 7.321 seconds `

gtully commented 1 year ago

it looks like the readiness probe fails for ss-0 (which it should not) and as a result the dns entry is whacked as expected. and the scale down cannot deal with that.

gtully commented 1 year ago

$ kubectl exec scale-ss-0 -- /bin/bash -c "cat /tmp/r*" Count: 30, sleep: 1 Wed Jul 26 10:52:23 UTC 2023 Connect: 0 ========================= OUTPUT ========================== scaleDown value tcp://scale-ss-0.scale-hdls-svc.default.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300 scaleDown port 61616 Transport is listening on port 61616 ========================= ERROR ========================== ==========================================================

gtully commented 1 year ago

that count: 30 should be count: 1 such that the probe does not loop and the probe config can control the loop, otherwise the probe times out!