jing-c-tyagi
commented
5 months ago I tried to get into a shell into the pod while it has a short running status, and got the following ver-nextrelease-pulsar-broker.txt
Closed jing-c-tyagi closed 5 months ago
jing-c-tyagi
commented
5 months ago I tried to get into a shell into the pod while it has a short running status, and got the following ver-nextrelease-pulsar-broker.txt
lhotari
commented
5 months ago I believe that this is fixed by #489 which is going to be released with 3.4.1 version of the chart. The release is currently in voting: https://lists.apache.org/thread/4stgyrtsnj9jhrvw0b1t8bqr4knj8bfx
lhotari
commented
5 months ago @jing-c-tyagi Pulsar Helm Chart 3.4.1 has been released. This problem should be fixed with #489 which is part of 3.4.1 release. I'll close the issue. Please reopen if the problem persists.
lhotari
commented
5 months ago Actually, now I noticed that the error message contains "Error e reading ledger - ledger=74597 - operation=Failed to read entry - entry=0" . That's a sign of a corrupted or missing ledger. You will need to repair that somehow. One solution is to accept the data loss and allow Pulsar to continue while ignoring the problem by setting autoSkipNonRecoverableData=true. That's a dangerous setting and you will face data loss.
However, in this case, since the error is about persistent://public/functions/assignments topic, that is used for Pulsar Functions coordination and the data loss isn't a problem. (You could disable Pulsar Functions completely unless you are using the feature.)
lhotari
commented
5 months ago Closing this since this isn't a pulsar-helm-chart issue.
we are running pulsar chart version pulsar-3.3.1 /app version 3.0.3 in aws eks. $ helm list -n pulsar NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION pulsar pulsar 8 2024-05-24 13:10:17.2221715 -0400 EDT deployed pulsar-3.3.1 3.0.3
due to some application side error, we are trying to restart pulsar pods. we ran kubectl scale statefulsets for pulsar-broker but it failed to start with CrashLoopBackOff status and describe the pod shows Liveness probe failed: HTTP probe failed with statuscode: 404 and Readiness probe failed: HTTP probe failed with statuscode: 404
To Reproduce kubectl scale statefulsets pulsar-broker --replicas=0 -n pulsar then kubectl scale statefulsets pulsar-broker --replicas=3 -n pulsar
But the pulsar-broker pods failed to start, $ kgpo -n pulsar
Describe the pulsar-broker-2 pod has the following events: Events: Type Reason Age From Message
Normal Scheduled 29m default-scheduler Successfully assigned pulsar/pulsar-broker-2 to ip-10-249-18-54.ec2.internal Normal Pulled 29m kubelet Container image "apachepulsar/pulsar-all:3.0.3" already present on machine Normal Created 29m kubelet Created container wait-zookeeper-ready Normal Started 29m kubelet Started container wait-zookeeper-ready Normal Pulled 29m kubelet Container image "apachepulsar/pulsar-all:3.0.3" already present on machine Normal Created 29m kubelet Created container wait-bookkeeper-ready Normal Started 29m kubelet Started container wait-bookkeeper-ready Normal Pulled 29m (x2 over 29m) kubelet Container image "apachepulsar/pulsar-all:3.0.3" already present on machine Normal Created 29m (x2 over 29m) kubelet Created container pulsar-broker Normal Started 29m (x2 over 29m) kubelet Started container pulsar-broker Warning Unhealthy 28m (x2 over 28m) kubelet Liveness probe failed: Get "http://100.64.15.103:8080/status.html": dial tcp 100.64.15.103:8080: connect: connection refused Warning Unhealthy 28m (x2 over 28m) kubelet Readiness probe failed: Get "http://100.64.15.103:8080/status.html": dial tcp 100.64.15.103:8080: connect: connection refused Warning Unhealthy 28m (x4 over 28m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 404 Warning Unhealthy 24m (x18 over 28m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 404 Warning BackOff 4m39s (x78 over 27m) kubelet Back-off restarting failed container pulsar-broker in pod pulsar-broker-2_pulsar(a41a0f02-17bd-48ca-9e8f-a93afc673fc4)
$ k logs -n pulsar pulsar-broker-2 --tail=100 -p
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-nt-2.12.1.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-cl-2.12.1.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.10nal.jar:4.1.100.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.100.Final.jar:4.0.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.10nal.jar:4.1.100.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[itty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.l] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.10nal.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.100.Final.jar:4.0.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.100.Final:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.1inal] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.100.Final.jar:400.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.l] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: Topic cron encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_message={\"errorMsg\":\"Error e reading ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"pulsar-broker-2.pulsar-broker.pulsvc.cluster.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$ServiitNotReadyException: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments erressage={\"errorMsg\":\"Error while reading ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"ar-broker-2.pulsar-broker.pulsar.svc.cluster.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\tat org.apache.pulsar.broker.service.BrokerSce.lambda$getTopic$28(BrokerService.java:1065)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat jaase/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postlete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apachesar.client.impl.PulsarClientImpl.lambda$createSingleTopicReaderAsync$14(PulsarClientImpl.java:687)\n\tat java.base/java.util.concurrent.CompletableFuture.uceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat jbase/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptily(CompletableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ConsumerImpl.lambda$connectionOpened$17(ConsumerImpl.java:893)\n\tat java.base/java.utincurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(ComableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CetableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:793)\n\tat org.apapulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:192)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChaHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.chanAbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(BytessageDecoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.channel.AbstractChannellerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannellerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.flFlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChlHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.cha.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelReafaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.nethannel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelReafaultChannelPipeline.java:919)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)\t io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)\n\ta.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecuap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:84"} at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960) ~[org.glassfish.jersey.core-jersey-client-2.3r:?] at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] ... 54 more Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: Topic creation entered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_message={\"errorMsg\":\"Error while reg ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"pulsar-broker-2.pulsar-broker.pulsar.svc.ter.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNodyException: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_messa\"errorMsg\":\"Error while reading ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"pulsar-br-2.pulsar-broker.pulsar.svc.cluster.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\tat org.apache.pulsar.broker.service.BrokerService.la$getTopic$28(BrokerService.java:1065)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postCompletepletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.pulsar.nt.impl.PulsarClientImpl.lambda$createSingleTopicReaderAsync$14(PulsarClientImpl.java:687)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptlly(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.basea.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CetableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ConsumerImpl.lambda$connectionOpened$17(ConsumerImpl.java:893)\n\tat java.base/java.util.concur.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Completablere.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.Completabture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:793)\n\tat org.apache.pulsommon.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:192)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandontext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessacoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.channel.AbstractChannelHandlerCxt.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerCxt.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.flush.FlussolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHanContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaulnnelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.channbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaulnnelPipeline.java:919)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)\n\tat ioty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)\n\tat io.nettil.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.j74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\n"}
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254) ~[org.apache.pulsar-pulsar-client-adoriginal-3.0.3.jar:3.0.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?] at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296) ~[org.apache.pulsar-pulsar-nt-admin-original-3.0.3.jar:3.0.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?] at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-21.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.1jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.4.1.100.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Fi at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.4.1.100.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-y-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Fina 100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-05-24T16:58:28,764+0000 [main] WARN org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1 java.lang.Exception: Stacktrace for immediate shutdown at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:55) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3] at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:50) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3] at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:82) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3] at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:359) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3]