Closed truong-hua closed 2 years ago
Could you provide some broker logs when this error happens?
Could you provide some broker logs when this error happens?
Dear @Technoboy- ,
The broker logs has no more information than the exception
2022-06-02T09:36:44,454+0000 [pulsar-2-2] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform setSubscriptionDispatchRate on topic persistent://public/ticdc/inbox_message
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:207) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0]
at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:100) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0]
at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:48) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0]
at org.apache.pulsar.client.util.RetryUtil.lambda$null$1(RetryUtil.java:57) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-06-02T09:36:44,457+0000 [pulsar-2-2] INFO org.eclipse.jetty.server.RequestLog - 10.0.0.134 - - [02/Jun/2022:09:36:29 +0000] "POST /admin/v2/persistent/public/ticdc/inbox_message/subscriptionDispatchRate?authoritative=false HTTP/1.1" 500 1277 "-" "curl/7.68.0" 14901
Could you provide some broker logs when this error happens?
Dear @Technoboy- ,
The broker logs has no more information than the exception
2022-06-02T09:36:44,454+0000 [pulsar-2-2] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform setSubscriptionDispatchRate on topic persistent://public/ticdc/inbox_message org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init. at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:207) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:100) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:48) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0] at org.apache.pulsar.client.util.RetryUtil.lambda$null$1(RetryUtil.java:57) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:829) [?:?] 2022-06-02T09:36:44,457+0000 [pulsar-2-2] INFO org.eclipse.jetty.server.RequestLog - 10.0.0.134 - - [02/Jun/2022:09:36:29 +0000] "POST /admin/v2/persistent/public/ticdc/inbox_message/subscriptionDispatchRate?authoritative=false HTTP/1.1" 500 1277 "-" "curl/7.68.0" 14901
Can you search for any other error logs in this broker?
it happened to me and my team as well, we are currently testing topic policies(so topic policies are enabled) , this error pops up randomly for us (probably related to tests), the way around we found is to unload namespace, we also did not find a way to reproduce
It also happens to us when we try to determine a topic compaction threshold.
We use Pulsar 2.8.3.
Error when issuing command to get compaction threshold:
root@rev-pulsar-toolset-0:/pulsar/bin# ./pulsar-admin topics get-compaction-threshold persistent://dbus/test/dummy-objects
HTTP 500 Server Error
Reason: HTTP 500 Server Error
Error seen on broker:
10:02:50.919 [pulsar-io-4-4] INFO org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://dbus/dgt-time-series-point/status}][rev-pulsar-4-4116] Closed producer on cnx /10.72.168.226:33920. producerId=8040
10:02:51.016 [pulsar-2-1] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform getCompactionThreshold on topic persistent://dbus/test/dummy-objects
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:185) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3]
at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:83) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3]
at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:49) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3]
at org.apache.pulsar.client.util.RetryUtil.lambda$executeWithRetry$1(RetryUtil.java:58) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Do you have schemaValidationEnforced
turned on on a namespace level? I'm seeing the following in logs just before receiving the Topic policies cache have not init.
exception.
2022-07-01T11:25:15,816+0000 [pulsar-io-4-5] WARN org.apache.pulsar.client.impl.PulsarClientImpl - [persistent://showcase/user/__change_events] Failed to get create topic reader
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: {"errorMsg":"Schema not found and schema auto updating is disabled.","reqId":4307185312891752491, "remote":"dev01-neuron-pulsar-broker-0.dev01-neuron-pulsar-broker.dev01-neuron-pulsar.svc.cluster.local/10.130.2.163:6650", "local":"/10.130.2.163:56456"}
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:777) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) [?:?]
at org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:715) [org.apache.pulsar-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:177) [org.apache.pulsar-pulsar-common-2.9.2.jar:2.9.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: {"errorMsg":"Schema not found and schema auto updating is disabled.","reqId":4307185312891752491, "remote":"dev01-neuron-pulsar-broker-0.dev01-neuron-pulsar-broker.dev01-neuron-pulsar.svc.cluster.local/10.130.2.163:6650", "local":"/10.130.2.163:56456"}
at org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:1170) ~[org.apache.pulsar-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:716) ~[org.apache.pulsar-pulsar-client-original-2.9.2.jar:2.9.2]
... 20 more
And to me it suggest that the reader that the broker creates on this internal __change_events
topic can't work in a namespace with schemaValidationEnforced
enabled since it doesn't specify any schema.
Edit: Actually the issue is with is_allow_auto_update_schema
. See #16331
It also happens to us when we try to determine a topic compaction threshold.
We use Pulsar 2.8.3.
Error when issuing command to get compaction threshold:
root@rev-pulsar-toolset-0:/pulsar/bin# ./pulsar-admin topics get-compaction-threshold persistent://dbus/test/dummy-objects HTTP 500 Server Error Reason: HTTP 500 Server Error
Error seen on broker:
10:02:50.919 [pulsar-io-4-4] INFO org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://dbus/dgt-time-series-point/status}][rev-pulsar-4-4116] Closed producer on cnx /10.72.168.226:33920. producerId=8040 10:02:51.016 [pulsar-2-1] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform getCompactionThreshold on topic persistent://dbus/test/dummy-objects org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init. at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:185) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3] at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:83) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3] at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:49) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3] at org.apache.pulsar.client.util.RetryUtil.lambda$executeWithRetry$1(RetryUtil.java:58) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:829) [?:?]
It also happens to us just the same error as describe above. Also Pulsar 2.8.3.
Pulsar 2.9.2, same issue. #16331 won't help us because we have is_allow_auto_update_schema
enabled
I solved my problem anyway.
I tracked the code and found that "Topic policies cache have not init" most likely because something wrong with the __change_events
topic. The broker failed to consume the system topic. In this scene of mine, the system topic can not be loaded.
I checked the system topic and found that one partition (__change_events-partition-1
) is terminated
and the ledger had already beed deleted. I tried deleting the __change_events-partition-1
but failed. Finally I delete the znode /managed-ledgers/tenant/namespace/persistent/__change_events-partition-1
and restart the broker which is the owner of __change_events-partition-1
, the broker auto created the partition (allowAutoTopicCreation=true
) and then restart all brokers. Everything back to normal.
@dragonls thank for that, is it safe to manually update the znode. Do we have any document about metadata stored in the zk?
@dragonls thank for that, is it safe to manually update the znode. Do we have any document about metadata stored in the zk?
I'm not sure whether it is safe, but for my case, the topic has beed terminated and no ledgers left, which make the topic can not be loaded. I also reproduce my case in test env and do not see anything else wrong.
You'd better check what is wrong with the __change_events
and then decide what to do.
I do not see any document about metadata stored in the zk.
I solved my problem anyway.
I tracked the code and found that "Topic policies cache have not init" most likely because something wrong with the
__change_events
topic. The broker failed to consume the system topic. In this scene of mine, the system topic can not be loaded.I checked the system topic and found that one partition (
__change_events-partition-1
) isterminated
and the ledger had already beed deleted. I tried deleting the__change_events-partition-1
but failed. Finally I delete the znode/managed-ledgers/tenant/namespace/persistent/__change_events-partition-1
and restart the broker which is the owner of__change_events-partition-1
, the broker auto created the partition (allowAutoTopicCreation=true
) and then restart all brokers. Everything back to normal.
What do you mean the partition is terminated
? Do you mean that the partition state is terminated
which can't produce any messages but can consume ?
Yes, someone may have teminated the topic via pulsar admin.
I've the same issue in a production environment. It happened just after Pulsar upgrade from 2.6.3 to 2.9.2. Strange enough I didn't see it happen on staging environment with the same configuration. Real difference between environments is traffic volume: streams much more used in production, some streams on staging aren't really used for long time...
Same the same issue today. Out of nowhere broker unable to create topic due to "Topic policy cache have not init".
I have enabled systemTopic and topic level policy, been using the same config for almost a month and suddenly this pops up in production.
Delete topic, unload topic, restart broker, delete __change_events
topic, all did not help.
Had to urgently pull down production pulsar cluster and disable systemTopic and topic level policy. And it works after.
Pulsar 2.10.1
@Raven888888 @diegosalvi @truong-hua We have found the root cause and pushed out the fix https://github.com/apache/pulsar/pull/17562 Thanks for reporting the issue.
Awesome to hear that @Technoboy- appreciate it Hope to use back topic level policy in the next release.
@Raven888888 @diegosalvi @truong-hua We have found the root cause and pushed out the fix #17562 Thanks for reporting the issue.
@Technoboy-
A similar issue is happening in 2.8.4. I see that fix was backported to 2.8 branch to be scheduled for 2.8.5 but such version was never released. Is branch 2.8.x unmaintained? Could version 2.8.5 be released? We are in an urgent situation here as it is too risky to update to 2.9.5 in our production systems.
We see following errors after we have deleted the namespace with system topics. From then on our system does not responde and we are not able to reconnect to Pulsar broker (standalone mode).
dev-standalone-1 | 13:43:47.559 [pulsar-web-63-2] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [22/Sep/2023:13:43:47 +0200] "GET /admin/v2/persistent/public/functions/coordinate/stats?getPreciseBacklog=false&subscriptionBacklogSize=false HTTP/1.1" 200 2061 "-" "Pulsar-Java-v2.8.4" 1
dev-standalone-1 | 13:43:49.418 [pulsar-timer-88-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://dbus/test/__change_events] [reader-13b0b91b2e] Reconnecting after connection was closed
dev-standalone-1 | 13:43:49.419 [pulsar-io-29-5] WARN org.apache.pulsar.broker.web.PulsarWebResource - Namespace dbus/test is deleted
dev-standalone-1 | 13:43:49.419 [pulsar-io-29-4] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://dbus/test/__change_events] failed to send lookup request : Namespace is deleted
dev-standalone-1 | 13:43:49.419 [pulsar-io-29-4] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://dbus/test/__change_events] [reader-13b0b91b2e] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Namespace is deleted
dev-standalone-1 | 13:43:49.419 [pulsar-io-29-4] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://dbus/test/__change_events] [reader-13b0b91b2e] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Namespace is deleted -- Will try again in 55.718 s
If we try to restart Pulsar it is always shutdown with following error:
dev-standalone-1 | 13:46:01.085 [AsyncHttpClient-85-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/functions/assignments] Failed to perform http put request: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | 13:46:01.086 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
dev-standalone-1 | org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:247) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:217) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:276) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503) ~[?:?]
dev-standalone-1 | ... 39 more
dev-standalone-1 | Caused by: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:278) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | ... 36 more
dev-standalone-1 | Caused by: org.asynchttpclient.handler.MaxRedirectException: Maximum redirect reached: 5
dev-standalone-1 | 13:46:01.099 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:580) ~[org.apache.pulsar-pulsar-functions-worker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1597) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:763) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:297) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:247) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:217) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:276) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:276) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:278) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: org.asynchttpclient.handler.MaxRedirectException: Maximum redirect reached: 5
dev-standalone-1 | 13:46:01.105 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
dev-standalone-1 | org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:791) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:297) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | Caused by: java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:580) ~[org.apache.pulsar-pulsar-functions-worker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1597) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:763) ~[org.apache.pulsar-pulsar-broker-2.8.4.jar:2.8.4]
dev-standalone-1 | ... 2 more
dev-standalone-1 | Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:247) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
dev-standalone-1 | at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:217) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:276) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:276) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Maximum redirect reached: 5
dev-standalone-1 | at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:278) ~[org.apache.pulsar-pulsar-client-admin-original-2.8.4.jar:2.8.4]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
dev-standalone-1 | at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
dev-standalone-1 | at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:473) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.readFailed(HttpHandler.java:161) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:154) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
dev-standalone-1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]
dev-standalone-1 | Caused by: org.asynchttpclient.handler.MaxRedirectException: Maximum redirect reached: 5
dev-standalone-1 | 13:46:04.149 [Thread-0] INFO org.apache.pulsar.broker.service.GracefulExecutorServicesTerminationHandler - Starting termination handler for 4 executors.
dev-standalone-1 | 13:46:04.152 [Thread-0] INFO org.apache.pulsar.broker.service.GracefulExecutorServicesTerminationHandler - Shutdown completed.
Pulsar will not start anymore.
Describe the bug I think my bug is related to this issue https://github.com/apache/pulsar/issues/10231 but actually it's not because our Pulsar has disabled the
brokerDeleteInactiveTopicsEnabled
in broker level. And the issue "Topic policies cache have not init" happened on both namespaces with "inactiveTopicPolicies" enabled and disabled.We are all using partition topics.
Our full adjusted configs:
To Reproduce Steps to reproduce the behavior: I have no idea of how to reproduce this issue but it happens on our production environment and not in staging. May be it's related to production data
Expected behavior We can set topic policies properly without exception that:
Desktop (please complete the following information):
Additional context Currently is there any Rest API for manually init the Topic Policies Cache or any way to check it via command line and manually init it?