search

streamnative / pulsar-archived

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org
Apache License 2.0
72 stars 25 forks source link

ISSUE-7432: Namespace isolation policy :: No broker was available to own persistent #1177

Open sijie opened 4 years ago

sijie commented 4 years ago

Original Issue: apache/pulsar#7432

Appreciate your help to resolve this issues:

Thanks in Advance :-)

Created an single plusar cluster setup with four broker services running on four different servers, my intend is to point the applicaiton to the broker running on same server, following are the steps has been followed

Note : Customize ports are used for zookeeper,bookie and broker instead of default ports

1) Create zookeeper setup on 3 servers 2) Initilize the cluster setup with zookeeper,configruation (used zookeeper ip and port), webservice and broker url pointed to one of the server where broker service is running 3) create the tenants and namespaces, 4) created an namespaces isolation policy, assigned the primary and secondary broker url

try to consume topic on the namespace using the pulsar-client tool for which ns-islation policy is created and attached, following error is thrown in console,

./pulsar initialize-cluster-metadata --cluster cluster1 --zookeeper server1.com:8041 --configuration-store server1.com:8041 --web-service-url http://server1.com:8008 --broker-service-url pulsar://server1.com:8050

./pulsar-daemon start broker ./pulsar-admin tenants create testtenant ./pulsar-admin namespaces create testtenant/ns1

./pulsar-admin ns-isolation-policy set --auto-failover-policy-params min_limit=1,usage_threshold=100,isPrimary=true --auto-failover-policy-type min_available --namespaces "testtenant/ns1" --primary "server1.com:8008,server2.com:8008" --secondary "server3.com:8008,server4.com:8008" cluster1 ns-cluster1

Error

07:04:19.842 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xbd927667, L:/xxxx:59344 - R:server/xxxxx:8050]] Connected to server 07:04:19.853 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ClientCnx - [id: 0xbd927667, L:/xxxxx:59344 - R:server1/xxxxx:8050] Close connection because received internal-server error No broker was available to own persistent://testtenant/ns1/11 07:04:19.854 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://testtenant/ns1/11] failed to send lookup request : org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://testtenant/ns1/11 07:04:19.854 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://testtenant/ns1/11] lookup failed : org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://testtenant/ns1/11 java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://testtenant/ns1/11 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_252] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_252] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) ~[?:1.8.0_252] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) ~[?:1.8.0_252] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [?:1.8.0_252] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) [?:1.8.0_252] at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:474) [org.apache.pulsar-pulsar-client-original-2.6.0.jar:2.6.0] at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:131) [org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) [io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [io.netty-netty-transport-native-epoll-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [io.netty-netty-transport-native-epoll-4.1.48.Final.jar:4.1.48.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-native-epoll-4.1.48.Final.jar:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252] Caused by: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://testtenant/ns1/11 at org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:991) ~[org.apache.pulsar-pulsar-client-original-2.6.0.jar:2.6.0] at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:475) ~[org.apache.pulsar-pulsar-client-original-2.6.0.jar:2.6.0] ... 20 more 07:04:19.856 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [testtenant/ns1/11] [test] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://testtenant/ns1/11 07:04:19.856 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [viswa/ns1/11] [test] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: No broker was available to own persistent://testtenant/ns1/11 -- Will try again in 6.234 s 07:04:19.856 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xbd927667, L:/xxxxx:59344 ! R:server1/xxxxxxx:8050] Disconnected

ns-isolation-policy:

./pulsar-admin ns-isolation-policy brokers cluster1 { "brokerName" : "server4.com:8008", "isPrimary" : false } { "brokerName" : "server3.com:8008", "isPrimary" : false } { "brokerName" : "server2.com:8008", "isPrimary" : false, "namespaceRegex" : [ "testtenant/ns1" ] } { "brokerName" : "server1.com:8008", "isPrimary" : false, "namespaceRegex" : [ "testtenant/ns1" ] }

./pulsar-admin ns-isolation-policy broker cluster1 --broker server1.com:8008 { "brokerName" : "server1.com:8008", "policyName" : "ns-cluster1", "isPrimary" : true, "namespaceRegex" : [ "testtenant/ns1" ] } Pls advice how "isPrimary" parameter value is different in Brokers list and Individual Broker command output.

Thanks

dineshkothuru2020 commented 4 years ago

Appreciate Your help here to share any steps of configuration or documents on this. Eagerly waiting for responses to fix this issue. Thanks..

Jennifer88huang-zz commented 4 years ago

@jiazhai Could you please help have a look at this issue?