search

datastax / pulsar-helm-chart

Apache Pulsar Helm chart
Apache License 2.0
47 stars 38 forks source link

Use Luna Streaming 2.8.0_1.1.26, by default #159

Closed michaeljmarshall closed 2 years ago

michaeljmarshall commented 2 years ago

There have been several important patches since 2.8.0_1.1.15. Most notably, a fix to the health check logic. Based on recent test failures, I have reason to believe this should improve the tests. Since these two versions are compatible, there is no major risk for this version bump.

Here is an excerpt from the logs of a failed helm chart test.

09:23:26.579 [pulsar-web-40-8] INFO  org.apache.pulsar.broker.admin.impl.BrokersBase - Running healthCheck with topic=persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck
09:23:26.589 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck] [null] Creating producer on cnx [id: 0x4ba794bb, L:/10.244.1.11:54582 - R:10.244.1.11/10.244.1.11:6650]
09:23:26.596 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.11:54582][persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck] Creating producer. producerId=8
09:23:26.620 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Subscribing to topic on cnx [id: 0x4ba794bb, L:/10.244.1.11:54582 - R:10.244.1.11/10.244.1.11:6650], consumerId 8
09:23:26.624 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.11:54582] Subscribing on topic persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck / reader-1481ebc5f8
09:23:26.642 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.11:54582] Created new producer: Producer{topic=SystemTopic{topic=persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck}, client=/10.244.1.11:54582, producerName=pulsar-1-10, producerId=8}
09:23:26.643 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck] [pulsar-1-10] Created producer on cnx [id: 0x4ba794bb, L:/10.244.1.11:54582 - R:10.244.1.11/10.244.1.11:6650]
09:23:26.670 [Thread-22] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Creating non-durable subscription at msg id 9223372036854775807:9223372036854775807:-1:-1
09:23:26.672 [Thread-22] INFO  org.apache.bookkeeper.mledger.impl.NonDurableCursorImpl - [pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck] Created non-durable cursor read-position=0:8 mark-delete-position=0:7
09:23:26.673 [Thread-22] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck] Opened new cursor: NonDurableCursorImpl{ledger=pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck, ackPos=0:7, readPos=0:8}
09:23:26.682 [Thread-22] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck-reader-1481ebc5f8] Rewind from 0:8 to 0:8
09:23:26.683 [Thread-22] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck] There are no replicated subscriptions on the topic
09:23:26.683 [Thread-22] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Created new subscription for 8
09:23:26.683 [Thread-22] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.11:54582] Created subscription on topic persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck / reader-1481ebc5f8
09:23:26.685 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Subscribed to topic on 10.244.1.11/10.244.1.11:6650 -- consumer: 8
09:23:26.689 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.admin.impl.BrokersBase - Running healthCheck with topic=persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck
09:23:26.690 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Unsubscribing
09:23:26.692 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.Consumer - Disconnecting consumer: Consumer{subscription=PersistentSubscription{topic=persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck, name=reader-1481ebc5f8}, consumerId=8, consumerName=f69a3, address=/10.244.1.11:54582}
09:23:26.694 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ClientCnx - [10.244.1.11/10.244.1.11:6650] Broker notification of Closed consumer: 8
09:23:26.695 [pulsar-io-4-2] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck] [reader-1481ebc5f8] Closed connection [id: 0x4ba794bb, L:/10.244.1.11:54582 - R:10.244.1.11/10.244.1.11:6650] -- Will try again in 0.1 s
09:23:26.698 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.AbstractDispatcherSingleActiveConsumer - Removing consumer Consumer{subscription=PersistentSubscription{topic=persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck, name=reader-1481ebc5f8}, consumerId=8, consumerName=f69a3, address=/10.244.1.11:54582}
09:23:26.700 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Successfully closed subscription [NonDurableCursorImpl{ledger=pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck, ackPos=0:7, readPos=0:9}]
09:23:26.701 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Successfully closed dispatcher for reader
09:23:26.702 [broker-topic-workers-OrderedScheduler-2-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck-reader-1481ebc5f8] Rewind from 0:9 to 0:8
09:23:26.703 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Successfully closed subscription [NonDurableCursorImpl{ledger=pulsar/pulsar/10.244.1.11:8080/persistent/healthcheck, ackPos=0:7, readPos=0:8}]
09:23:26.703 [pulsar-web-40-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Successfully disconnected and closed subscription
09:23:26.707 [pulsar-web-40-1] ERROR org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://pulsar/pulsar/10.244.1.11:8080/healthcheck][reader-1481ebc5f8] Error deleting subscription
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: reader-1481ebc5f8
    at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1108) ~[?:?]
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) ~[?:?]
    at org.apache.pulsar.broker.service.persistent.PersistentSubscription.delete(PersistentSubscription.java:915) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at org.apache.pulsar.broker.service.persistent.PersistentSubscription.deleteForcefully(PersistentSubscription.java:878) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at org.apache.pulsar.broker.admin.impl.BrokersBase.lambda$healthcheck$4(BrokersBase.java:323) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[com.datastax.oss-pulsar-common-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at org.apache.pulsar.broker.admin.impl.BrokersBase.lambda$healthcheck$5(BrokersBase.java:321) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
    at org.apache.pulsar.broker.admin.impl.BrokersBase.healthcheck(BrokersBase.java:320) ~[com.datastax.oss-pulsar-broker-2.8.0.1.1.15.jar:2.8.0.1.1.15]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
michaeljmarshall commented 2 years ago

Just noticed that the necessary commit (https://github.com/datastax/pulsar/commit/148a0c607812bb5dc18d66622436e2ec76f71369) isn't in the latest version of luna streaming. Once 2.8.0_1.1.26 is released, I expect tests will pass.

michaeljmarshall commented 2 years ago

Closing this one, superseded by https://github.com/datastax/pulsar-helm-chart/pull/161

michaeljmarshall commented 2 years ago

I misunderstood #161. We still need this PR.

michaeljmarshall commented 2 years ago

Closing and re-opening to get latest changes from master. (Not sure if I'll need to rebase.)