[Bug] topicsPattern cannot subscribe to non-persistent topics

[a11enhuang]

Search before asking

• [X] I searched in the issues and found nothing similar.

Version

ApachePulsar 2.11.0

Minimal reproduce step

pulsarClient.newConsumer()
        .subscriptionName("my-sub")
        .topicsPattern("public/default/.*")
        .subscriptionTopicsMode(RegexSubscriptionMode.NonPersistentOnly)
        .subscribe();

What did you expect to see?

Ability to subscribe to non-persistent topics

What did you see instead?

Not subscribed to non-persistent topics. After DEBUG, it was found that the Topic list could not be obtained from Broker:

 // The topicsList is empty
List<String> topicsList = getTopicsResult.getTopics();
if (!getTopicsResult.isFiltered()) {
   topicsList = TopicList.filterTopics(getTopicsResult.getTopics(), conf.getTopicsPattern());
}
conf.getTopicNames().addAll(topicsList);
ConsumerBase<T> consumer = new PatternMultiTopicsConsumerImpl<>(conf.getTopicsPattern(),
        getTopicsResult.getTopicsHash(),
        PulsarClientImpl.this,
        conf,
        externalExecutorProvider,
        consumerSubscribedFuture,
        schema, subscriptionMode, interceptors);

consumers.add(consumer);

It appears that information on non-persistent topics is not synchronized across the cluster. My cluster has a total of three nodes. The non-persistent topic list was not obtained during the first two startups, but it obtained da during the third startup.

Anything else?

No response

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[syhily]

This bug only exists in Pulsar 2.11.0 version. The Admin API for querying non-persistent API also meet this BUG.

[syhily]

https://github.com/apache/pulsar/issues/19316

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[syhily]

This bug hasn't been fixed in 3.0.0. Should we glance at this bug and find the root cause? @Technoboy-

[Technoboy-]

This bug hasn't been fixed in 3.0.0. Should we glance at this bug and find the root cause? @Technoboy-

We only store partition info for non-persistent topics. So if there are no producers/consumers service for the topics, we don't list them or can't get them.

The non-persistent topic list was not obtained during the first two startups, but it obtained da during the third startup.

The default polling time is 60s, you should ensure that all the non-persistent topics are loaded and please check for more than 60s. You can update the default polling time by patternAutoDiscoveryPeriod

@syhily @a11enhuang

[syhily]

@Technoboy- I don't think your explanation make sense to me. From my side, the only issue that I can tell is that I can't query the created non-persistent topics immediately by using any API provided by Pulsar. Such as the topic endpoint in admin API. The LookupService in client API. This behavior is wired. We don't meet such issue before the 2.12.0 release.

It's ok for Pulsar to continue improve the existed functions by some internal optimizations. But this shouldn't break the existing behaviors. A user with all functions work well shouldn't care about any new changes if he don't need any new features.

That means, as a user, I don't care about patternAutoDiscoveryPeriod or something else. The existing behaviors have been changed and break the functions. THAT MUST BE A BUG.

[syhily]

Here is a test for reproducing this BUG.

https://github.com/apache/flink-connector-pulsar/blob/3b6c3aff8aeca3cc17673bbc84b90b70c1c680a9/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/source/enumerator/subscriber/PulsarSubscriberTest.java#L203

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.