search

Aiven-Open / opensearch-connector-for-apache-kafka

Aiven's OpenSearch® Connector for Apache Kafka®
Apache License 2.0
66 stars 36 forks source link

topics.regex fields not working #192

Closed fhgbaguidi closed 1 year ago

fhgbaguidi commented 1 year ago

Hello,

I'm working with the 2.0.4 version. 

"topics.regex": "my_super_topic_*",

When I use the file topics.regex with a topic pattern, topics are not found. Only the topics fields works well with a list of topics.

"topics": "my_super_topic_FR,my_super_topic_EN,my_super_topic_RU,my_super_topic_KZ,my_super_topic_ES"

I have more than 50 topics to sink and I can have more to come with some dynamic addition.

Plz can you check?

regards

fhgbaguidi commented 1 year ago

Maybe the same as https://github.com/aiven/opensearch-connector-for-apache-kafka/issues/140

gharris1727 commented 1 year ago

@fhgbaguidi I don't believe that your topics.regex value is appropriate for your use-case.

I believe that my_super_topic_* will match the following topic names:

my_super_topic
my_super_topic_
my_super_topic__
my_super_topic__________

and similar. I think you meant to use my_super_topic_.* or my_super_topic_[A-Z]* or similar. You can check the behavior of the regex engine using https://regex101.com/ set to the "Java 8" flavor.

Maybe the same as #140

Are you receiving a similar stacktrace, or using the regex router? None of the details of that issue seem relevant to this issue, other than mentioning the topics.regex configuration. Can you clarify in what way it is similar?

I hope this helps.

fhgbaguidi commented 1 year ago

@gharris1727 Thanks for your help. My regex was not good at all. I'm not having any stacktrace. Definitely not similar. I was not sure at all because of the name of the issue.

sorry for inconvenience.