search

RADAR-base / radar-output-restructure

Reads avro files in HDFS and outputs json or csv per topic per user in local file system
Apache License 2.0
1 stars 0 forks source link

Add support for wildcard based topic configurations #542

Closed yatharthranjan closed 1 year ago

yatharthranjan commented 1 year ago

From @blootsvoets

There would be some code modification needed to use wildcards, because the current config is also prescriptive: if a topic does not yet exist in any specification, adding the topic to the config will cause it to be created. If you modify the code, it would be possible, not with a full regex but a wildcard, e.g. empatica*, ?mpatica. You could add a property isWildcard that automatically gets set to yes if a or ? is detected. On line https://github.com/RADAR-base/RADAR-Schemas/blob/45d8c8ae4ddb997966b8ef9d98d98b8be[…]/src/main/java/org/radarbase/schema/registration/KafkaTopics.kt you would need to exclude any wildcards. And any references to toolConfig.topics[t] in KafkaTopics needs to be changed to findTopicConfig(t) where findTopicConfig does a wildcard match. Also, you would have to check the rest of the code if the literal name of the topic is used. This is the case at least in SchemaRegistry in the same directory.

blootsvoets commented 1 year ago

This issue description belongs in radar-schema’s, right? It would be fine to me to add wildcards to these configurations as well, but I don’t have an overview of the implications right now.

yatharthranjan commented 1 year ago

sorry yeah, i messed up the repo (too many open tabs, haha)