IBMStreams / streamsx.kafka

Repository for integration with Apache Kafka
https://ibmstreams.github.io/streamsx.kafka/
Apache License 2.0
13 stars 9 forks source link

KafkaConsumer: support submission time parameter for multiple topics #228

Closed ghost closed 3 years ago

ghost commented 3 years ago

The KafkaConsumer's topic operator parameter is of type rstring with cardinality -1, which allows to specify multiple topics to subscribe, like this:

stream <MessageType.StringMessage> Consumed = KafkaConsumer() {
    param
        topic: "t1", "t2";
        ...
}

There seems to be no way, to use a submission time parameter for multiple values. The SPL function getSubmissionTimeValue returns always a single rstring value, which would be interpreted as a single topic, i.e. when an applicatioin is submitted with -P topics_param=t1,t2, the operator tries to subscribe to the single topic "t1,t2" instead of two topics.

The SPL function getSubmissionTimeListValue returns a list of rstrings, which is incompatible with the generated operator model: CDISP0048E ERROR: The topic operator parameter requires values of the rstring type, but values of the list<rstring> type are specified.

A workaround must be implemented here:

Try to split given topic(s) at a comma character, subscribe to the tokens. Pro: No new operator parameter Con: topics with a comma in it cannot be subscribed. Currently not an issue as a comma is not (yet) a valid character within a topic. Chances that a comma will be valid in future are very low.

ghost commented 3 years ago

fixed in v3.2.2