Using Kafka operators with the message hub in IBM Streams 4.2

[ddebrunner]

Has any figured out how to connect the kafka operators to the Message Hub Bluemix service using a 4.2 application configuration?

Looking at this: https://www.ibm.com/blogs/bluemix/2015/10/streaming-analytics-message-hub-2/

it's not obvious how the user name/password would be specified using the app config.

[Alex-Cook4]

That is an excellent idea @ddebrunner. I have used the app config with Kafka and it works great. @alexpogue is that something you would be interested in adding to your article? I am open to helping.

[ddebrunner]

@Alex-Cook4 Is it even possible though?

[Alex-Cook4]

Not solely using app config. The issue is that username and password are specified as part of the jaas.conf file, so users would still need to provide the specific jaas.conf. But topic, and any kafka properties can be specified via app config.

[ddebrunner]

So the requirement would be to support using MessageHub storing all connection information (specific to the instance) in an application configuration.

[ddebrunner]

As an initial update I did manage to get a connection to message hub working using the app config but keeping the user/password in the jaas.conf file. I set all the fixed attributes in the SPL application and only had `bootstrap.servers' set in the app config.

SPL Code

        () as KafkaProducer_2 = KafkaProducer(B)
        {
        param
        topic: "sms";

        appConfigName: "debrunne-message-hub";
        appConfigPropertyName: "bootstrap.servers";

        kafkaProperty:
            "client.id=Dan001",
            "acks=0",
            "security.protocol=SASL_SSL",
            "ssl.protocol=TLSv1.2",
            "ssl.enabled.protocols=TLSv1.2",
            "ssl.truststore.type=JKS",
            "ssl.endpoint.identification.algorithm=HTTPS";

        jaasFile: getThisToolkitDir() + "/etc/jaas.conf";        
        }

The value of bootstrap.servers was taken from the kafka_brokers_sasl property in my Message Hub credentials. Unlike the article above I don't have a simple kafka_brokers property:

[Alex-Cook4]

Looks like there is a new sasl.jaas.config Kafka Property that would allow us to set the jaas configuration via properties. This also fixes the problem of operators fused into a single JVM having to share a JAAS configuration (at least that is how I read this property). Does this sufficiently solve #279 #280 and #281 once we upgrade?

[schubon]

Customers are using the (from-scratch) new toolkit https://github.com/IBMStreams/streamsx.kafka that was started from scratch. If you still encounter the issue there, open an issue over there, please.