Proposal: MessageHub repository and toolkit

[cancilla]

Proposal

I would like to propose that a new toolkit and repository be created to provide easy integration with the BlueMix MessageHub Service. The toolkit will provide "MessageHub" operators that wrap the Kafka operators from the com.ibm.streamsx.kafka toolkit. Having a set of "MessageHub" operators that are separate from the Kafka operators provides the following benefits:

• The "MessageHub" operators can contain default values for operator parameters and Kafka properties to make connectivity very simple
• The version of Kafka used by the MessageHub service is not always in sync with the latest stable release of Kafka. Thus, updating the Kafka operators to support the latest version of Kafka runs the risk of breaking MessageHub support. By maintaining a separate MessageHub toolkit, we can ensure that the operators are always in-sync with the version of Kafka being used by the MessageHub service.

Naming

I propose the following names:

• Repository: streamsx.messagehub
• Toolkit: com.ibm.streamsx.messagehub

Initial Contribution

The toolkit will initially contain 2 operators for consuming and producing messages:

• MessageHubConsumer - wraps the KafkaConsumer operator
• MessageHubProducer - wraps the KafkaProducer operator

[schubon]

+1

[chanskw]

I am unsure if we want a new repository for this. Or do we just want the streamsx.kafka repository to have a separate toolkit, which creates a separation between the base toolkit and message hub. I am concerned that having a separate repository will create confusion for the ned user.

We have the following repositories containing code related to Kafka: streamsx.messaging streamsx.kafka and now the streamsx.messagehub

Would keeping the toolkiti inside the streamsx.kafka repository provide enough separation/ What's the pros and cons related to having a new repository vs not having a repository?

I can see that we followed a similar pattern with the IoT repository which provides wrapper to the MQTT operator from the messaging toolkit.

[cancilla]

@chanskw A separate repo allows for maintaining documentation and roadmaps separate from the kafka operators. It also makes it easier for users to find releases for a specific toolkit on the Releases page. Having to hunt for the latest MessageHub toolkit release vs the latest Kafka toolkit release is going to be confusing for users.

Furthermore, users who want to use MessageHub but are not aware that it is backed by Kafka won't know to look for a MessageHub-specific toolkit in the streamsx.kafka repo. Users are more likely to search for a "messagehub" repo on the IBMStream landing page. Having streamsx.messagehub show up as a search result will make it obvious where this toolkit is.

[mikespicer]

+1

[ddebrunner]

+1

[chanskw]

Thanks @cancilla I thought about this a little bit more. I agree with the added visibility. I can also see how putting it in the same repo as kafka does not give enough separation.

In this new world where we favor microservices, I am wondering if this toolkit should also contain microservices where we are just exporting data stream. This is how the IoT repository is structured where it contains microsercies that people can launch to ingest data. Would having microservices for message hub also make sense?

+1 for creating the repository.

[ddebrunner]

+1 for microservices.

They are always there as an option then, no requirement to use them.

[cancilla]

+1 to microservices

[leongor]

+1 for microservices.

2017-05-30 20:59 GMT+03:00 James Cancilla notifications@github.com:

+1 to microservices

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/IBMStreams/administration/issues/116#issuecomment-304959027, or mute the thread https://github.com/notifications/unsubscribe-auth/AGvlA6gpWs-6_aFJKORwDdNB8AzuQvLIks5r_FkYgaJpZM4NqK1M .

-- Best regards, Leonid Gorelik.

[Yifat-Yulevich]

+1 for microservices.

[chanskw]

Done!