Proposal: Separate RabbitMQ operators from Messaging toolkit

[schubon]

Proposal

As it has been discussed for Kafka operators in issues https://github.com/IBMStreams/streamsx.messaging/issues/246 and https://github.com/IBMStreams/administration/issues/111, and as already proposed by @mikespicer: I think we should separate the RabbitMQ operators from the Messaging toolkit now, too.

How

• The current RabbitMQ code stays in streamsx.messaging for backward compatibility
• The current RabbitMQ code is used as base for a new toolkit streamsx.rabbitmq
• The new toolkit gets a slightly different namespace, for example 'com.ibm.streamsx.rabbitmq'
• The new toolkit gets a new set of message IDs for the already defined messages
• The new toolkit is allowed to break the old operator's interface, so we may improve the code more easily
• The old RabbitMQ operators are not deprecated

Challenges

• We will have a redundant code base for toolkit, sample, and test code for a while (it's not exactly the same code as namespaces / message IDs differ)
• Accordingly, we will need to decide where defects will be corrected

[Alex-Cook4]

+1 This has the same benefit of giving greater visibility to our support for RabbitMQ.

RabbitMQ as a product is much more stable than Kafka (not as many major, breaking releases), so it is less likely to cause the versioning issues we have seen.

However, the RabbitMQ operators themselves are relatively "new", so may be getting more feature additions given that they haven't matured.

If we are not going to deprecate the "old" RabbitMQ operators, then I feel we have to fix defects in both branches.

Our driver for customers to move to the new toolkit would be new features/version support, rather than fixes.

I'm not a big fan of redundant code (I know no one is :-) )...should we have a plan for eventually removing the operators from the messaging toolkit?

[chanskw]

"The old RabbitMQ operators are not deprecated" - I think we should deprecate the old operators. Otherwise, it becomes confusing as to which operators are supposed to use. And as Alex say, if both sets of operators are in development, it requires double maintenance.

"The new toolkit gets a new set of message IDs for the already defined messages" - I also think the new operators should have the same same of message IDs if the errors are the same. Message IDs are tend to tie to tech notes where customers can find the message IDs, figure out what the problems are and find resolutions. Changing the message IDs require extra work as we change message IDs for all the messages in the new operator. It also causes double maintenance effort as we have to tie technotes to the message IDs. This also means extra work in translation when we NL-enabled the operators again. If the message IDs remain the same, no extra NL work is required. If the problems are the same, the same message IDs are used. If new problems are found there we will have new message IDs.

We should view that the new operators are continuable of the old operators, not a set of completely new operators.

[schubon]

As @ddebrunner said in the Kafka operator discussion, not rushing the 'new' operators into the product gives us more opportunity for change and restart. As I understand deprecation, it is unusual to deprecate functionality without pointing the user to a viable alternative in the product. So, I thought as long as the new operators aren't in the product we cannot deprecate the old ones. Regarding the Message IDs @ZollnaPa warned that we will need new ranges for new operators. Currently, these IDs are to be chained to an operator.

[chanskw]

I agree with not rushing a release to give us a chance to evolve. But the new operators are the alternative to the deprecated operators. In the main development stream, the old operators are deprecated and the viable alternatives are is in a new development stream in another toolkit.

Github projects are independent of Streams product releases. The decisions on what to do with the open-source toolkits should not be affected with what we do with product decisions. If the product is not involved in this decision, then we would deprecate the old operators and continue development in the new toolkit. We would not make a copy of the operators, spin off another repository and have duplicated operators in the IBMStreams Github organization.

[mikespicer]

+1 assuming you agree on an appropriate deprecation path so we do not have dual maintenance.

[chanskw]

@schulz2 any more thoughts about our discussions on deprecation paths?

[schubon]

If it is acceptable / been done before, to deprecate the RabbitMQ operators of the Messaging toolkit - which is part of the Streams product - while hinting to the then available open source streamsx.rabbitmq toolkit, I'm fine with that. There are currently no open issues for RabbitMQ operators, so we may create a new toolkit using the old code now, and correct errors in the new toolkit only, thus avoiding the rdundant code issue.

Summary

• The current RabbitMQ code stays in streamsx.messaging for backward compatibility
• The current RabbitMQ code is used as base for a new toolkit streamsx.rabbitmq
• The new toolkit gets a slightly different namespace, for example 'com.ibm.streamsx.rabbitmq'
• The new toolkit gets a new set of message IDs for the already defined messages
• The new toolkit is allowed to break the old operator's interface, so we may improve the code more easily
• The old RabbitMQ operators are deprecated
• Development and error corrections happen in the new toolkit only

Product team needs to decide if the new toolkit is added to the product.

@chanskw @Alex-Cook4 @mikespicer @ddebrunner @cancilla Is this it?

[brandtol]

@schulz2 last proposal looks fine to me.

[Alex-Cook4]

+1 to last proposal

[ZollnaPa]

+1 to last proposal

[chanskw]

+1 to the latest proposal

[chanskw]

created streamsx.rabbitmq