Make JMS a better fit for using Kafka

[hutchig]

I was wondering what the 'gaps' were between a future JMS 3.0 specification and its use to read and write from Kafka topic queues.

Of course JMS is an API so perhaps some aspects are just completely outside its scope. Nonetheless, I think it would be interesting to look at.

If JMS was advancing to 3.0 what might be missing that would enable it to fit 'well' over a subscope of Kafka use cases:

For example how would the JMS {que,topic} work for the Kafka {consumer-group} concept... could JMS shared subscriptions be evolved slightly to provide a good fit?

https://www.ibm.com/support/knowledgecenter/en/SSWMAJ_2.0.0/com.ibm.ism.doc/Developing/sharedsubscriptionsinjms.html "Shared subscriptions can be bound to a client ID, or can exist within a global namespace. If a client ID is specified for a connection that is used to create or join a shared subscription, then the subscription is bound to only that client ID. In this case, the client ID specifies the namespace for the subscription name. If no client ID is specified for a connection that is used to create or join a shared subscription, then the global namespace is used. By using a global namespace, it is possible to share a subscription between multiple connections. This configuration can be used, for example, to allow load balancing of a message-driven bean application that runs on an application server cluster."

If it could be made to have a good fit - could the environment that exits in a JEE context allow some real customer value add from the container that might be currently missing in the Java SE Kafka client?

[hutchig]

I do not seem to be able to create or attach any labels like {enhancement, jms3.0, kafka} :-) See also https://www.eclipse.org/lists/jms-dev/msg00018.html

[mattrpav]

@hutchig I'm with you.. I think a JMS v3.0 is long over due. There are a couple other initiatives looking to do something similar-- OpenMessaging, MicroProfile, etc. Kafka not having a standardized API is straight-up tech debt and the gaps are manageable.

In my strongly opinionated view -- the current JMS data models and interfaces should be punted into the sun:

1. My read of the EE Compatibility requirements seem to hamstring the ability to move JMS forward. We may need to have some new spec. Specifically, my read is that we would not be able to completely deprecate or replace APIs. If we can't get agreement that JMS suffers from "functional defect" (Which I believe JMS suffers from by my biased reading of Section 6), we may be better served going down the path of starting a new EE spec.

From Section 6:

The only exception to this is if there is a serious security or functional defect in the method or interface.

Reference: EE CompatibilityRequirements

I think we need to resolve #1 to set the direction-- JMS v3.0 or new spec-- before going further.

My quick list of JMS issues and wish-list:

1. Browse API should go away. It is severely limited and-- instead it should be a consume mode "no ack" and support limits and offsets
2. Selector / Query should be expanded to support message body and expressions (XPath, JSonPath, etc)
3. Ability to retrieve metadata-only view of messages
4. Handle large messages
5. Fix the message and API model to have clear lines as to what is a data structure and what is an operation (solve for the TTL, Priority, etc.. must be set on Producer object problem)
6. Add new ack modes-- individual message, no ack, etc. Consider dropping client acknowledge.
7. Fix queue/topic addressing model-- The current "2 domain" design seems arbitrary and tightly-coupled. Code should not have to be changed to change message flow from point-to-point to pub-sub and vice-versa. Solution: Define a single address uri.
8. Be comprehensive, not lightweight. All brokers define a INDV_ACK alternative and a way to do exclusive consumers, but they are all different and require code change to interoperate. The spec should be comprehensive to support all possible consuming modes vs "leave it up to the vendor".

[bshannon]

Different compatibility requirements could be defined for Jakarta EE. That's something we should discuss once planning for Jakarta EE 9 is started, which is still a ways away since we haven't really started Jakarta EE 8 yet.

Pruning out some rarely used functionality that otherwise complicates the API might be reasonable, but would you really want to break all the applications using current messaging systems? Making some pieces optional or dependent on the capabilities on the messaging system in use might be a better approach.

[mattrpav]

@bshannon You bring up some good points. I've been amp'd up to get this addressed for a while, so my injecting of some humor (punting the spec into the sun) shouldn't be taken literally.

The one issue (off the top of my head) that would break legacy would be the collapsing of the topic and queue address space (#7 above) . It seems like a big thing, but if we can get that kind of deprecation approved, I think it would be for the best for the community to keep the standard branded as JMS.

I think this is a we-really-should-do-it type deal, since currently application code is tightly coupled to the messaging pattern and cannot change by config only. Major brokers support alternatives, so the net effect is that clients are coded to the implementation, and the value of having the API is lost.

[bshannon]

I'm not an expert on the JMS spec, but I didn't think the spec defined the address space for topics and queues. I thought it was up to the messaging system implementation you were using. And of course the Java EE model was that these things were configured outside of the application, at deployment time.

Having some standardization around the names that can be used for topics and queues might be good, if it's done carefully so that it doesn't limit the kinds of existing messaging systems that can be used with JMS.

[keilw]

@hutchig I'm not sure, which role or team you're in here. I applied "enhancement". If "Kafka" is not too narrow as a keyword, why not, but please let's use GitHub more effectively and a 3.0 milestone for JMS 3.0 and potential features or enhancements targeting that release. There is a big mess like jms21-forreview-major or jms20-forreview-minor and similar. Maybe JIRA had some shortcomings or the way it was set up on java.net but it seems very odd to create all those labels instead of combining others properly. Something like "Glassfish 5.2" seems useful, because it's a cross-cut from other projects and repositories, but the actual version of the spec/API seems better via milestones.

[mattrpav]

@bshannon I've scanned the JMS specs, and it looks like you are right. While the notion that PTP and PubSub are separate domains, the naming needing to be distinct isn't specifically called out. The major implementations have treated them as such, and I must have extrapolated that to be from the spec.

[mattrpav]

Yah! We have a milestone =)

Seems like there is sufficient room for discussion on improvements and growing interest to make a JMS v3.0 move forward. How about we move the smaller feature changes into their own tickets for discussion and we can review PRs from there?

[keilw]

Yes, beside the one officially mentioned in https://projects.eclipse.org/projects/ee4j.jms I created 2.1 (if it ends up 2.0.3 or so we can still change, but it's meant to capture those originally planned for 2.1) and 3.0