unexpected backout behaviour with DMLC and Websphere MQ

[apinske]

Affects: 5.3.15

Websphere MQ has a strange way of backout handling (i.e. moving poison messages). It does not move poison messages to the backout queue immediately on the "final" rollback, but only increments the internal BackoutCount (to be one more than it should be able to be) and leaves the message on the initial queue. The actual move is performed in a subsequent (unrelated!) receive-call. This is documented here.

If an application receives messages synchronously, by calling one of the following methods, the IBM MQ classes for JMS requeue a poison message within the unit of work that was active when the application tried to get the message:

This means that if the application is using either a transacted JMS context or session, then the moving of the message to the backout queue is not committed until the transaction is committed.

This behaviour does not effect e.g. JavaEE-MDB-Listeners.

If an application is receiving messages asynchronously via a MessageListener, the IBM MQ classes for JMS requeue poison messages without affecting message delivery. The requeue process takes place outside of any unit of work associated with actual message delivery to the application.

DMLC uses a polling approach and thus is susceptible to the described behaviour. It is not really possible to observe this behaviour, as any attempt to "see" the messages moves it to the right queue. Only way is to "not observe" the message already being in the backout queue.

The practical implications are two-fold:

• A single poison message is not immediately visible on the appropriate backout queue for handling. Is is only moved when a next receive-pass commits (takes in the worst case the amount of receive-timout + handling of an unrelated msg).
• In a scenario where temporarily all/many messages are poison the move-to-backout-operations may also rollback. (We found out about this by observing MQ-"uncommited"-metrics showing rising and falling backout queue counts.)

As a browse also seems to (undocumented) trigger the move, which in itself is weird for a read-only-operation, we found a workaround: After the "final" rollback try to browse to the message which will move it. This does not seem feasible as a generic solution, as the app needs to know the backout threshold.

Spring probably cannot fix this. The best way would be to fix this in MQ. As this seems unlikely, maybe spring could at least document/warn/... the presence of this behaviour.

Reproducing projects:

• Websphere-MQ https://github.com/apinske/playground-mq/tree/backout-wmq
• Active-MQ https://github.com/apinske/playground-mq/tree/backout-amq

[bclozel]

We don't typically document broker-specific behavior in our reference documentation (we already have quite a lot to deal with already), especially if it is quite niche. Thanks for raising this, I believe that this issue documents what the community will need. I'm closing this now.