centiservice / mats3

Mats3: Message-based Asynchronous Transactional Staged Stateless Services
https://mats3.io/
Other
63 stars 7 forks source link

Message Size Sanity Check #14

Open stolsvik opened 6 years ago

stolsvik commented 6 years ago

Sometimes, you get crazy results out of a pretty small coding error - i.e. missing a WHERE-clause in SQL is interesting when doing a SELECT - and even more fun when doing a DELETE.

Some endpoints might normally produce 2-3 kB messages, but in degenerate cases could produce 500 MB messages - i.e. if it ususally adds a WHERE customerId=123, but then in the unhandled case includes instead ALL customers.

This can give a very unpleasant situation on the entire system, where the MQ ends up behaving erratically due to going out of memory.

Thus, introcude a message size sanity limit.

Something like

  1. Have a clear WARNing log line on some limit X.
  2. Refuse the message if the message exceeds 50X. The refusal would happen in the actual context.reply(), .request() and .next() methods. It would then eventually end up on the DLQ. Notice: The oversized message would not end on DLQ, but the message causing the oversized message to be created. Notice: This means that the size limit would be on the outgoing / producing side, not consuming.

X could be e.g. 50 kB., i.e. WARN on 50kB and refuse on 2.5MB.

Then we'd also have some way to say to Mats that "I want big messages here", e.g. context.sizeLimits(100 1024, 50 1024 * 1024).

The problem with the DLQ solution, is that it would be impossible to get those messages flowing again unless you did a redeploy of the service with the context.sizeLimits(..) added. BUT, we could allow for a "flag" in the actual JMS message props. This would at least make it somewhat easier to force through that processing: Pick of the message from DLQ, add that flag as a JMS message property, and then re-issue the message on the ordinary queue. This time, it would create its oversized message, but NOT be blocked by the size check.