centiservice / mats3

Mats3: Message-based Asynchronous Transactional Staged Stateless Services
https://mats3.io/
Other
63 stars 7 forks source link

"MatsInitiator implements StartStoppable"? Functionality for "back pressure" - thoughts. #8

Open stolsvik opened 6 years ago

stolsvik commented 6 years ago

MatsInitiator could implement StartStoppable. But it would be a bit concept mismatchy. Point is that it would be nice to be able to stop initiations if the downstream had problems.

There should be some "isStopped()" or "isActive()" type of method that could be invoked before firing off, potentially altering any GUI if it returned non-OK.

What would happen if you do initiate when it is non-OK? Throwing exceptions? Also, there is obviously a race condition here.

stolsvik commented 5 years ago

(Funny, that - exactly one year later, I am again thinking about this, about to create an issue, but the "The following issues might be related." caught me.)

I think I am leaning towards NOT implement StartStoppable, but that the current "implements Closeable" is most correct. (It should not be AutoCloseable, since that would imply "use it once" e.g. per initiation, while this is supposed to be a long-lived resource, as in "through the application lifetime").

The rationale is that an Initiator is not per target endpoint. That is, one Initiator can be used to target any resource. Therefore, stopping an Initiator would stop all usages of it - both initiations to the congested queues/endpointIds, but also to any other queue the same Initiator was used to.

Was this to give meaning, an Initiator would have to be specific to a target queue. It is currently not.

I am currently thinking that such backpressure should be implemented as a tool outside Mats proper. I envision a tool whereby the MQ/Broker monitored all queues, and periodically (e.g. every 30 seconds) sent updates (on a topic) about the queue lengths for all queues having either queue length above some threshold Y, and/or first-message-of-queue-is-older-than-X-seconds. (If there was no such queue, then it would not need to send such update for 10 mintes - given that it the last message it had sent out had "reset" the situation to "good" for all queues).

Any "user facing Initiators" would then check its memory cached view about the current MQ world, and if the queue length or age-of-first-message was above some threshold, it would deny sending the message.

It would be important when using this to not let the user get a bad experience of this - not e.g. drop all his 4289 filled values on the floor without him having an option of storing the data. Possibly both check status before the user starts to fill a form - and let him retry if things fails - and store the values in localStorage or similar, so that when he came back some hours later, the form would be prefilled.