petersilva
commented
2 hours ago one way of doing it... could perhaps have an automatism...
- if three downloads fail in a row, then go into "standby" mode.
- once in standby mode, perhaps return to "running" once every housekeeping interval.
- It will try three downloads, and go back into "standby" mode.
- if the a download succeeds, go from "standby" to "catchup" mode.
- in catchup mode,
- read only from the retry queue...
- append stuff from the socket to the end of the retry queue.
At some point, we decide we have caught up, (e.g. retry queue is empty) return to running normally ("running")
So analyst intervention would be zero. They just start up the subscriber, it automatically goes into standby when it realizes it cannot write... and when it detects things are back, it catches up, and then goes back to normal.
For prolonged outages on the client (a subscriber's disk goes down, as an example) where a subscriber (or sender) knows that they won't be able to resume transfers any time soon.
The subscriber will try to download each file (3 times), and put them in a retry_queue. The effort of trying to download will be slow, is a wasted load, and might not keep up with the messages being posted.
Then you know you won't be able to complete transfers for some time, there should be a way to just put the messages directly in the retry_queue without trying the download.
When returned to service, we would need to always replay that retry_queue, and let new stuff queue up... or even add new stuff to the retry_queue instead of playing current stuff.... a "catch-up" mode.
Clarification:
The above describes what happens if you leave the subscriber running when you know downloads will fail. An alternate strategy would be to stop the subscribers, and let the queue build up on the brokers. All tuning/performance advice for all messages brokers recommends keeping queues short. Large Queues slow down the entire broker, compromising other flows. It is considered far better to persist the queues to local storage rather then have the backlog build up (into millions) on the broker.