Use a dropping buffer to avoid error-chan causing fink-nottle to block

[joelittlejohn]

Hi @moea. We're using fink-nottle with SQS.

We noticed that our system would occasionally lock up and thread dumps showed that the block was in trying to put onto SQS. After pouring over our code, fink-nottle and some of the supporting libraries, a colleague of mine noticed that the fink-nottle can cease doing put operations and block if nothing consumes from the error-chan.

We appear to have fixed out problems by making sure that there is always a consumer for each error-chan.

Obviously, it's a good idea to read and handle/log errors :) However, this seems like an odd default and one that causes unexpected results. Do you think it's worth using a dropping or sliding buffer for the default error-chan created here:

(defn batching-channel*
  [issue-fn
   & [{:keys [period-ms threshold in-chan error-chan timeout-fn close?]
       :or {period-ms 200 threshold 10 timeout-fn a/timeout close? true}}]]
  (let [in-chan    (or in-chan (a/chan))
        error-chan (or error-chan (a/chan))]

?

This basically creates a fire-hose for errors. If you don't consume them they'll be dropped, but a single error will not cause the system to lock.

In our case, I think this was a really simple communication error (a timeout, connection reset, or similar caused by normal network weather) that would only occur once a week or less. We would find every so often that our service had locked up, but it was hard to pin down.

[moea]

Hi there. I'm sorry this caused problems for you - it's intentional/documented, but the project was undocumented for some time, and I agree that it's pretty aggressive. More than happy to merge a PR which uses a dropping buffer (very slight preference for that over sliding buffer - open to suggestions), else I could get to this a little later in the week.

[joelittlejohn]

@moea thanks for the super quick response. Thanks for the link to the docs :blush:

I'll submit a PR for this shortly!

Cheers