Support for triggering custom hook on failure?

[dmarkhas]

Sometimes when a request fails and is retried, we need to trigger some custom code - for example, update a metric or let some other component in the system know about this failure.

Currently there is no support for running arbitrary hooks before a failed request is retried, can that be added? It can receive the same argument as failed?.

[BrunoBonacci]

Hi @dmarkhas,

I long debated with myself the need for custom handlers. Twice, I made a plausible implementation, twice I ended up removing it. The reason, I removed it is that once you add custom handlers safely becomes just a glorified try/catch block.

safely design principle is to provide declarative error handling. If you add custom handlers you soon want to be able to control the retries and then the behaviour it cannot be determined declaratively. There is more, what if the handler itself throws an exception?

Certainly, there are cases in which safely doesn't fit the bill, that was never my intention to have a general-purpose error handler. I still find myself using try/catch in some cases, and sometimes I combine try/catch with safely. I have an inner safely block to manage the retries wrapped into a try/catch to manage the error when the maximum retries have been reached.

If all you want to do is to track the number of failures, like increment counters and publish metrics, you should know that this is automatically supported by safely (via the :track-as option). Just provide a name for the metric to be tracked and safely will track several aspects automatically.

It will track the error rate, the success rate, the number of attempts, the different errors by type, execution time of each attempt and the overall execution

For more information about the tracking see: https://github.com/BrunoBonacci/safely/blob/master/doc/tracking.md

The metrics are tracked with a library which I wrote, called: https://github.com/samsara/trackit which wraps the Yammer's Metrics library. Out of the box, you can publish the metrics to a number of external systems.

However, I have to say that I'm now entirely happy with metrics in general as a way to track quantitative and qualitative measures about systems. For this reason, I'm working on a new library called mulog a micro logging library for events and distributed traces. It is still on early stage, but it is my intention to use it with safely to record and publish events and traces via mulog. The advantage is that you can register as an observer and react to the events.

Finally, if the tracking capabilities of TrackIt are not suited for your case and you can't sample the metrics which are exposed by this library you can always write a little macro like the following one.

let's assume that we have two functions, one that tracks the successful execution and the other one that tracks the failure, given a metric name.

(defn track-success [name])
(defn track-error [name])

Then you can write a macro as follow:

(defmacro tracker
  {:style/indent 1}
  [name & body]
  `(try
     (let [result# (do ~@body)]
       (track-success ~name)
       result#)
     (catch Exception x#
       (track-error ~name)
       (throw x#))))

This macro, won't change the execution flow, but just call track-success and track-error maintaining the original behaviour.

For example:

;; it will call (track-success "foo")
(tracker "foo"
  (/ 1 2))
;;=> 1/2

and

;; it will call (track-error "foo")
(tracker "foo"
  (/ 1 0))
;;=> java.lang.ArithmeticException: Divide by zero

I hope this help.

[dmarkhas]

Thanks for the very detailed response. As you have guessed, I do want a custom hook like that for metrics - we have our own metrics infrastructure that I need to plug into. The macro you propose would do that, of course, but it adds another level of wrapping the code that I don't really see a justification for.

Instead of doing : (safely (/ 1 0)) I would have to do: (safely (tracker "foo" (/ 1 0)))

I'm just saying that it would be great if safely allowed me to get notified of failures without me having to bake that logic into my own code. Notice that I'm not asking to support a custom handler but rather a custom hook that allows others to subscribe to the retries without altering the internal logic of safely.

[BrunoBonacci]

Hi, currently, you can track and push metrics automatically to a large number of commonly used systems. safely can publish the metrics in any of the following systems:

• Console
• Ganglia
• Graphite
• Statsd
• Infuxdb
• Reimann
• NewRelic
• AWS Cloudwatch
• Prometheus
• JMX Beans (expose)

And it can be easily extended to any other system which supports Dropwizard's Metrics

The next version of safely (likely 0.6.0) will replace Metrics with µ/log which provides the event system which can be used to add custom observers to track the error events. With µ/log you should be able to use the events to update any sort of counters or notify any tracking system of your choice.