Automatic instrumentation

[alexandergunnarson]

Two questions about automatic instrumentation.

The first is with respect to this quote, from the documentation:

According to our benchmarks, the performance impact of automatic instrumentation should not exceed 20% in worst-case situations like articulated and math-intensive Clojure code, while in more common-case scenarios the slow-down should not be noticeable at all.

Are these benchmarks made available to the public? If so, where? And will this performance hit apply only when "math-intensive functions" (whatever that means with that level of generality, as math is ubiquitous in and essential to code of virtually any complexity) are actually executed, or is it a blanket hit, dragging down the performance ("up to 20%") of all components of a JVM containing this type of instrumented code?

The second is this: is it possible to do AOT automatic instrumentation in such a way that the performance hit is not incurred, or is it incurred because of inherent inefficiencies in the bytecode generated by automatic instrumentation?

Thanks for your help!

[pron]

Automatic instrumentation is an experimental feature; use it only if it significantly helps you.

Are these benchmarks made available to the public?

They might be in the Pulsar codebase, but I don't remember. You can look.

And will this performance hit apply only when "math-intensive functions" ...

Automatic instrumentation only applies to Clojure code and does not instrument anything written in Java. It basically treats any Clojure function call as a potentially blocking one.

AOT automatic instrumentation in such a way that the performance hit is not incurred

As a general rule, AOT instrumentation is never better, and possibly worse than dynamic instrumentation, as more gets instrumented. In any event, AOT instrumentation would only work for AOT-compiled Clojure code.

[alexandergunnarson]

Thanks for your point-by-point reply!

Automatic instrumentation is quite useful in terms of reducing code, so I'd have to say that (aside from its possible performance implications) does significantly help me.

As for the performance hit, is that incurred only once when the Clojure code is dynamically compiled, or does it apply to JIT situations, possibly being incurred as many times as the JIT compiler further "tunes" the bytecode? Or is the performance hit referring more to the runtime hit — that is, the compilation isn't necessarily the slowdown referred to, but rather, automatic instrumentation simply generates "up to 20%" less performant code than manual instrumentation? If so, are mainly "math-intensive functions" affected, and if so, why?

Sorry for all the questions — I think it's the last round of them haha!

[pron]

Automatic instrumentation is quite useful in terms of reducing code

How is that? All it does is allow you to use fefn instead of defsfn when defining blocking functions.

As for the performance hit, is that incurred only once when the Clojure code is dynamically compiled, or does it apply to JIT situations, possibly being incurred as many times as the JIT compiler further "tunes" the bytecode?

The instrumentation itself adds negligible overhead.

Or is the performance hit referring more to the runtime hit

Yes. The difference is that Quasar instruments only functions defined with defsfn, while with automatic instrumentation, most Clojure functions are instrumented.

If so, are mainly "math-intensive functions" affected, and if so, why?

As automatic instrumentation instruments even functions that are computation-intensive rather than only functions that block on IO, those functions' performance may suffer a significant hit (although YMMV). Code written in Java is not affected.

Again, automatic instrumentation is generally useful if you want to work with 3rd-party code without modifying it or adding separate suspendable! instructions.

[alexandergunnarson]

Thanks for your reply!

How is that? All it does is allow you to use defn instead of defsfn when defining blocking functions.

I suppose you're right in that it doesn't reduce code. That said, it certainly does reduce some cognitive overhead. Automatic instrumentation is nice in the same way that not having to think about <! vs. <!! is. I have some complex code in which maybe not any code is potentially blocking, but certainly a sizable portion, and I could go back and use defsfn instead of defn everywhere, but that might also require retooling of macros, etc. etc. that would (potentially) be more of a pain to do than using automatic instrumentation.

As automatic instrumentation instruments even functions that are computation-intensive rather than only functions that block on IO, those functions' performance may suffer a significant hit (although YMMV).

So what you're saying is that I should use defsfn or suspendable! and limit their scope as much as possible, ensuring that as little work as possible is done in blocking functions and is rather delegated to non-blocking ones, like so?:

(defn complex-non-blocking-fn []
  (do-tons-of-complex-math-intensive-calculations))

(defsfn blocking-fn []
  (<! some-previously-defined-channel)
  (complex-non-blocking-fn))

[pron]

So what you're saying is that I should use defsfn or suspendable! and limit their scope as much as possible, ensuring that as little work as possible is done in blocking functions

That would be premature optimization. Blocking functions are, well, blocking, and so usually have a significant latency anyway. Write your code in the clearest, most natural way, and only if you have reason to suspect some instrumentation-related performance issues (my guess is that those would be unlikely), try to do as you propose.

[circlespainter]

@alexandergunnarson I think the benchmarks used at that time are still here https://github.com/circlespainter/pulsar-auto-instrument-bench, feel free to have a look.