Closed istathar closed 1 year ago
Does this force code which is encapsulated by instrumentation to be strictly evaluated when crossing span enclosing boundaries?
It could well be that the metric
values accumulated in the telemetry are not forced until needed by the HTTP I/O code to send out. This happens shortly after a span concludes (because the Datum is sent along the queue, and then the exporter does its thing, leading to the Haskell runtime deciding to evaluate).
So, if there was a problem pending in a lazy value (thunk, evaluation of which would result in an exception being thrown) in, say an HTTP request going wrong and a value about that HTTP request being in the telemetry being on the path that would raise that exception, then maybe yes we'd see the spooky action at a distance we're seeing.
See also https://hackage.haskell.org/package/base-4.16.2.0/docs/Control-Exception.html#v:evaluate
We've had some suggestions that maybe we're getting a bit of delayed exception problems when using
encloseSpan
. The evidence is circumstantial, but it could be that thecatch ... SomeException
in Core.Telemetry.Honeycomb is at last what's actually forcing a value that was, in fact, going to bear an exception.We know we are using the telemetry values submitted, so better to just get on with evaluating them at the use site rather than risking it for later.