GoDays, Berlin, 2020-01-21→23

[metalmatze]

Cfp closes 2019-09-30 https://sessionize.com/godays-2020/

[beorn7]

I submitted a talk “How to instrument Go code in 2020”, with the intention of giving an overview of ways of instrumenting your code for logs, traces, metrics, including OpenTelemetry (whatever state it will have) and prometheus/client_golang.

[beorn7]

My talk got accepted.

[kakkoyun]

@bwplotka and I also submitted a talk, titled "Are You Testing Your Observability? Patterns for Instrumenting Your Go Services", with the exact same intentions as you @beorn7, and it's accepted. (Haven't prepared anything concrete though)

Shall we sync up when we can, not to bore people with redundant content? :)

[bwplotka]

Yes @beorn7 quite funny coincidence (: We planned to focus really on metrics only and go through common mistakes of instrumenting metrics in Go, in very details. However let's sync to avoid overlap (:

Looks like Go loves observability ((:

[beorn7]

There might not be that much overlap (and I guess the committee checked our abstracts). My plan is to do a high level review of various instrumentation libraries and techniques, more to help with the initial decision than with the actual usage. Your talk seems way deeper, focusing on metrics, and more hands-on. For reference, here is my abstract:

If you are an astute follower of Peter Bourgon's Go best practices, you have known it since 2016: “You should be instrumenting every significant component of your codebase.” But what does that even mean? Back then, Peter was mostly talking about metrics (and flatteringly recommended Prometheus for it). However, logging can also be seen as instrumentation. (Peter considered it related but separate in his post from 2016.) And then there is instrumentation for distributed tracing. All three of them, metrics, logs, and traces, are nowadays often referred to as the three pillars of observability. Wouldn't it be nice if you could do all three of them within the same framework? OpenTelemetry is a project promising exactly that. There is a wide field of techniques between simple standard Go packages like expvar and log and the newest and most ambitious projects like OpenTelemetry. This talk doesn't intend to give you hands-on instructions for using any of them. Instead, it will provide a (necessarily incomplete) overview of established and new approaches and some guidelines how to navigate the many choices.

[beorn7]

For the record, that's @bwplotka 's and @kakkoyun 's abstract:

Are You Testing Your Observability? Patterns for Instrumenting Your Go Services

Observability is the key to understanding your application running in production. This is especially true for Cloud Native environments like Kubernetes, where users run microservices often written in Go as cattle.

During this talk, two engineers from Red Hat: Kemal and Bartek (Prometheus and Thanos project maintainer) will discuss valuable patterns for instrumenting your Go software with various observability signals like metrics, logs, tracing and profiles. In addition, this talk will demonstrate, leveraging popular open-source libraries and systems including, but not limited to Prometheus, ElasticSearch, Loki, Jaeger! Last but not least, the talk will cover a demo of the example instrumented Go application based on the experience and projects we maintain.

[beorn7]

Not to forget that there is another related talk at the same conference by @johananl :+1:

Monitoring Go Applications with OpenTelemetry

OpenTelemetry is a CNCF sandbox project for standardizing application tracing and monitoring across multiple programming languages, platforms and monitoring vendors. In this talk we’ll provide a brief introduction to OpenTelemetry, explore the OpenTelemetry Go library and demonstrate how it can be used to make Go applications observable.

I guess it should be fairly simple to coordinate all three talks so they complement each other nicely instead of overlapping too much. @johananl , would you like to share a bit of your plans what you'll focus on and such?

[beorn7]

Note that the previous installment of GoDays had @ChimeraCoder 's talk Building Resilient Services in Go, where observability was a big topic, including Veneur and the Standard Sensor Format, which is another take on instrumentation tooling. Returning attendees will appreciate if we can connect the dots and put all those tools into perspective.

(And BTW: @ChimeraCoder is speaking at GoDays again, but with a very different topic. Doubledowning on Veneur would have been fun, we would collectively turn GoDays into an observability conference. :o)

[beorn7]

Or perhaps it has been turned already: There is also I see what you did there! (Tracing in Go) and Introduction to Time-Series. And eBPF, and probably more…

Also, I'll be at Schwuz for the first time in 30 years… :laughing:

[johananl]

@johananl , would you like to share a bit of your plans what you'll focus on and such?

Nice to see so many interesting talks around this area :-)

My plan is to focus purely on OpenTelemetry with an emphasis on tracing. I am assuming at least a portion of the audience is going to be new to distributed tracing (as I was until recently) so I'll probably do a brief intro about tracing in general. Then, I'd like to outline a very short history of OpenTelemetry and talk about the current project status and what can be done with it today. The main part of the talk should be about tracing Go code using OpenTelemetry, with an emphasis on Go-specific patterns/quirks/issues. I might also look a bit at how the library is implemented, not sure yet. I'll also probably do a demo instrumenting multiple services so that we have context propagation over the wire and export traces to Jaeger to have some shiny visuals :-)

My talk isn't written yet so I am rather flexible to changes at this point. I've labeled the talk as "advanced" but now I'm thinking that maybe it should have been intermediate. Oh well :woman_shrugging:

Also @beorn7, seems to me like you are going for breadth whereas @kakkoyun and @bwplotka are going for depth. I think I can also go relatively deep specifically on tracing, which I think nobody else is planning to do.

[beorn7]

There is the talk by the other Johannes and Maximilian:

I see what you did there! (Tracing in Go)

One of the main challenges brought by distributed systems is observability. Besides monitoring and logging, distributed tracing is emerging as a new tool for showing what is going on under the hood. In our talk we introduce the concept of tracing, what a trace consists of and how it is collected, as well as the different tools and formats of traces. We also show how traces can be generated and delivered from functions in Golang and how header propagation can be used to complete traces in more complex setups.

It's marked as beginner level, so perhaps you could leverage that by doing directly into the more advanced details of tracing with OpenTelemetry.

It seems this can all work out quite nicely. My talk is the last of all the observability-touching talks. I love to modify my talks in the last minute to refer to other talks and connect the dots. A rather extreme example was last SREcon EMEA, but this could become even better. :smile:

[johananl]

It's marked as beginner level, so perhaps you could leverage that by doing directly into the more advanced details of tracing with OpenTelemetry.

What you're suggesting makes a lot of sense @beorn7. I'll do my best as I'm still a tracing newbie :-) I'll start thinking in this direction and would love to get feedback from this forum when my talk becomes more concrete.