Monitoring, observability and instrumentation with OpenTelemetry for Altinn 3

martinothamar commented 2 weeks ago

Status

Proposed

Context

Instrumentation and telemetry is typically thought of as an "Ops" thing, used mostly when debugging and investigating bugs and outages in systems. For a long time developers relied mainly on application logs to debug issues. As various vendors brought their monitoring products to market, such as Azure Application Insights, distributed tracing and metrics have become a part of the toolkit for debugging/monitoring. Still, instrumentation of traces and metrics are largely automatic and therefore with minimal context (some app/domain-specific context is typically provided in custom middlewares, enrichers or similar). Application log mesages are often added after issues are discovered which leads to high Mean-Time-To-Recovery (MTTR) and prolonged debugging sessions. Over time application architectures have grown more distributed (microservices) and therefore a lot more complex, and time to recovery takes longer as lots of sparse log lines need to be understood to root-cause an issue.

There has in recent years been a shift in perspective and approach to monitoring/instrumentation/telemetry, brought on at least partially by

OpenTelemetry - an open standard and libraries for instrumentation and telemetry signals such as logs, metrics and traces.
The Observability "movement" - trying to bring a cultural shift to software development, where software developers consider the ability to observe the inner workings of the system while developing.

When trying to develop observable systems, we consider the observability of components in our system. Components should emit some telemetry that will help us understand how it executes in production both in terms of performance and failure modes. Optimizing for observability leads to custom instrumentation as that improves our ability to understand the context of the telemetry. An example of this would be to create a custom OTel span for Instance.Create - including custom domain-specific tags/labels, in addition to the Platform Storage API spans provided by automatic HTTP client instrumentation.

A well-customized instrumentation setup that brings app/domain-specific context can improve debugging-ability a lot, but can also be utilized across several other Software Development LifeCycle (SDLC) phases:

Planning, requirements, design
- How can we measure app success? E.g. P50 duration to fill out the form could be an important metric, and is an indicator of good UX
- Considering the Site Reliability Engineering (SRE) practice, goals/objectives might take the form of Service Level Objectives (SLOs)
Development, testing
- Distributed tracing for building a mental model of how the app is working - e.g. "what runs when?" "how many times does this thing run?"
Deployment, maintenance
- Metrics to detect shifts in latencies, failure rates or similar
- Traces for complex debugging

With this in mind, we can consider observability another tool in the toolbox for improving developer experience in Altinn 3, in addition to improving operational quality of apps and services.

Current state

For service owner apps and services in the Altinn platform we are using Microsoft.ApplicationInsights for automatic instrumentation and Azure Monitor for monitoring. There are a couple of issues with Azure Montitor and this approach in general

Vendor lock-in
- One of the core values for the Altinn platform is openness
- Some service owners don't use Azure Monitor
- app-lib can't/shouldn't easily expose custom instrumentation abstractions
Inflexible architecture
- No system/agent to do enrichment, filtering, routing -
- Team Apps want's a centralized "sink" for app-lib telemetry
- Each service owner want their own "sink" for their telemetry
- Team Platform/Core are interested in their own telemetry
- A pod's node placement might be relevant information for outages where host systems affect runtime
- Buffering with good network locality?
- Ideally there is some indirection between app-lib and agent/sink (it's hard to distribute changes in libraries)
Poor custom instrumentation story
- The APIs for instrumentation are poor (the mechanics of recording)
- Knowing which type of telemetry to use is not straight forward (events, traces, dependencies, requests, exceptions, metrics, logs)
No good way to expose existing Application Insights to app developers, as there is one global log analytics workspace per environment
Service owners don't know what the "out of the box" monitoring story is, or what they need to do themselves

Material

What is Observability? - by Honeycomb.io
Observability Engineering - by Charity Majors, Liz Fong-Jones & George Miranda
Observability for Operational Excellence - part of Microsoft Azures "Well-Architected Framework"

Decision

We should

Use OpenTelemetry
- Instrument using BCL/System.Diagnostics APIs to minimize dependencies
- Instrument using domain-specific contextualization (tags, attributes, naming)
- Create metrics for important SLIs (e.g. request latency, error rate, preferably also domain specific ones)
- app-lib should
- Expose IOpenTelemetryBuilder for custom config for apps
- Unit-test and/or snapshot-test telemetry naming, treat it as public contract (for dashboard, alerting stability)
- Feature-flag OpenTelemetry with UseOpenTelemetry (opt-in, default is still Application Insights SDK for v8)
- Deprecate usage of Application Insights SDK, so that it can be replaced with OpenTelemetry fully in v9
Add APPLICATIONINSIGHTS_CONNECTION_STRING env variable containing App Insights connection string to deployments (we can remove ApplicationInsights__InstrumentationKey later in the case of application deployments)
Make instrumenation a part of the normal developer process across teams
- Reason about failure modes
- How will you know the feature is being used
- Is the feature working as intended (performance etc)
- Is alerting needed (typically async processes, queues etc require extra consideration)
Use Grafana Alloy for telemetry collection on each app cluster
- So that we have a local (in terms of networking) buffer for telemetry
- So that we can do more intelligent and flexible sampling
- So that we can do custom enrichment, filtering, routing (pipeline model)
Use Grafana LGTM (Loki, Grafana, Tempo, Mimir) stack for data storage
- So that we can visualize, explore and alert based on telemetry, using Grafana
TBD: Alerting
- Potentially Alertmanager & PrometheusRule Custom Resource Definitions (CRDs), deployed as dynamic config from Studio?

Why OpenTelemetry

OpenTelemetry is quickly becoming the de facto standard for telemetry instrumentation and shipping (the protocol). Most (if not all) vendors, databases, programming languages and frameworks/libraries are moving in the direction of using the relevant aspects of OpenTelemetry (semantic convetions, collector, wire-protocol etc).

For us as platform builders, it enables us to

Decouple configuration from source code - sampling strategies, resource attributes, exporter destination etc
Minimize vendor-related dependencies in libraries
Decouple architecturally from specific vendors (Azure Monitor)
Enable service owners to bring their own monitoring/observability solution, while still utilizing our instrumentation (custom sink/exporter)
Use BCL for instrumentation

Why Grafana products

Grafana
- Service owners have requested it
- It seems to be the most widely used visualization tool for the monitoring/operational domain
Grafana Alloy
- It is Grafana's OTel Collector distribution
- Supports native OTel pipelines (whatever comes from standard OTel collector)
- Can do logs and metrics scraping as well
- Pyroscope/continuous profiling support (which is currently not OTel standardized)
- Can be viewed as a superset of OTel Collector
Grafana Loki, Tempo, Mimir
- Databases for logs, traces and metrics respectively
- Prioritizes low Total Cost of Ownership (TCO) over performance and indexing capabilities
- Uses object storage for persistence (and typically Write Ahead Logs (WAL) on Persistant Volumes (PVs) for durability)
- Builds on a lot of existing open source (Prometheus, Alertmanager, Parquet etc..)
- Highly configurable (which is both good and bad)
- Highly available (hashicorp/memberlist gossip protocol)
- Good self-monitoring capabilities in the helm-charts, compatible with Prometheus Operator/kube-prometheus-stack (wich is in use today)

Instrumentation principles

Prefer traces over logs (traces are much better correlated and visualized)
Don't make traces too deep
Put as much context on the top trace as possible (user IDs, client information)
Don't make traces too long (in number of spans and/or duration)
- For background/batch jobs, be particularly careful
Be mindful of high cardinality in labels/attributes/tags
- Query performance can degrade significantly, very few databases have solved the cardinality problem well
Metrics are for coarse aggregation, timeseries visualization
Avoid "baggage" if possible, is it can be difficult to use safely/correctly

Plan

First delivery - Q2 2024
- Instrument app-lib
- Export telemetry directly to Azure Monitor in deployed environments
- Export to local LGTM stack when running locally with app-localtest
- Add Azure Monitor datasource to Grafana in app clusters
- Altinn Studio docs
- What comes out of the box - exploration/debugging
- Custom instrumentation
- Custom exporter/sink
Agents, architectural flexibility
- Implement agent in app clusters
- Export telemetry through agents
- Consider enrichment, better routing/cetralization per team
Grafana databases
- Implement telemetry databases
- Built in dashboard
- Custom dashboards
- Alerting

Consequences

We should be able to drive more decision making and analysis from telemetry
- "3 service owners are using the new Notifications abstractions in app-lib, let's ask them for feedback"
- "P99 duration for filling out the form on this process task just 2x'ed, we need to investigate"
Improved operational quality
- Less bugs
- Faster bugfixes
- Proactive communication
Easier to onboard app developers - use trace exploration as a way to learn the lifecycle of instances and processes, tasks
Simplifies debugging for platform developers and app developers
Enables custom instrumentation for app developers
Enables service owners to have custom monitoring/observability solutions where both app-lib and user code telemetry is correlated and contextualized
Increased operational/maintenance burden in terms of LGTM stack
Cheaper Azure bill - self hosted LGTM based on Blob storage is cheaper than Azure Log Analytics
OpenTelemetry still in active development (though stable both in spec and .NET impl)
- Need to be thoughtful around abstractions exposing OTel, i.e. configuration
Code changes required for any existing custom instrumentation except for logs (due to Microsoft.Extensions.Logging), such as telemetry processors/middleware/filters/enrichers

Considerations

Both OTel and App Insights SDK use W3C Trace-Context propagation by default for distributed traces
Both Application Insights SDK and OpenTelemetry builds on Microsoft.Extensions.Logging, so application logs should look the same after a migration (no code changes)
How to solve frontend telemetry/RUM use-cases
MS will be moving to OTel protocol (meaning, likely deprecating the classic SDK at some point): https://github.com/MicrosoftDocs/azure-docs/commit/25d58a0c1e5a1d5740d99fd68d89a9372042838e

Previous work

Example implementation of OTel and instrumentation (and migration from App Insights SDK): https://github.com/Altinn/app-lib-dotnet/pull/589
localtest implementation of OTel, Grafana LGTM stack and dashboards: https://github.com/Altinn/app-localtest/pull/97

martinothamar commented 2 weeks ago

cc @sduranc @bengtfredh - this proposal reflects our current thoughts on the topic. Would appreciate any reviews/edit from you guys. In particular the infra stuff is just guessing on my part still

HauklandJ commented 2 weeks ago

Because comment diff is not so good, here are the changes I made: some typos write out the abbreviations that may not be so obvious: (SDLC, SLO, CRD, TCO, WAL og PV)

EDIT: never mind, there is a good way to see diff, TIL

martinothamar commented 1 week ago

Seems like MS plans for moving to OTel protocol has been public for some time: https://github.com/MicrosoftDocs/azure-docs/commit/25d58a0c1e5a1d5740d99fd68d89a9372042838e

Additionally, ingestion based on instrumentation key is going to go out of support on 31.03.2025. Considering the Azure Monitor OTel exporter only supports connection string anyway, I'll add another point about migrating from ApplicationInsights__InstrumentationKey to APPLICATIONINSIGHTS_CONNECTION_STRING for the env variables added to the pod in app clusters. app-lib has been updated to support APPLICATIONINSIGHTS_CONNECTION_STRING in addition to the older variants (prefers connection string in case there are both) in https://github.com/Altinn/app-lib-dotnet/pull/589/commits/0ba8f5abc4552fde56aa952afcfddf7dcf56f09a

https://learn.microsoft.com/en-us/azure/azure-monitor/app/asp-net-core?tabs=netcorenew

martinothamar commented 4 days ago

NAIS just posted a blogpost on their work around OTel: https://nais.io/blog/posts/otel-from-0-to-100/

Altinn / architecture-decision-log