Local monitoring setup with OTEL and Grafana LGTM-stack

martinothamar commented 5 months ago

Description

This PR includes a separate Docker compose file to setup OTEL collector and the Grafana LGTM stack for local monitoring Includes

Backends for logs, traces and metrics
Correlation is built into OTEL, navigation between logs and traces in Grafana UI is configured
Dashboards published by .NET team for .NET 8/Aspire is automatically provisioned
Custom dashboard for Altinn apps

Altinn app dashboard:

ASP.NET core dashboard:

Explore logs:

Explore traces:

Explore metrics:

Decisions:

How "built in" should monitoring stack be - included in the primary docker compose file or a separate one? The backends are relatively lightweight, but not free
Grafanas own LGTM-stack development image can also be considered but it is far less flexible, and harder to match the fully deployed solution: https://github.com/grafana/docker-otel-lgtm

TODOs

Instrument app-lib a lot better
Continuous profiling setup? It is being considered for OTEL AFAIK, might be too early
Dashboard for runtime metrics (exceptions, threads count, lock contention etc)

Related Issue(s)

Verification

[x] Your code builds clean without any errors or warnings
[x] Manual testing done (required)
[ ] Relevant automated test added (if you find this hard, leave it and we'll help out)
[x] All tests run green

Documentation

[ ] User documentation is updated with a separate linked PR in altinn-studio-docs. (if applicable)

ivarne commented 5 months ago

Not a requirement for this PR, but I'm missing a link in the localtest dotnet app to make this more discoverable, and ideally a message informing that local telemetry is not running if it isn't (just send a http request to verify).

Docs does currently not suggest using the Makefile, so adding everything together in a single makefile would be nice. Potentially behind a --profile flag (as we use in --profile localtest).

For the other services we have separate compose files for docker and podman. Would that be required here also?

sduranc commented 5 months ago

This is very nice @martinothamar 💯.

How "built in" should monitoring stack be - included in the primary docker compose file or a separate one? The backends are relatively lightweight, but not free

IMO it should be in a separate one.

Grafanas own LGTM-stack development image can also be considered but it is far less flexible, and harder to match the fully deployed solution: https://github.com/grafana/docker-otel-lgtm

What are we missing exactly from running that image? Is it adding dashboards?

martinothamar commented 5 months ago

but I'm missing a link in the localtest dotnet app to make this more discoverable

Good point, now there is a Grafana menu item

RE:

Docs does currently not suggest using the Makefile, so adding everything together in a single makefile would be nice. Potentially behind a --profile flag (as we use in --profile localtest).

and

IMO it should be in a separate one.

Opted for using compose profiles in the latest commit

--profile "monitoring" brings up existing setup + monitoring profile
--profile "*" brings up existing setup + every other profile
No profile specified brings up the existing setup

What are we missing exactly from running that image? Is it adding dashboards?

I was guessing that configuration would be more of a hassle but looking at it now maybe it isn't. I won't be able to specify command in the compose file, and for dashboards I will have to get the grafana version number, but the rest seems like it should work fine (the actual config files and such). I will give it a shot

martinothamar commented 4 months ago

Running out of time a bit here so I'll keep the LGTM stack as is for now. Other than that

Grafana is behind proxy (no exposed port)
Only OTel GRPC port exposed on OTel collector
Some updates to the dashboard (will update screenshot in a bit)
Updated README
Implemented podman stuff
Implemented a simpler run.cmd cross-platform script and documented it as preview

Altinn / app-localtest