Provide default Grafana dashboards for the produced metrics

[Inithron]

Description

Hi, I am using Quarkus with the micrometer and some other extensions. Everything works fine but for example the Kafka extension creates a huge amount of metrics. I am new to this topic so I have problems to figure out which of the metrics are really important or valuable. I have found a dashboard visualizing some basic metrics: https://grafana.com/grafana/dashboards/14370-jvm-quarkus-micrometer-metrics which I like very much.

Implementation ideas

My proposal now is to provide similar dashboards for the other extensions: https://quarkus.io/guides/telemetry-micrometer#quarkus-extensions-using-micrometer. This would help everybody visualizing the metrics without spending days in getting familiar with Grafana and creating their own dashboards.

After the creation of this enhancement I have seen that there is a Grafana Dashboard Generator. Unfortunately the generator is not really easy to install/use and the last commit was 4 years ago. So I am not sure if it still works but it gives me another idea: Instead of creating manually Grafana dashboards the dashboard generator could be added to the Quarkus Dev Services or into an own Quarkus extension.

[quarkus-bot[bot]]

/cc @ebullient (metrics), @jmartisk (metrics)

[geoand]

cc @alesj @brunobat

[alesj]

We've just added this:

• https://quarkus.io/version/main/guides/observability-devservices

If there is some easy way to pre-add custom / specific dashboard(s) via Grafana configuration, this would be the way to do it. e.g. adding the right config into the container

• https://github.com/quarkusio/quarkus/blob/main/extensions/observability-devservices/testcontainers/src/main/java/io/quarkus/observability/testcontainers/GrafanaContainer.java

[melloware]

@Inithron i had just asked this same thing on Zulip. I have that same Grafana dashboard as you and I think the goal is to have that in the LGTM stack out of the box.

[Inithron]

@alesj, @melloware I think there was a misunderstanding. My intention was not to add the existing dashboard to the Dev Services. I would like to have for each Quarkus extension one dashboard containing the metrics which are created by this extension. The alternative would be to integrate the dashboard generator into the Dev Services to generate the dashboard(s) automatically.

[melloware]

@Inithron I see so ideally we have dashboards for all the things like Kafka and any other MicroMeter Metrics. Those deashboards should default in the LGTM DevResource so they come out of the box. People can then "export" those dashboards from Grafana to use in their production Grafana environment.

[Inithron]

Yes, that would be great. But first someone needs to create these dashboards.

[melloware]

Yep exactly what I asked on Zulip: https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/Observabilty.20Dev.20Services.20GRAFANA.20OTEL.20LGTM

[melloware]

it doesn't work but here is something of a prototype where we need to add a Prometheus scraper to scrap /q/metrics to get the Micrometer data into dashboard.

public class LgtmContainer extends GrafanaContainer<LgtmContainer, LgtmConfig> {
    protected static final String LGTM_NETWORK_ALIAS = "ltgm.testcontainer.docker";

    protected static final String PROMETHEUS_CONFIG = """
                global:
                  scrape_interval: 15s
                  evaluation_interval: 15s
                storage:
                  tsdb:
                    out_of_order_time_window: 10m
                scrape_configs:
                  - job_name: 'quarkus-micrometer'
                    metrics_path: '/q/metrics'
                    scrape_interval: 15s
                    static_configs:
                      - targets: ['ltgm.testcontainer.docker:8080']
            """;

    public LgtmContainer() {
        this(new LgtmConfigImpl());
    }

    public LgtmContainer(LgtmConfig config) {
        super(config);
        addExposedPorts(config.otlpPort());
        addFileToContainer(PROMETHEUS_CONFIG.getBytes(), "/otel-lgtm/prometheus.yaml");

    }

[melloware]

Ok I got it working. Next I need to figure out how to load the dashboard I want. But making good progress!

[alesj]

Do we need this scraping? Since isn't OTLP supposed to push this into collector, and the this fwds to Prometheus -> Grafana?

[brunobat]

Correct @alesj. On the grafana devservice there is no scraping. It's all OTLP protocol because it includes an OTel Collector. The dashboard is usually placed on a specific folder of Grafana, which we have no access to... We can always ask the Grafana folks. Another solution is to add the dashboard through API or create a dashboard on the Grafana dashboard registry.

[melloware]

@alesj I do not see the micrometer metrics available out of the box with your current LGTM stack. I heard there is something called a "Micrometer Bridge" but i can't find any reference but I can confirm the JVM and Micrometer metrics are not flowing through to LGTM container. Adding the scraper above is the only way I am able to get metrics into my LGTM Grafana Dashboard.

[melloware]

@alesj @brunobat according to this document the Micrometer stuff is disabled by default: https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0

Micrometer metric bridge has been disabled by default. You can enable it using OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true or -Dotel.instrumentation.micrometer.enabled=true.

[brunobat]

@alesj @brunobat according to this document the Micrometer stuff is disabled by default: https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0

Micrometer metric bridge has been disabled by default. You can enable it using OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true or -Dotel.instrumentation.micrometer.enabled=true.

We don't support that yet. I'm not sure if it works out of the box.

[melloware]

@brunobat @alesj yes that is why i am working on my PR to have a kick button default Quarkus Dashboard. I am going to keep going on it and submit a PR you guys can choose to do what you want with it.

[brunobat]

Sure! Bring it on!

[melloware]