quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.82k stars 2.69k forks source link

http server metrics have wrong uri for load shedded requests #44437

Open mhajas opened 5 days ago

mhajas commented 5 days ago

Describe the bug

For load shedded requests metrics http_server_requests_seconds* contain incomplete uri when REST subresource is targetted.

Expected behavior

When triggering URL: http://localhost:8080/hello/sub/in-resource

it is expected that both success and error requests have the same uri in metric:

http_server_requests_seconds_count{method="HEAD",outcome="SUCCESS",status="200",uri="/hello/sub/in-resource",}
http_server_requests_seconds_count{method="HEAD",outcome="SERVER_ERROR",status="503",uri="/hello/sub/in-resource",}

Actual behavior

The actual behavior is the uri is incomplete for error response:

http_server_requests_seconds_count{method="HEAD",outcome="SERVER_ERROR",status="503",uri="/hello/sub",}

How to Reproduce?

quarkus create
quarkus ext add quarkus-micrometer-registry-prometheus

Greetings.java

package org.acme;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/hello")
public class GreetingResource {
    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello() {
        return "Hello from Quarkus REST";
    }

    @Path("/sub")
    public  SubResource getSubResource() {
        return new SubResource();
    }
}

SubResource.java

package org.acme;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

public class SubResource {

    @GET
    @Path("/in-resource")
    @Produces(MediaType.TEXT_PLAIN)
    public String getInSubResource() throws InterruptedException {
        Thread.sleep(500);
        return "in-resource";
    }
}

To simulate load-shedding we need the following: application.properties

quarkus.thread-pool.queue-size=1
quarkus.thread-pool.max-threads=1

Then execute this command:

curl -I http://localhost:8080/hello/sub/in-resource & \
curl -I http://localhost:8080/hello/sub/in-resource & \
curl -I http://localhost:8080/hello/sub/in-resource

This should cause 1 request to be load shedded:

HTTP/1.1 503 Service Unavailable

And quarkus log should contain:

WARN  [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-1) Worker thread pool exhaustion, no more worker threads available - returning a `503 - SERVICE UNAVAILABLE` response.

Then navigate to http://localhost:8080/q/metrics and observe the difference mentioned in actual and expected behavior.

Output of uname -a or ver

Darwin mhajas-mac 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:00:32 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6030 arm64

Output of java -version

openjdk version "21.0.4" 2024-07-16 LTS OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)

Quarkus version or git rev

3.16.2 ### Build tool (ie. output of `mvnw --version` or `gradlew --version`) Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937) Maven home: /Users/mhajas/.m2/wrapper/dists/apache-maven-3.9.9-bin/33b4b2b4/apache-maven-3.9.9 Java version: 21.0.4, vendor: Eclipse Adoptium, runtime: /Users/mhajas/.sdkman/candidates/java/21.0.4-tem Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "15.1", arch: "aarch64", family: "mac" ### Additional information In [`VertxHttpServerMetrics`](https://github.com/quarkusio/quarkus/blob/ea7996c7f2fb82e549dd79d21629648c87e0b9e2/extensions/micrometer/runtime/src/main/java/io/quarkus/micrometer/runtime/binder/vertx/VertxHttpServerMetrics.java#L188-L190) path is obtained from context with key `UrlPathTemplate`. This URL is written there by the [`ObservabilityHandler`](https://github.com/quarkusio/quarkus/blob/ea7996c7f2fb82e549dd79d21629648c87e0b9e2/extensions/resteasy-reactive/rest/runtime/src/main/java/io/quarkus/resteasy/reactive/server/runtime/observability/ObservabilityHandler.java#L45-L47) However, the `ObservabilityHandler` for subresources is running only when request is processed therefore it is never triggered for load shedded requests. We encountered this in Keycloak project while creating Grafana dashboard for server errors percentage and error response changes per uri.
quarkus-bot[bot] commented 5 days ago

/cc @ebullient (metrics), @jmartisk (metrics)