http server requests metrics missing with spring boot 3.0.5 and `@EnableReactiveMethodSecurity`

[fabioacsilva]

Describe the bug If I deploy an application with spring boot 3.0.5 and micrometer prometheus exporter to kubernetes, the http_server_requests metrics are missing. We bumped version from spring-boot 3.0.2 to 3.0.5 and we lost http_server_requests metrics.

spring-boot 3.0.2 uses micrometer 1.10.3 and it works fine.

Environment

• Micrometer version 1.10.5
• Micrometer registry prometheus
• OS: Linux
• Java version: 17

To Reproduce How to reproduce the bug: Use spring-boot version 3.0.5

Expected behavior http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.004194304",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.005592405",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.006990506",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.008388607",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.009786708",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.011184809",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.01258291",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.013981011",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.015379112",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.016777216",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.022369621",} 0.0 http_server_requests_seconds_bucket{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="$my_endpoint",le="0.027962026",} 0.

Similar to #3207

[jonatan-ivanov]

Are you sending requests to the application before checking the metrics? The http server metrics won't show up until a request has been processed.

ON top of that you need to enable histograms and the prometheus endpoint to see the _bucket time series:

management.endpoints.web.exposure.include=prometheus
management.metrics.distribution.percentiles-histogram.http.server.requests=true

Also, could you please to go to start.spring.io and generate a project there to see if it works for you? If I follow your reproduction steps ("Use spring-boot version 3.0.5"), I can't reproduce the issue.

If it works could you please provide a minimal simple reproducer in Java that can reproduce the issue? If it does not, could you pease tell us more about your local environment?

[fabioacsilva]

Hey @jonatan-ivanov, indeed we have multiple requests before checking the metrics. Our application.yml has the following configuration:

management:
  endpoints:
    web:
      exposure:
        include:
          - "health"
          - "metrics"
          - "prometheus"

  endpoint:
    health:
      show-details: always
      show-components: always

  health:
    probes:
      enabled: true

  metrics:
    distribution:
      percentiles[http.server.requests]: 0.5, 0.90, 0.95, 0.99
      percentiles-histogram[http.server.requests]: true
      slo[http.server.requests]: 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s

      percentiles[http.client.requests]: 0.5, 0.90, 0.95, 0.99
      percentiles-histogram[http.client.requests]: true
      slo[http.client.requests]: 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s, 30s

Kubernetes version: 1.24.6 Dockerfile: gcr.io/distroless/java17-debian11:nonroot

I'll try to provide a reproducer ASAP, but everything was working with spring-boot 3.0.2 and micrometer 1.10.3, it started to fail on 3.0.5 and 1.10.5.

[jonatan-ivanov]

Can you reproduce it locally or is this only happening in Kubernetes? If you can repro locally, can you try the link I sent you above and generate a sample project and try with it? If you cannot, is it possible hat you have an extra property source there that disables metrics (/actuator/env might help figuring this out)?

[fabioacsilva]

Hey @jonatan-ivanov although the issue is still happening, I can't find a way to reproduce it. So it all points that some other dependency of the project is messing with micrometer metrics.

I'll close it for now, since I can't provide you a way of reproducing. Sorry for the trouble. If I find anything useful meanwhile I'll reopen.

[jonatan-ivanov]

I think I still have two ideas:

1. Can you try going to 3.0.3 and then 3.0.4?

2. Do you happen to have any Java Agents in the environment where this is happening?

[fabioacsilva]

Hey again @jonatan-ivanov

1. Yes, it starts to stop providing metrics once we upgrade to 3.0.3
2. No

We are still investigating the issue and will let you know if we find anything relevant for micrometer

[fabioacsilva]

Hey @jonatan-ivanov I'll reopen the issues as we have found what is causing the metrics to disappear once we bump spring from 3.0.2 to 3.0.3+

If you have a class with @EnableReactiveMethodSecurity, HTTP metric will disappear, for example:

@Configuration
@EnableWebFluxSecurity
@EnableReactiveMethodSecurity
class AuthConfiguration {

    @Bean
    fun configureSecurity(http: ServerHttpSecurity): SecurityWebFilterChain {
        return http.httpBasic().and().authorizeExchange { it.anyExchange().permitAll() }.build()
    }
}

You can find a reproducible project here: https://github.com/fabioacsilva/springboot-303-micrometer-bug

If you change the spring-boot version to 3.0.2 it works. If you leave 3.0.3+ and comment out @EnableReactiveMethodSecurity it works as well.

[bclozel]

This is a duplicate of https://github.com/spring-projects/spring-security/issues/12780

[fabioacsilva]

Thank you @bclozel

[shweta5595]

Hi , http_server_requests_seconds metric is missing with spring boot 3.2.1 . However , we have all other http_server_requestsseconds* metrics available. What is the correct metric name after spring boot upgrade.

[shakuzen]

@shweta5595 the metric name hasn't changed. Please try with at least the latest patch release in the minor version of Spring Boot you're using - at the moment that's Spring Boot 3.2.10. If you still have the issue, please open a new issue with the details for us to investigate, ideally a minimal sample project that reproduces the issue.