Sentry starter turs off prometheus metrics from spring-boot

[llaros]

Integration

sentry-spring

Java Version

11

Version

5.6.1

Steps to Reproduce

Use spring-boot 2.7.2 application, add spring-boot-starter-actuator, micrometer-registry-prometheus and any @RestController. After a call there is http.server.requests metrics available at /actuator/metrics/http.server.requests endpoint.

After adding sentry-spring-boot-starter metrics still works provided sentry is not turned on via sentry.dsn property. After autoconfiguring sentry starter, the metrics is missing.

Expected Result

Bring back http.server.requests metrics and be able to use sentry at the same time.

Actual Result

WebMvcMetricsFilter is not registered as OncePerRequestFilter filter chain when sentry is autoconfigured. The problem lies probably in declaring in the FilterRegistrationBean in SentryAutoConfiguration what overrides spring boot configuration which sets the WebMvcMetricsFilter.

[adinauer]

Hello @llaros thank you for raising this. I'm not sure when I'll get to reproducing this - sorry. In the meantime could you please test with the latest version?

[llaros]

I did and it doesn't work either.

[adinauer]

Thanks for testing! We'll update here once we did some testing.

[llaros]

@adinauer Is there maybe any update regarding the possible date of confirming the issue?

[adinauer]

@llaros I'm unable to reproduce. Both with Sentry enabled and disabled the metric http.server.requests shows up in actuator/metrics and the endpoint also works under actuator/metrics/http.server.requests after I've sent a request to one of the controllers. Is it possible you didn't send a request to any endpoint yet?

[llaros]

@adinauer no, please have a look at my test

/**
 * Ensure that there is sentry on the classpath.
 * Remove {@link xyz.abc.ActuatorWorkaroundSentryAutoConfiguration}
 * and this tests if the test does not pass.
 */
@ExtendWith(ObhRestAssuredExtension.class)
@DirtiesContext
@SpringBootTest(
    webEnvironment = DEFINED_PORT,
    properties = {
        "obh.starter.http.sentry.workaround.enabled=false",
        "sentry.dsn=http://8a9cca79c3544258827c24d1e23a651a@obh-sentry-tst:9000/5"
    }
)
class BuildInHttpServerMetricsDoesNotWorkWithSentryIT {

    private static final String SERVER_REQUEST_METRIC_NAME = new MetricsProperties.Web.Server.ServerRequest().getMetricName();

    @Autowired
    MeterRegistry meterRegistry;

    @Test
    void shouldProveThatBuildInMetricsIsMissingWhenSentryIsOn() {
        assertThat(meterRegistry.find(SERVER_REQUEST_METRIC_NAME).meter()).isNull();

        given()
            .when()
            .get("/internal/v1/echo/" + TestSequencers.nextString())
            .then()
            .statusCode(200);

        assertThatWorkaroundIsStillNeeded(SERVER_REQUEST_METRIC_NAME);
    }

    private void assertThatWorkaroundIsStillNeeded(String name) {
        assertThat(meterRegistry.find(name))
            .extracting(Search::meter)
            .isNull();
    }

}

As you see this is checked before and after get to the conroller. Maybe you've got different order of initialization of spring-boot/sentry beans?

[adinauer]

Can you please provide a repro? Maybe your test is asserting too soon and the metric hasn't been produced yet?

[llaros]

@adinauer We wrote the test after we realized we have no http.server.request metrics on production in all our microservices since we configured sentry via sentry.dsn. So no, this is proving that something is not working. If I turn on our workaround the metrics registry in this test (via property obh.starter.http.sentry.workaround.enabled=true) it' s returning a valid meter.

As for repo I haven't got any at hand...

[adinauer]

@llaros can you please share more details on what your workaround is doing that you can enable/disable with obh.starter.http.sentry.workaround.enabled.

Also can you please share debug logs around the filters / configs. Maybe filter them by metrics (case insensitive), ideally with some context lines around it.

I tried with Sentry enabled and disabled, then diffed the files (using ag "[mM]etrics" --context=3 --nonumbers no-sentry.full.log > no-sentry.log && sed -i .bak -e 's/.*\[.*main\]//g' no-sentry.log). The webMvcMetricsFilter doesn't show up in the diff for me:

--- /Users/adinauer/dev/debug/with-sentry.log   Thu Sep 22 11:03:30 2022
+++ /Users/adinauer/dev/debug/no-sentry.log Thu Sep 22 11:03:24 2022
@@ -1,6 +1,6 @@
-o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'sentryTracingFilter'
-o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'sentryTracingFilter' via factory method to bean named 'sentryHub'
-o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'sentryTracingFilter' via factory method to bean named 'transactionNameProvider'
+o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
+w.s.c.ServletWebServerApplicationContext : Published root WebApplicationContext as ServletContext attribute with name [org.springframework.web.context.WebApplicationContext.ROOT]
+w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 846 ms
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'webMvcMetricsFilter'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'org.springframework.boot.actuate.autoconfigure.metrics.web.servlet.WebMvcMetricsAutoConfiguration'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'management.metrics-org.springframework.boot.actuate.autoconfigure.metrics.MetricsProperties'
@@ -54,7 +54,7 @@
 o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'errorPageSecurityFilter' via factory method to bean named 'org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@4eb386df'
 --
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'characterEncodingFilter'
-o.s.b.w.s.ServletContextInitializerBeans : Mapping filters: filterRegistrationBean urls=[/*] order=-2147483648, filterRegistrationBean urls=[/*] order=-2147483647, filterRegistrationBean urls=[/*] order=-2147483647, springSecurityFilterChain urls=[/*] order=-100, filterRegistrationBean urls=[/*] order=2147483647, filterRegistrationBean urls=[/*] order=2147483647, characterEncodingFilter urls=[/*] order=-2147483648, formContentFilter urls=[/*] order=-9900, requestContextFilter urls=[/*] order=-105
+o.s.b.w.s.ServletContextInitializerBeans : Mapping filters: filterRegistrationBean urls=[/*] order=-2147483647, springSecurityFilterChain urls=[/*] order=-100, filterRegistrationBean urls=[/*] order=2147483647, characterEncodingFilter urls=[/*] order=-2147483648, formContentFilter urls=[/*] order=-9900, requestContextFilter urls=[/*] order=-105
 o.s.b.w.s.ServletContextInitializerBeans : Mapping servlets: dispatcherServlet urls=[/]
 o.s.b.a.m.w.servlet.WebMvcMetricsFilter  : Filter 'webMvcMetricsFilter' configured for use
 o.s.b.w.s.f.OrderedRequestContextFilter  : Filter 'requestContextFilter' configured for use
@@ -71,9 +71,9 @@
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'org.springframework.security.config.annotation.authentication.configuration.AuthenticationConfiguration'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'enableGlobalAuthenticationAutowiredConfigurer'
 --
-o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'sentrySpanRestTemplateCustomizer'
-o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'io.sentry.spring.boot.SentryAutoConfiguration$HubConfiguration$SentryPerformanceRestTemplateConfiguration'
-o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'sentrySpanRestTemplateCustomizer' via factory method to bean named 'sentryHub'
+o.s.b.a.AutoConfigurationPackages        : @EnableAutoConfiguration was declared on a class in the package 'io.sentry.samples.spring.boot'. Automatic @Repository and @Entity scanning is enabled.
+o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'jacksonObjectMapper' via factory method to bean named 'jacksonObjectMapperBuilder'
+o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'mappingJackson2HttpMessageConverter' via factory method to bean named 'jacksonObjectMapper'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'metricsRestTemplateCustomizer'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'org.springframework.boot.actuate.autoconfigure.metrics.web.client.RestTemplateMetricsConfiguration'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'restTemplateExchangeTagsProvider'
@@ -83,10 +83,7 @@
 o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'restTemplateBuilder' via factory method to bean named 'restTemplateBuilderConfigurer'
 o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'restTemplate' via factory method to bean named 'restTemplateBuilder'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'webClient'
---
-o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'sentrySpanWebClientCustomizer'
-o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'io.sentry.spring.boot.SentryAutoConfiguration$HubConfiguration$SentryPerformanceWebClientConfiguration'
-o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'sentrySpanWebClientCustomizer' via factory method to bean named 'sentryHub'
+o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'org.springframework.boot.autoconfigure.web.reactive.function.client.WebClientAutoConfiguration'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'metricsWebClientCustomizer'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'org.springframework.boot.actuate.autoconfigure.metrics.web.client.WebClientMetricsConfiguration'
 o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'defaultWebClientExchangeTagsProvider'

[llaros]

@adinauer Hello, I've updated the sentry to 6.4.2 and now it seems working. Probably the first time yoy asked me something IDE did not reload the dependency or something. Now I also updated spring to 2.7.3, anyway I'm closing the issue. Sorry for that.

[adinauer]

Glad you got it to work :-)