Make metric export filters applicable only to Dynatrace and not Spring Boot Admin (Spring Actuator)

[yleguern]

Description:

I am currently working on a project where we utilize both Dynatrace and Spring Boot Admin (via Spring Actuator endpoints) for metric collection and monitoring. We have implemented filters to export specific metrics to Dynatrace. However, these filters are also being applied to the metrics sent to Spring Boot Admin, which is not the desired behavior for our use case.

Request:

We would like to request a feature or configuration option that allows us to apply metric export filters exclusively to Dynatrace, without impacting the metrics exported to Spring Boot Admin (Actuator). This would help us maintain a complete set of metrics for internal monitoring via Spring Boot Admin, while selectively exporting metrics to Dynatrace.

Use Case:

• Dynatrace: We need to export a specific subset of metrics for external monitoring. The objective is to limit the cost associated with the use of metrics that are chargeable in Dynatrace, especially for metrics that Dynatrace already collects natively (such as CPU and memory usage).
• Spring Boot Admin (Actuator): We require a full set of metrics for comprehensive internal monitoring.

Current Behavior:

• Filters are applied globally, affecting both Dynatrace and Spring Boot Admin.

Desired Behavior:

• Filters should be configurable to apply only to the Dynatrace export, leaving the Spring Boot Admin metrics unaffected.

[jonatan-ivanov]

It seems you are registering your MeterFilters on the CompositeMeterRegistry instead of DynatraceMeterRegistry. The MeterFilter will be effective on the registry you set it on:

• DynatraceMeterRegistry: Dynatrace only
• SimpleMeterRegistry: The in-memory registry only (/actuator/metrics uses this)
• CompositeMeterRegistry: both

Also, we don't recommend using /actuator/metrics for production purposes, SimpleMeterRegistry was created for testing and metrics-troubleshooting purposes.

[yleguern]

Thank you for your response. However, I would like to clarify that we are not using the CompositeMeterRegistry; we are only using the DynatraceMeterRegistry. The issue we are facing is that the filters applied to the DynatraceMeterRegistry are also affecting the metrics received (in our case not received) by Spring Boot Admin via the Spring Actuator endpoints.

[jonatan-ivanov]

Are you 100% sure you are not using it? :)

If you use Spring Boot Actuator and you also add the micrometer-registry-duynatrace dependency, you are using the composite by default: the actuator endpoint uses SimpleMeterRegistry and you also need DynatraceMeterRegistry; Boot will create and configure a CompositeMeterRegistry for you that contains them.

You can verify this by looking at the /actuator/beans endpoint and search for CompositeMeterRegistry. You can also inject a MeterRegistry to one of your components and either print out its class name or check it with a debugger: you should see a CompositeMeterRegistry that has two registries in it: SimpleMeterRegistry (for the /actuator/metrics endpoint) and DynatraceMeterRegistry.

When you create a MeterFilter bean, Boot will register it to the composite so it will affect both Dynatrace and the Actuator endpoint, this is intentional, if you don't want this behavior (see my previous comment) you need to set things up "manually", for example:

@Bean
MeterRegistryCustomizer<DynatraceMeterRegistry> dynatraceMeterRegistryCustomizer() {
    return (registry) -> registry.config().meterFilter(...); // this will only affect Dynatrace
}

@Bean
MeterFilter commonMeterFilter() {
    return ...; // this will effect both Dynatrace and Actuator's metrics endpoint since filters are registered to the composite
}

// No custom config for `SimpleMeterRegistry`, we are not customizing anything

But there are multiple ways to solve this, you can also create a DynatraceMeterRegistry bean and set the filters there or I guess a BeanPostProcessor can also work.

[yleguern]

Yes I confirm I'm not using CompositeMeterRegistry (only DynatraceMeterRegistry) :

(this is the spring-boot-admin view of /actuator/metrics endpoint).

Besides the debug output says:

SimpleMetricsExportAutoConfiguration:
   Did not match:
      - @ConditionalOnMissingBean (types: io.micrometer.core.instrument.MeterRegistry; SearchStrategy: all) found beans of type 'io.micrometer.core.instrument.MeterRegistry' dynatraceMeterRegistry (OnBeanCondition)
   Matched:
      - @ConditionalOnEnabledMetricsExport management.simple.metrics.export.enabled is true (OnMetricsExportEnabledCondition)

CompositeMeterRegistryConfiguration:
   Did not match:
      - NoneNestedConditions 1 matched 1 did not; NestedCondition on CompositeMeterRegistryConfiguration.MultipleNonPrimaryMeterRegistriesCondition.SingleInjectableMeterRegistry @ConditionalOnSingleCandidate (types: io.micrometer.core.instrument.MeterRegistry; SearchStrategy: all) found a single bean 'dynatraceMeterRegistry'; NestedCondition on CompositeMeterRegistryConfiguration.MultipleNonPrimaryMeterRegistriesCondition.NoMeterRegistryCondition @ConditionalOnMissingBean (types: io.micrometer.core.instrument.MeterRegistry; SearchStrategy: all) found beans of type 'io.micrometer.core.instrument.MeterRegistry' dynatraceMeterRegistry (CompositeMeterRegistryConfiguration.MultipleNonPrimaryMeterRegistriesCondition)

From what I understand, I can only have one configuration or the other, unless I manually create every bean, which is not very practical (see SimpleMetricsExportAutoConfiguration and DynatraceMetricsExportAutoConfiguration)

[jonatan-ivanov]

Yes I confirm I'm not using CompositeMeterRegistry (only DynatraceMeterRegistry) :

In this case I have zero idea how your application works and how do you have any data recorded both for Dynatrace and the Actuator endpoint. Could you please check what happens if you try to inject MeterRegistry into one of your component? Will you get a dynatrace or a simple meter registry or something else?

From what I understand, I can only have one configuration or the other, unless I manually create every bean, which is not very practical

I'm not sure I get this, you mean you can only have one registry? You don't need to create beans manually, you can use the MeterRegistryCustomizer as I showed you above.

[yleguern]

Hello, and thanks for your responses

In this case I have zero idea how your application works and how do you have any data recorded both for Dynatrace and the Actuator endpoint. Could you please check what happens if you try to inject MeterRegistry into one of your component? Will you get a dynatrace or a simple meter registry or something else?

There must be a misunderstanding. My point is that there is one injected MeterRegistry bean (DynatraceMeterRegistry), used to export metrics to Dynatrace and by the /actuator/metrics actuator endpoint.

If I create an application with only:

// build.gradle
plugins {
    id 'java'
    id 'org.springframework.boot' version '3.3.1'
    id 'io.spring.dependency-management' version '1.1.5'
}

group = 'ylg.meter'
version = '0.0.1-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'io.micrometer:micrometer-registry-dynatrace:1.13.2'

    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

# application.yml
management:
    endpoints.web.exposure.include: '*'
    metrics.enable:
        all: false
        system: true # to test that only those metrics are exported to Dynatrace
    dynatrace.metrics.export.enabled: true

// MeterRegistryApplication.java
@SpringBootApplication
public class MeterRegistryApplication {

    private static final Logger log = LoggerFactory.getLogger(MeterRegistryApplication.class);

    public static void main(String[] args) { SpringApplication.run(MeterRegistryApplication.class, args); }

    @Bean
    public ApplicationRunner applicationRunner(MeterRegistry meterRegistry) {
        return args -> log.info("MeterRegistry: {}", meterRegistry.getClass());
    }
}

I confirm the DynatraceMeterRegistry bean is injected:

y.m.m.MeterRegistryApplication   : MeterRegistry: class io.micrometer.dynatrace.DynatraceMeterRegistry

And if I request /actuator/metrics actuator endpoint:

curl localhost:8080/actuator/metrics
{"names":["system.cpu.count","system.cpu.usage","system.load.average.1m"]}%

Only the filtered metrics are exposed which make sense because the MetricsEndpoint is using the injected MeterRegistry.

[jonatan-ivanov]

I did debug a Spring Boot app and got the same behavior: the MetricsEndpoint bean used the same registry that my own component did (I used Prometheus but in your case: Dynatrace). This was unexpected to me, sorry for the confusion and for the back and forth, I thought Boot will create a Composite registry in this case but it seems it does not.

However it did create a composite when I created a SimpleMeterRegistry bean manually (expected):

@Bean
SimpleMeterRegistry simpleMeterRegistry() {
    return new SimpleMeterRegistry();
}

On the other hand, since MetricsEndpoint needs a MeterRegistry to be injected, both the MetricsEndpoint and my own component used the same registry. The way I was able to make MetricsEndpoint to use a different MeterRegistry (SimpleMeterRegistry) than my components (composite) was creating the MetricsEndpoint bean manually and tell it that I want to use SimpleMeterRegistry there:

@Bean
SimpleMeterRegistry simpleMeterRegistry() {
    return new SimpleMeterRegistry();
}

@Bean
MetricsEndpoint metricsEndpoint(SimpleMeterRegistry simpleMeterRegistry) {
    return new MetricsEndpoint(simpleMeterRegistry);
}

I'm not sure if this works for you but if does not, could you pease create a new issue in the Spring Boot issue tracker? Please feel free to cc me in the issue. Micrometer already let you do what you want, it's how things wired together in your app prevents you doing what you need.

[yleguern]

Hello @jonatan-ivanov and thanks for your help

When creating manually a MetricsEndpoint with its SimpleMeterRegistry and a custom MeterFilter for DynatraceMeterRegistry it's working as expected.

It would be nice to have a default MeterFilter for Dynatrace like the one used by the SimpleMeterRegistry (cf PropertiesMeterFilter.java). What do you think? Is it worth submitting a pull request?

Here is a full working example (Dynatrace calls are mocked):

@SpringBootApplication
public class MeterRegistryApplication {

    private static final Logger log = LoggerFactory.getLogger(MeterRegistryApplication.class);

    public static void main(String[] args) { SpringApplication.run(MeterRegistryApplication.class, args); }

    @Bean
    public ApplicationRunner applicationRunner(MeterRegistry meterRegistry) {
        return args -> log.info("MeterRegistry: {}", meterRegistry.getClass());
    }

    @Bean
    SimpleMeterRegistry simpleMeterRegistry() {
        return new SimpleMeterRegistry();
    }

    @Bean
    MetricsEndpoint metricsEndpoint(SimpleMeterRegistry simpleMeterRegistry) {
        return new MetricsEndpoint(simpleMeterRegistry);
    }

    @Bean
    MeterRegistryCustomizer<DynatraceMeterRegistry> dynatraceMeterRegistryCustomizer() {
        return registry -> registry.config().meterFilter(new MeterFilter() {
            @Override
            public MeterFilterReply accept(Meter.Id id) {
                return id.getName().startsWith("jvm.memory") ? MeterFilterReply.ACCEPT : MeterFilterReply.DENY;
            }
        });
    }

    @Bean
    RouterFunction<ServerResponse> mockDynatraceIngestion() {
        return RouterFunctions.route().POST("/metrics/ingest", req -> {
                    log.info("--- Received by DYNATRACE:\n{}", req.body(String.class));
                    return accepted().body("{ \"error\": null, \"linesOk\": 1, \"linesInvalid\": 0}");
                })
                .build();
    }
}

# application.yml
server.port: 8080
management:
    endpoints.web.exposure.include: '*'
    metrics.enable:
        all: false
        system: true
    dynatrace.metrics.export:
        enabled: true
        uri: http://localhost:${server.port}/metrics/ingest
        step: 5s

[jonatan-ivanov]

I'm glad setting it up that way worked, thanks for confirming it!

I'm not sure I understand this part:

It would be nice to have a default MeterFilter for Dynatrace like the one used by the SimpleMeterRegistry (cf PropertiesMeterFilter.java). What do you think? Is it worth submitting a pull request?

As far as I know, PropertiesMeterFilter is a general MeterFilter in Boot, it can be registered into any registry not only SimpleMeterRegistry. You mean having something in the meter registry config where you can define properties that would cause accepting and denying meters? If so, I think that's what MeterFilter is for but maybe it worth opening an issue to Boot to have a property like this per registry?