The spanFactoryInspectorPostProcessor breaks prometheus metric binding

[hatzlj]

Basic information

• Inspector Axon Connector version: 0.1.4
• JDK version: 1.11 (eclipse-temurin)
• Axon Framework version (if applicable): 4.7.4 (Spring boot 2.7.11)

Steps to reproduce

• a spring boot project 2.7.11 with io.micrometer:micrometer-registry-prometheus and org.springframework.boot:spring-boot-starter-actuator (which includes the spring-boot-actuator-autoconfigure dependency)
• include the io.axoniq.inspector:inspector-axon-spring-boot-starter:0.1.4 dependency to your project

Expected behaviour

• spring boot default metrics (jvm, ...) are exported via micrometer-registry-prometheus

Actual behaviour

• spring boot default metrics (jvm, ...) are missing in the actuator/prometheus endpoint
• it seems that the spanFactoryInspectorPostProcessor bean configured in AxonInspectorAutoConfiguration triggers early initialization of the PrometheusMeterRegistry which leads to certain MeterBinders not being picked up by the MeterRegistryPostProcessor. When debugging the initialization of the application context and breakpointing on initialization of the PrometheusMeterRegistry i can see the following order of post processors in org.springframework.context.support.PostProcessorRegistrationDelegate#registerBeanPostProcessors(org.springframework.beans.factory.config.ConfigurableListableBeanFactory, org.springframework.context.support.AbstractApplicationContext):

nonOrderedPostProcessorNames = {ArrayList@13759}  size = 10
 0 = "hypermediaRepresentionModelProcessorConfigurator"
 1 = "hypermediaRestTemplateBeanPostProcessor"
 2 = "webServerFactoryCustomizerBeanPostProcessor"
 3 = "errorPageRegistrarBeanPostProcessor"
 4 = "spanFactoryInspectorPostProcessor"
 5 = "meterRegistryPostProcessor"
 6 = "healthEndpointGroupsBeanPostProcessor"
 7 = "preventDataSourceCloseBeanPostProcessor"
 8 = "metricsRepositoryMethodInvocationListenerBeanPostProcessor"
 9 = "projectingArgumentResolverBeanPostProcessor"

whereas as far as I understand https://github.com/micrometer-metrics/micrometer/issues/513#issuecomment-378755211 the PrometheusMeterRegistry should only be initialized after the meterRegistryPostProcessor.

[hatzlj]

I just encountered that in one of our projects and had no time to reproduce it on a minimal project yet. However I just wanted to document the issue here in case someone stumbles upon it, since it took me quite some time to track down why my metrics are missing.

I'm not a spring auto-configuration specialist but I guess that this might be fixable by some spring @AutoConfiguration(after=...) or @Order magic?

[CodeDrivenMitch]

Hello @hatzlj, thanks for reporting the issue! We have not encountered this in our own applications, hence why it's not picked up. Reading some more about this, we might have implemented the wrong BeanPostProcessor method, so I will correct that. I will also make the BeanPostProcessor implement PriorityOrdered, so it's initialized after.

If I release a snapshot, would you be willing to test it out?

[CodeDrivenMitch]

Apparently, PriorityOrdered post processors are always before non-prioritized ones. Since MeterRegistryPostProcessor doesn't implement it, it will only make matters worse. As suggested, I will make the autoconfiguration initialize after the MetricsAutoConfiguration of micrometer.

[CodeDrivenMitch]

I have it reproduced, and the fix with autoconfigure-ordering does not work, unfortunately. The issue occurs, for me, when running in a docker container. When running from IDEA, it totally works

[CodeDrivenMitch]

I found out why; it's not the autoconfigure ordering causing this; it's the required Configuration parameter to the method, causing initialization of the metrics before this point. I will look into a solution for this, but this might take some time

[hatzlj]

Hey @Morlack! Thanks for having a look on this (and fixing it) so quick. Yes, also here it only happens within a docker container. I would be curious if you found out why it works in the IDE but not within a container.

[CodeDrivenMitch]

No worries, an issue of this magnitude should be fixed fast. To be honest, I have no idea why it differs between IntelliJ and Docker (or perhaps running the Spring Boot jar). It might just be the classpath scanning between a dynamic classpath and the uberjar works similarly, but just differently enough to botch the ordering. In order to keep my sanity, I will just keep that as a fact! ;)

[CodeDrivenMitch]

The fix is available: https://repo1.maven.org/maven2/io/axoniq/inspector/inspector-axon/0.1.5/