search

codecentric / spring-boot-admin

Admin UI for administration of spring boot applications
Apache License 2.0
12.39k stars 3.08k forks source link

ApplicationContext's ObjectMapper is not used by reactive stack within InstancesProxyController (servlet-app, not webflux) #3830

Open DmitryTen opened 1 week ago

DmitryTen commented 1 week ago

Spring Boot Admin Server information

Client information

Description

Well, i'm not sure whether this is a bug in spring-boot-admin, or i'm just a dumb. I've asked my question on SO: https://stackoverflow.com/questions/79180318/spring-boot-admin-how-to-configure-objectmapper-which-is-used-by-its-reactor-li , but no informative answers or views appeared yet. If i'm wrong coming here, take my apologises please.

So, i'm trying to setup spring-boot-admin into my kubernetes, i've started with https://github.com/codecentric/spring-boot-admin-runtime-playground, and it worked quite well: it sees my app, and i'm able to view 'Insights' page, 'Logging' page, while other don't work yet. Right now i'm struggling with 'JVM->Thread dump', and i ran into issues with jackson objectMapper.

The thing is that my client-services use the specific config:

        objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);

therefore when i request '/actuator/threaddump' in spring-boot-admin, it fails (because spring-boot-admin requests my service, and it returns not valid json) with error below (the error is quite long, so i've shortened it):

    2024-11-12 07:37:52.425 DEBUG 1 --- [nio-8080-exec-3] o.apache.coyote.http11.Http11Processor   : Error state [CLOSE_NOW] reported while processing request
    org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.core.codec.DecodingException: JSON decoding error: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.ST
    ART_OBJECT`); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
     at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
            at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
            at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) ~[tomcat-embed-core-9.0.82.jar!/:na]
            at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.82.jar!/:na]
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.82.jar!/:na]
...
...
Caused by: org.springframework.core.codec.DecodingException: JSON decoding error: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputExcep
tion: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
 at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
        at org.springframework.http.codec.json.AbstractJackson2Decoder.processException(AbstractJackson2Decoder.java:242) ~[spring-web-5.3.30.jar!/:5.3.30]
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        *__checkpoint <E2><87><A2> Body from GET http://10.0.3.223:8081/actuator/threaddump [DefaultClientResponse]
Original Stack Trace:
                at org.springframework.http.codec.json.AbstractJackson2Decoder.processException(AbstractJackson2Decoder.java:242) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.http.codec.json.AbstractJackson2Decoder.decode(AbstractJackson2Decoder.java:198) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.http.codec.json.AbstractJackson2Decoder.lambda$decodeToMono$1(AbstractJackson2Decoder.java:179) ~[spring-web-5.3.30.jar!/:5.3.30]
                at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.4.33.jar!/:3.4.33]
...
...
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.Mono.block(Mono.java:1742) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at de.codecentric.boot.admin.server.web.servlet.InstancesProxyController.instanceProxy(InstancesProxyController.java:125) ~[spring-boot-admin-server-2.7.16.jar!/:2.7.16]
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) ~[na:na]
                at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
...
...
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
 at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) ~[jackson-databind-2.13.5.jar!/:2.13.5]
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1741) ~[jackson-databind-2.13.5.jar!/:2.13.5]
...

At first glance it seems to be an easy issue: just find the objectMapper and configure ACCEPT_SINGLE_VALUE_AS_ARRAY=true. BUT I JUST CAN'T FIND IT I mean i can't find the one (seems spring-boot-admin have several ones), which responsible for deserializing of requests to other services.

What i tried:

/**
 * this one actually is somehow used by spring-boot-admin, since if i configure
 * it wrong the whole app stops working properly,
 * but doesn't help with my issue
 * */   
    @Bean
    public ObjectMapper objectMapper() {
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY)
    ...

/**also tried this*/
@Bean
public Jackson2ObjectMapperBuilder jackson2ObjectMapperBuilder() {
    return new Jackson2ObjectMapperBuilder()
            .featuresToEnable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
}

/**also tried this*/
@Bean
Jackson2ObjectMapperBuilderCustomizer jackson2ObjectMapperBuilderCustomizer(){
    return jacksonObjectMapperBuilder -> {
    jacksonObjectMapperBuilder.featuresToEnable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
    };
}

/**
 * also, from the error stacktrace, i've found that it uses org.springframework.http.codec.json.AbstractJackson2Decoder
 * which is actually used by reactive WebFlux library (in my case my spring-boot-admin works in classic: servlet mode, non-reactive)
 * therefore i've tried the below, but also without success, it doesn't use it
 * */
@Bean
Jackson2JsonDecoder jackson2JsonDecoder(ObjectMapper mapper){
    mapper.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
    return new Jackson2JsonDecoder(mapper)

So, as i understand the issue, it's the following: the spring-boot-admin class de.codecentric.boot.admin.server.web.servlet.InstancesProxyController uses reactive stack (Flux,Mono,etc) even for Servlet-application, which uses separated from other app objectMapper, which seems to be not that easy to reconfigure (if possible)

Am i doing something wrong?

configs i use, pom.xml:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
  <version>2.7.17</version>
</dependency>

<!-- Spring Boot Admin -->
<dependency>
  <groupId>de.codecentric</groupId>
  <artifactId>spring-boot-admin-starter-server</artifactId>
  <version>2.7.16</version>
</dependency>

application.yml

server:
  port: 8080
  max-http-header-size: 65536
  forward-headers-strategy: none
spring:
  application: # Application-Infos for the Info-Actuator
    name: "@pom.artifactId@"
  cloud:
    kubernetes:
      discovery:
        # set this to false if running namespaced
        all-namespaces: false
  # Spring Boot Admin
  boot:
    admin:
      ui:
        public-url: https://hello-world.net/spring-boot-admin
        # public-url: ${SPRING_BOOT_ADMIN_UI_PUBLIC_URL:http://localhost:8080}
        title: ${SPRING_BOOT_ADMIN_UI_TITLE:Spring Boot Admin}
        brand: <img src="assets/img/icon-spring-boot-admin.svg"><span>${SPRING_BOOT_ADMIN_UI_TITLE:Spring Boot Admin}</span>
      discovery: # Filter discovery to tagged services
        instances-metadata:
          spring-boot-admin: true # is added as annotation in service.yaml in helm chart

management: # Actuator Configuration
  server:
    port: 8081
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint: # Health-Actuator
    health:
      show-details: always
      probes:
        enabled: true
        add-additional-paths: true
    env: # Environment-Actuator
      show-values: always # caution: can reveal passwords
    configprops: # Configuration-Actuator
      show-values: always # caution: can reveal passwords
  info: # Info-Actuator
    java:
      enabled: true
    os:
      enabled: true
    build:
      enabled: true
    env:
      enabled: true
    git:
      enabled: true
info: # Application-Infos for the Info-Actuator
  group: "@pom.groupId@"
  artifact: "@pom.artifactId@"
  description: "@pom.description@"
  version: "@pom.version@"
  spring-boot: "@pom.parent.version@"
  spring-boot-admin: "@spring-boot-admin.version@"
  spring-cloud: "@spring-cloud.version@"
  # Tags for the Spring Boot Admin UI
  tags:
    spring-boot: "@pom.parent.version@"
    spring-boot-admin: "@spring-boot-admin.version@"
    spring-cloud: "@spring-cloud.version@"
logging: # Logging-File for the Logfile-Actuator
  file:
    name: "spring-boot-admin.log"
  level:
    root: DEBUG
    org.apache.coyote: TRACE
    org.springframework.web: DEBUG
erikpetzold commented 1 week ago

Hi @DmitryTen

ok, so lets sort out some things

  1. the json for the actuator endpoints is provided by your services
  2. spring boot admin just passes through the data it receives from the actuator endpoints from your services
  3. so serialization in your services is relevant
  4. spring boot admin backend/ui expects the json in the default format
  5. the actuator endpoints in your services use the default object mapper (in Spring Boot 2)
  6. so if you change the default object mapper, the json of the actuator endpoints is changed and Spring Boot Admin cannot handle it

So the best solution would be to not modify the default ObjectMapper and provide a separate one for your own business services. Or update to Spring Boot 3, where they introduced a separate ObjectMapper for the actuator endpoints.

There have been already many issues regarding that topic, see for example https://github.com/codecentric/spring-boot-admin/issues/751 and the issues referenced there.

DmitryTen commented 1 week ago

Thank you for your answer @erikpetzold !

So, as i understand in my case i need to reconfigure my client ObjectMapper (yes we modified default one, therefore this would be not that easy) OR switch to spring 3.x.x, since then actuator endpoints could be linked to separated (not default) ObjectMapper (also not easy change). Therefore let me ask another, maybe stupid question. Spring-boot-admin works in two modes: servlet and webflux, in servlet-mode seems it uses non-default ObjectMapper. What if i'll switch to web-flux, will it take the default-one? (I know this is a silly assumption, but who knows)

Thank you

erikpetzold commented 1 week ago

As I said before, the SBA backend will just pass-through the data it received from the services. The ObjectMapper in SBA is not relevant for your problem.

The actuator json is generated in your services. So you have to adjust that in your services.

One solution would be to have 2 ObjectMappters in your services: the default one which would be used by actuators and an own used in your business functionality.

This can also be done without explicitly talking about ObjectMappers, but instead using 2 MessageConverters being used by the web layer. The Converters are then bound to mime-types. See https://github.com/codecentric/spring-boot-admin/issues/751#issuecomment-575999785 on the issue I already mentioned.

DmitryTen commented 1 week ago

Okay, thanks a lot! Your advice helped, i've configured actuator's ObjectMapper in my client to be default. Finally it passed, and jackson not a problem anymore. But other reactor-related issue appeared, see below:

        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
        at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99) ~[spring-core-5.3.30.jar!/:5.3.30]
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        *__checkpoint <E2><87><A2> Body from GET http://10.0.3.43:8081/actuator/threaddump [DefaultClientResponse]
Original Stack Trace:
                at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99) ~[spring-core-5.3.30.jar!/:5.3.30]
                at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:92) ~[spring-core-5.3.30.jar!/:5.3.30]
                at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:58) ~[spring-core-5.3.30.jar!/:5.3.30]
                at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:119) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.FluxMap$MapSubscriber.

Fast googling of this issue shows, that spring.codec.max-in-memory-size config should help. I've tried this, but no luck. Which is not surprising, while my app works in servlet-mode, while the config seems to be webFlux related. Is there a way to cope with that?

erikpetzold commented 4 days ago
  1. Is the Exception thwrown in SBA Server or in your client service?

Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:

this looks like reactive stack (webFlux)

  1. where did you set the property? In SBA Server or in your client service?
DmitryTen commented 4 days ago

I'm talking about SBA server, i've tried that config on SBA server, but no luck let me place the full log:

2024-11-15 13:49:32.123 DEBUG 1 --- [nio-8080-exec-6] o.apache.coyote.http11.Http11Processor   : Error state [CLOSE_NOW] reported while processing request

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at de.codecentric.boot.admin.server.ui.web.servlet.HomepageForwardingFilter.doFilter(HomepageForwardingFilter.java:78) ~[spring-boot-admin-server-ui-2.7.16.jar!/:2.7.16]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.7.17.jar!/:2.7.17]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.82.jar!/:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
        at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99) ~[spring-core-5.3.30.jar!/:5.3.30]
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        *__checkpoint <E2><87><A2> Body from GET http://10.0.3.43:8081/actuator/threaddump [DefaultClientResponse]
Original Stack Trace:
                at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99) ~[spring-core-5.3.30.jar!/:5.3.30]
                at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:92) ~[spring-core-5.3.30.jar!/:5.3.30]
                at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:58) ~[spring-core-5.3.30.jar!/:5.3.30]
                at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:119) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:377) ~[reactor-netty-core-1.0.38.jar!/:1.0.38]
                at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:417) ~[reactor-netty-core-1.0.38.jar!/:1.0.38]
                at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:775) ~[reactor-netty-http-1.0.38.jar!/:1.0.38]
                at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114) ~[reactor-netty-core-1.0.38.jar!/:1.0.38]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) ~[netty-transport-classes-epoll-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) ~[netty-transport-classes-epoll-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
                at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
                at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at reactor.core.publisher.Mono.block(Mono.java:1742) ~[reactor-core-3.4.33.jar!/:3.4.33]
                at de.codecentric.boot.admin.server.web.servlet.InstancesProxyController.instanceProxy(InstancesProxyController.java:125) ~[spring-boot-admin-server-2.7.16.jar!/:2.7.16]
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) ~[na:na]
                at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
                at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at de.codecentric.boot.admin.server.ui.web.servlet.HomepageForwardingFilter.doFilter(HomepageForwardingFilter.java:78) ~[spring-boot-admin-server-ui-2.7.16.jar!/:2.7.16]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.7.17.jar!/:2.7.17]
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.30.jar!/:5.3.30]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.82.jar!/:na]
                at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
DmitryTen commented 2 days ago

i made some debug, and found that while my spring.codec.max-in-memory-size is taken into consideration, it's not used during requesting to clients. But what is taken: de.codecentric.boot.admin.server.web.client.LegacyEndpointConverters#convertUsing method

private static <S, T> Function<Flux<DataBuffer>, Flux<DataBuffer>> convertUsing(
        ParameterizedTypeReference<S> sourceType, ParameterizedTypeReference<T> targetType,
        Function<S, T> converterFn) {
    return (input) -> DECODER.decodeToMono(input, ResolvableType.forType(sourceType), null, null)
            .map((body) -> converterFn.apply((S) body)).flatMapMany((output) -> ENCODER.encode(Mono.just(output),
                    new DefaultDataBufferFactory(), ResolvableType.forType(targetType), null, null));

below the stack it uses org.springframework.http.codec.json.AbstractJackson2Decoder#decodeToMono

@Override
public Mono<Object> decodeToMono(Publisher<DataBuffer> input, ResolvableType elementType,
        @Nullable MimeType mimeType, @Nullable Map<String, Object> hints) {

    return DataBufferUtils.join(input, this.maxInMemorySize)
            .flatMap(dataBuffer -> Mono.justOrEmpty(decode(dataBuffer, elementType, mimeType, hints)));
}

which is by default uses 

private int maxInMemorySize = 256 * 1024;

and not the value from spring.codec.max-in-memory-size

DmitryTen commented 1 day ago

@erikpetzold could you please check the above comment? my current errors seems to be related to this LegacyEndpointConverters

static {
    ObjectMapper om = Jackson2ObjectMapperBuilder.json()
            .featuresToDisable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS).build();
    DECODER = new Jackson2JsonDecoder(om);
    ENCODER = new Jackson2JsonEncoder(om);
}

is it supposed to be so? Is there a way to cope with that? Only reflection?

erikpetzold commented 8 hours ago

The LegacyEndpointConverter should only be relevant for Spring Boot 1 applications.

With Spring Boot 2, the actuator has its own mime type. Probably you changed the response type to be application/json, which triggers the LegacyEndpointConverter. See #3368

You should keep the original actuator mime type.

-> In general, do not change the actuator endpoints