search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.86k stars 2.44k forks source link

Spring Cloud Hoxton.SR5 issue with Eureka and WebFlux: Exceeded limit on max bytes to buffer #3819

Closed vasiliy-sarzhynskyi closed 4 years ago

vasiliy-sarzhynskyi commented 4 years ago

We have Spring Cloud application using Eureka as service discovery, Config Service and WebFlux. Service works fine with Spring Cloud Hoxton.SR4, but fails with Hoxton.SR5 on app start during fetching data from Eureka. If Eureka has a small number of registered apps (e.g. 30), it works fine, but in case it has around 250 services, it fails with the following exception

DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144 Error has been observed at the following site(s): checkpoint ⇢ Body from GET http://eureka.xxx.com:8080/eureka/apps/?regions=us-east-1.

If we remove WebFlux from classpath, it also works fine even with Hoxton.SR5. I provided reproducible issue on github (but in order to reproduce, we need to connect to Eureka having several hundreds of registered apps). property spring.codec.max-in-memory-size doesn't help here.

Spring Boot version 2.3.1.RELEASE

Stack trace:

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
    at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ Body from GET http://eureka.xxx.com:8080/eureka/apps/?regions=us-east-1 [DefaultClientResponse]
Stack trace:
        at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:94) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:124) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:330) ~[reactor-netty-0.9.8.RELEASE.jar:0.9.8.RELEASE]
        at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:353) ~[reactor-netty-0.9.8.RELEASE.jar:0.9.8.RELEASE]
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:635) ~[reactor-netty-0.9.8.RELEASE.jar:0.9.8.RELEASE]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) ~[reactor-netty-0.9.8.RELEASE.jar:0.9.8.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) ~[netty-codec-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425) ~[netty-codec-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.50.Final.jar:4.1.50.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.50.Final.jar:4.1.50.Final]
        at java.base/java.lang.Thread.run(Thread.java:835) ~[na:na]
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at reactor.core.publisher.Mono.block(Mono.java:1679) ~[reactor-core-3.3.6.RELEASE.jar:3.3.6.RELEASE]
        at org.springframework.cloud.netflix.eureka.http.WebClientEurekaHttpClient.getApplicationsInternal(WebClientEurekaHttpClient.java:143) ~[spring-cloud-netflix-eureka-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.netflix.eureka.http.WebClientEurekaHttpClient.getApplications(WebClientEurekaHttpClient.java:124) ~[spring-cloud-netflix-eureka-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.netflix.eureka.config.EurekaConfigServerBootstrapConfiguration.lambda$eurekaConfigServerInstanceProvider$0(EurekaConfigServerBootstrapConfiguration.java:112) ~[spring-cloud-netflix-eureka-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.config.client.ConfigServerInstanceProvider.getConfigServerInstances(ConfigServerInstanceProvider.java:50) ~[spring-cloud-config-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.config.client.DiscoveryClientConfigServiceBootstrapConfiguration$HeartbeatListener.refresh(DiscoveryClientConfigServiceBootstrapConfiguration.java:120) ~[spring-cloud-config-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.config.client.DiscoveryClientConfigServiceBootstrapConfiguration$HeartbeatListener.startup(DiscoveryClientConfigServiceBootstrapConfiguration.java:106) ~[spring-cloud-config-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.config.client.DiscoveryClientConfigServiceBootstrapConfiguration$HeartbeatListener.onApplicationEvent(DiscoveryClientConfigServiceBootstrapConfiguration.java:98) ~[spring-cloud-config-client-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:404) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:361) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:898) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:554) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:212) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:117) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:74) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
        at com.vsarzhynskyi.springcloud.hoxtonsr5.ReproducibleIssueApplication.main(ReproducibleIssueApplication.java:10) ~[classes/:na]
spencergibb commented 4 years ago

Thanks for the issue. This happens when spring.cloud.config.discovery.enabled=true and when webflux is on the classpath. There are two issue. One, the webclient functionality should be opt in. I also need to connect the codec configuration with the WebClient that is built for config discovery. For now, you'll need to roll back to Hoxton.SR4