Make it easier to configure DefaultPartHttpMessageReader's max headers size

[lonelyleaf]

Spring boot version: 2.4.3

I'm using webflux and upload some file, when upload file a little bit larger, spring reports:

org.springframework.core.io.buffer.DataBufferLimitException: Part headers exceeded the memory usage limit of 8192 bytes
 at org.springframework.http.codec.multipart.MultipartParser$HeadersState.onNext(MultipartParser.java:360) ~[spring-web-5.3.3.jar!/:5.3.3]
 Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
 |_ checkpoint ⇢ Handler cn.waterstrategy.rest.LayerController#upload(Flux, String) [DispatcherHandler]
Stack trace:
         at org.springframework.http.codec.multipart.MultipartParser$HeadersState.onNext(MultipartParser.java:360) ~[spring-web-5.3.3.jar!/:5.3.3]
         at org.springframework.http.codec.multipart.MultipartParser.hookOnNext(MultipartParser.java:104) ~[spring-web-5.3.3.jar!/:5.3.3]
         at org.springframework.http.codec.multipart.MultipartParser.hookOnNext(MultipartParser.java:46) ~[spring-web-5.3.3.jar!/:5.3.3]
         at reactor.core.publisher.BaseSubscriber.onNext(BaseSubscriber.java:160) ~[reactor-core-3.4.2.jar!/:3.4.2]
         at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.2.jar!/:3.4.2]
         at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) ~[reactor-core-3.4.2.jar!/:3.4.2]
         at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.2.jar!/:3.4.2]
         at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:267) ~[reactor-netty-core-1.0.3.jar!/:1.0.3]
         at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:377) ~[reactor-netty-core-1.0.3.jar!/:1.0.3]
         at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:381) ~[reactor-netty-core-1.0.3.jar!/:1.0.3]
         at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:550) ~[reactor-netty-http-1.0.3.jar!/:1.0.3]
         at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.3.jar!/:1.0.3]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:253) ~[reactor-netty-http-1.0.3.jar!/:1.0.3]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
         at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:388) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
         at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar!/:4.1.58.Final]
         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar!/:4.1.58.Final]
         at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

my request http file is like:

POST {{host}}/gis/layer/upload
Content-Type: multipart/form-data; boundary=WebAppBoundary
Authorization: {{jwt}}

--WebAppBoundary
Content-Disposition: form-data; name="file"; filename="without_xy.dbf"

< shapefile/withoutXy/without_xy.dbf
--WebAppBoundary--

--WebAppBoundary
Content-Disposition: form-data; name="file"; filename="without_xy.prj"

< shapefile/withoutXy/without_xy.prj
--WebAppBoundary--

--WebAppBoundary
Content-Disposition: form-data; name="file"; filename="without_xy.shp"

< shapefile/withoutXy/without_xy.shp
--WebAppBoundary--

--WebAppBoundary
Content-Disposition: form-data; name="file"; filename="without_xy.shx"

< shapefile/withoutXy/without_xy.shx
--WebAppBoundary--

--WebAppBoundary
Content-Disposition: form-data; name="file"; filename="without_xy.sld"

< shapefile/withoutXy/without_xy.sld
--WebAppBoundary--

All of those filee combine is less than 500Kb,but when I use smaller file (about 20kb total),the controller works.

my controller code(using kotlin) is like:

    @PostMapping(value = ["/upload"], consumes = [MediaType.MULTIPART_FORM_DATA_VALUE])
    fun upload(
        @RequestBody partsFlux: Flux<Part>
    ): Mono<ShapefileUploadResp> {
        val dateStr = LocalDateTime.now().format(DATE_FORMATTER)
        val uploadCode = codeSerialService.generateCode("shapefile_upload", 12)
        val dirPath = "$dateStr/$uploadCode/"

        return partsFlux
            .filter { it is FilePart }
            .cast(FilePart::class.java)
            .collectList()
            .flatMap { parts: List<FilePart> ->
                ........
             }

Another thing is I can upload files when I'm devloping on my machine,but can't upload when deploy to server,where is wrong?

[wilkinsona]

Your request is exceeding the default header size limit in WebFlux's multipart support. In a Spring Boot application you can customize that limit using a CodecCustomizer:

@Bean
CodecCustomizer codecCustomizer() {
    return (codecs) -> codecs.defaultCodecs().configureDefaultCodec((codec) -> {
        if (codec instanceof MultipartHttpMessageReader) {
            HttpMessageReader<Part> partReader = ((MultipartHttpMessageReader) codec).getPartReader();
            if (partReader instanceof DefaultPartHttpMessageReader) {
                ((DefaultPartHttpMessageReader) partReader).setMaxHeadersSize(48 * 1024);
            }
        }
    });
}

I think we should explore making this easier, perhaps via an application property, or at least documenting how to do it. We can use this issue to track that work.

If you don't believe that the header size limit should have been exceeded in this case, please open a Spring Framework issue.

[lonelyleaf]

@wilkinsona Thank you , I solved my problem by your reply,though it dosen't work directly.Need to change a little bit in spring boot webflux.

@Configuration
@EnableWebFlux
public class WebConfig implements WebFluxConfigurer {

    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {

        configurer.defaultCodecs().configureDefaultCodec((codec) -> {
            if (codec instanceof MultipartHttpMessageReader) {
                HttpMessageReader<Part> partReader = ((MultipartHttpMessageReader) codec).getPartReader();
                if (partReader instanceof DefaultPartHttpMessageReader) {
                    ((DefaultPartHttpMessageReader) partReader).setMaxHeadersSize(128 * 1024);
                }
            }
        });

    }

}

[wilkinsona]

That’s strange as it worked for me. Note that by using @EnableWebFlux you are disabling all of Boot’s auto-configuration of WebFlux, including applying any CodecCustomizer beans. Did you already have @EnableWebFlux somewhere in your app?

[lonelyleaf]

@wilkinsona I did use @EnableWebFlux in a WebFluxConfigurer class like above I posted.

[wilkinsona]

That explains why the CodecCustomizer bean had no effect. In a Boot app, the equivalent of @EnableWebFlux will be auto-configured for you and any WebFluxConfigurer beans will therefore be picked up automatically. Please see the relevant section of the reference documentation for a few more details.

[lonelyleaf]

@wilkinsona Many thanks for your explain 😀