micronaut-projects / micronaut-core

Micronaut Application Framework
http://micronaut.io
Apache License 2.0
6.04k stars 1.06k forks source link

NettyHttpRequest.formRouteCompleter - java.util.NoSuchElementException: No value present #11049

Open goeh opened 1 month ago

goeh commented 1 month ago

Expected Behavior

After upgrade from Micronaut 3.x to 4.5.1 we see "random" No value present errors from a service that accepts multipart file uploads.

We also see that sometimes requests seems unauthenticated (injected Authentication parameter is null) but request from the same user before and after the failing request (few milliseconds) are authenticated and works ok. Seems the JWT token (cookie) get lost in some requests.

Actual Behaviour

Exceptions related to missing auth tokens.

Steps To Reproduce

It seems very random and I will try my best to create a reproducer. We've been debugging this for 2 weeks without success. But right now I just wanted to create a placeholder issue for the stacktrace. I'll add more info as debugging progresses.

java.util.NoSuchElementException: No value present
    at java.base/java.util.Optional.get(Unknown Source)
    at io.micronaut.http.server.netty.NettyHttpRequest.formRouteCompleter(NettyHttpRequest.java:230)
    at io.micronaut.http.server.netty.binders.NettyPublisherPartUploadBinder.bindForNettyRequest(NettyPublisherPartUploadBinder.java:86)
    at io.micronaut.http.server.netty.binders.NettyRequestArgumentBinder.bind(NettyRequestArgumentBinder.java:39)
    at io.micronaut.http.server.netty.binders.NettyPartUploadAnnotationBinder.bind(NettyPartUploadAnnotationBinder.java:67)
    at io.micronaut.http.server.netty.binders.NettyPartUploadAnnotationBinder.bind(NettyPartUploadAnnotationBinder.java:44)
    at io.micronaut.web.router.AbstractRouteMatch.fulfillValue(AbstractRouteMatch.java:366)
    at io.micronaut.web.router.AbstractRouteMatch.fulfillBeforeFilters(AbstractRouteMatch.java:322)
    at io.micronaut.web.router.DefaultUriRouteMatch.fulfillBeforeFilters(DefaultUriRouteMatch.java:39)
    at io.micronaut.http.server.binding.RequestArgumentSatisfier.fulfillArgumentRequirementsBeforeFilters(RequestArgumentSatisfier.java:57)
    at io.micronaut.http.server.netty.NettyRequestArgumentSatisfier.fulfillArgumentRequirementsBeforeFilters(NettyRequestArgumentSatisfier.java:50)
    at io.micronaut.http.server.RequestLifecycle.fulfillArguments(RequestLifecycle.java:513)
    at io.micronaut.http.server.netty.NettyRequestLifecycle.fulfillArguments(NettyRequestLifecycle.java:136)
    at io.micronaut.http.server.RequestLifecycle.executeRoute(RequestLifecycle.java:191)
    at io.micronaut.http.server.RequestLifecycle.lambda$normalFlow$1(RequestLifecycle.java:182)
    at io.micronaut.http.filter.FilterRunner.provideResponse(FilterRunner.java:272)
    at io.micronaut.http.filter.FilterRunner.filterRequest(FilterRunner.java:207)
    at io.micronaut.http.filter.FilterRunner.lambda$filterRequest$2(FilterRunner.java:180)
    at io.micronaut.http.filter.AroundLegacyFilter$FilterChainImpl.proceed(AroundLegacyFilter.java:122)
    at io.micronaut.security.filters.SecurityFilter.lambda$checkRules$8(SecurityFilter.java:179)
    at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46)
    at reactor.core.publisher.Flux.subscribe(Flux.java:8840)
    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:82)
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onComplete(MonoFlatMapMany.java:211)
    at reactor.core.publisher.MonoNext$NextSubscriber.onComplete(MonoNext.java:102)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onComplete(FluxConcatMapNoPrefetch.java:241)
    at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:357)
    at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:294)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerComplete(FluxConcatMapNoPrefetch.java:275)
    at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onComplete(FluxConcatMap.java:889)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:299)
    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.onComplete(FluxFilter.java:300)
    at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onComplete(FluxDefaultIfEmpty.java:139)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2573)
    at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.request(FluxDefaultIfEmpty.java:98)
    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.request(FluxFilter.java:321)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
    at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:339)
    at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108)
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribe(MonoFlatMapMany.java:141)
    at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164)
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201)
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.onNext(FluxFilter.java:247)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:129)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxSwitchMapNoPrefetch$SwitchMapInner.onNext(FluxSwitchMapNoPrefetch.java:408)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:181)
    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
    at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
    at reactor.core.publisher.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
    at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
    at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
    at io.micronaut.http.client.filters.ClientServerRequestTracingPublisher$1.lambda$onNext$1(ClientServerRequestTracingPublisher.java:60)
    at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:49)
    at io.micronaut.http.client.filters.ClientServerRequestTracingPublisher$1.onNext(ClientServerRequestTracingPublisher.java:60)
    at io.micronaut.http.client.filters.ClientServerRequestTracingPublisher$1.onNext(ClientServerRequestTracingPublisher.java:52)
    at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
    at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:880)
    at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:805)
    at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:163)
    at io.micronaut.http.client.netty.ForwardingSubscriber.onNext(ForwardingSubscriber.java:45)
    at io.micronaut.http.client.netty.NettyFuturePublisher$1.lambda$request$0(NettyFuturePublisher.java:54)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
    at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.forwardResponseToPromise(DefaultHttpClient.java:2206)
    at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2179)
    at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2147)
    at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:46)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at io.micronaut.http.client.netty.ResettableReadTimeoutHandler$NextInterceptor.channelRead(ResettableReadTimeoutHandler.java:92)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
    at io.micronaut.http.client.netty.ResettableReadTimeoutHandler.channelRead(ResettableReadTimeoutHandler.java:64)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Unknown Source)

Environment Information

Example Application

No response

Version

4.5.1

goeh commented 1 month ago

The signature of the method related to the exception above is:

@Post(value = "/{+path}", consumes = MediaType.MULTIPART_FORM_DATA, produces = MediaType.APPLICATION_JSON)
Mono<HttpResponse<List<CrmContent>>> upload(HttpRequest<?> request,
                                                       @NotNull String path,
                                                       @Part("file") Publisher<StreamingFileUpload> file,
                                                       @Nullable @Part("metadata") CrmContentDescription metadata)

The core part of the method looks like this:

return Flux.from(file)
    .subscribeOn(Schedulers.boundedElastic())
    .map(part -> {
        final File tempFile = createTempFile();
        return Mono.from(part.transferTo(tempFile))
                .map(success -> service.saveFile(tempFile));
    })
    .collectList()
    .map(HttpResponse::ok);
yawkat commented 1 month ago

error looks like a missing ROUTE_MATCH, potentially fixed by #11017

goeh commented 4 days ago

After upgrade to Micronaut 4.6.1 I still see No value present exceptions. Same endpoint as above.

java.util.NoSuchElementException: No value present
    at java.base/java.util.Optional.get(Unknown Source)
    at io.micronaut.http.server.netty.NettyHttpRequest.formRouteCompleter(NettyHttpRequest.java:230)
    at io.micronaut.http.server.netty.binders.NettyPublisherPartUploadBinder.bindForNettyRequest(NettyPublisherPartUploadBinder.java:86)
    at io.micronaut.http.server.netty.binders.NettyRequestArgumentBinder.bind(NettyRequestArgumentBinder.java:39)
    at io.micronaut.http.server.netty.binders.NettyPartUploadAnnotationBinder.bind(NettyPartUploadAnnotationBinder.java:67)
    at io.micronaut.http.server.netty.binders.NettyPartUploadAnnotationBinder.bind(NettyPartUploadAnnotationBinder.java:44)
    at io.micronaut.web.router.AbstractRouteMatch.fulfillValue(AbstractRouteMatch.java:367)
    at io.micronaut.web.router.AbstractRouteMatch.fulfillBeforeFilters(AbstractRouteMatch.java:323)
    at io.micronaut.web.router.DefaultUriRouteMatch.fulfillBeforeFilters(DefaultUriRouteMatch.java:38)
    at io.micronaut.http.server.binding.RequestArgumentSatisfier.fulfillArgumentRequirementsBeforeFilters(RequestArgumentSatisfier.java:57)
    at io.micronaut.http.server.netty.NettyRequestArgumentSatisfier.fulfillArgumentRequirementsBeforeFilters(NettyRequestArgumentSatisfier.java:50)
    at io.micronaut.http.server.RequestLifecycle.fulfillArguments(RequestLifecycle.java:582)
    at io.micronaut.http.server.netty.NettyRequestLifecycle.fulfillArguments(NettyRequestLifecycle.java:133)
    at io.micronaut.http.server.RequestLifecycle.executeRoute(RequestLifecycle.java:169)
    at io.micronaut.http.server.RequestLifecycle$2.provideResponse(RequestLifecycle.java:390)
    at io.micronaut.http.filter.FilterRunner.provideResponseAndHandleErrors(FilterRunner.java:372)
    at io.micronaut.http.filter.FilterRunner.filterRequest(FilterRunner.java:307)
    at io.micronaut.http.filter.FilterRunner.lambda$filterRequest$2(FilterRunner.java:280)
    at io.micronaut.http.filter.AroundLegacyFilter$FilterChainImpl.proceed(AroundLegacyFilter.java:122)
    at io.micronaut.security.filters.SecurityFilter.lambda$checkRules$8(SecurityFilter.java:179)
    at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46)
    at reactor.core.publisher.Flux.subscribe(Flux.java:8848)
    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:82)
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onComplete(MonoFlatMapMany.java:211)
    at reactor.core.publisher.MonoNext$NextSubscriber.onComplete(MonoNext.java:102)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onComplete(FluxConcatMapNoPrefetch.java:241)
    at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:357)
    at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:294)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerComplete(FluxConcatMapNoPrefetch.java:275)
    at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onComplete(FluxConcatMap.java:889)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:299)
    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.onComplete(FluxFilter.java:300)
    at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onComplete(FluxDefaultIfEmpty.java:139)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2573)
    at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.request(FluxDefaultIfEmpty.java:98)
    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.request(FluxFilter.java:321)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
    at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:339)
    at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108)
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribe(MonoFlatMapMany.java:141)
    at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164)
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201)
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.onNext(FluxFilter.java:247)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:129)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxSwitchMapNoPrefetch$SwitchMapInner.onNext(FluxSwitchMapNoPrefetch.java:408)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:181)
    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
    at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
    at reactor.core.publisher.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
    at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:108)
    at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
    at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:108)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:108)
    at io.micronaut.http.client.filters.ClientServerRequestTracingPublisher$1.lambda$onNext$1(ClientServerRequestTracingPublisher.java:60)
    at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:49)
    at io.micronaut.http.client.filters.ClientServerRequestTracingPublisher$1.onNext(ClientServerRequestTracingPublisher.java:60)
    at io.micronaut.http.client.filters.ClientServerRequestTracingPublisher$1.onNext(ClientServerRequestTracingPublisher.java:52)
    at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
    at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:880)
    at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:805)
    at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:163)
    at io.micronaut.http.client.netty.ForwardingSubscriber.onNext(ForwardingSubscriber.java:45)
    at io.micronaut.http.client.netty.NettyFuturePublisher$1.lambda$request$0(NettyFuturePublisher.java:54)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
    at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.forwardResponseToPromise(DefaultHttpClient.java:2208)
    at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2181)
    at io.micronaut.http.client.netty.DefaultHttpClient$FullHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2149)
    at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:46)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Unknown Source)