search

samagra-comms / transformer

0 stars 12 forks source link

[Bug] - Getting Error in Aggregate Postgres #84

Closed pankajjangid05 closed 1 year ago

pankajjangid05 commented 1 year ago

While multiple requests come on the transformer and the transformer gets the ODK form from Postgres we are getting the below error :

  1. Error 
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    | reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.dao.DataAccessResourceFailureException: executeMany; SQL [SELECT question.id, question.form_id, question.form_version, question.x_path, question.question_type, question.meta, question.updated, question.created FROM question WHERE question.x_path = $1 AND (question.form_id = $2) AND (question.form_version = $3)]; null; nested exception is io.r2dbc.postgresql.client.ReactorNettyClient$PostgresConnectionException
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    | Caused by: org.springframework.dao.DataAccessResourceFailureException: executeMany; SQL [SELECT question.id, question.form_id, question.form_version, question.x_path, question.question_type, question.meta, question.updated, question.created FROM question WHERE question.x_path = $1 AND (question.form_id = $2) AND (question.form_version = $3)]; null; nested exception is io.r2dbc.postgresql.client.ReactorNettyClient$PostgresConnectionException
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at org.springframework.r2dbc.connection.ConnectionFactoryUtils.convertR2dbcException(ConnectionFactoryUtils.java:226)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at org.springframework.r2dbc.core.DefaultDatabaseClient.lambda$inConnectionMany$8(DefaultDatabaseClient.java:147)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Flux.lambda$onErrorMap$29(Flux.java:6946)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Flux.lambda$onErrorResume$30(Flux.java:6999)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.deferredError(FluxUsingWhen.java:411)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxUsingWhen$RollbackInner.onComplete(FluxUsingWhen.java:488)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2058)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:209)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:234)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:209)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.complete(Operators.java:137)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.netty.FutureMono$ImmediateFutureMono.subscribe(FutureMono.java:83)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:236)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:299)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:367)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onComplete(FluxConcatMap.java:276)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:277)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:236)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFilter$FilterSubscriber.onError(FluxFilter.java:157)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.onError(FluxFilter.java:291)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:259)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onError(FluxUsingWhen.java:377)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:842)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:608)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:588)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFlatMap$FlatMapMain.innerError(FluxFlatMap.java:863)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onError(FluxFlatMap.java:990)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:203)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.onError(FluxTakeUntil.java:111)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onError(FluxFilterFuseable.java:162)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.util.FluxDiscardOnCancel$FluxDiscardOnCancelSubscriber.onError(FluxDiscardOnCancel.java:92)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BaseSink.error(FluxCreate.java:453)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:781)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BufferAsyncSink.error(FluxCreate.java:726)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$SerializedFluxSink.drainLoop(FluxCreate.java:230)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$SerializedFluxSink.drain(FluxCreate.java:206)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$SerializedFluxSink.error(FluxCreate.java:182)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$Conversation.onError(ReactorNettyClient.java:711)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.close(ReactorNettyClient.java:944)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient.drainError(ReactorNettyClient.java:475)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient.handleConnectionError(ReactorNettyClient.java:470)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient.lambda$new$2(ReactorNettyClient.java:154)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:894)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:259)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:259)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.FluxReceive.onInboundError(FluxReceive.java:433)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.ChannelOperations.onInboundError(ChannelOperations.java:488)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:126)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:381)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at java.base/java.lang.Thread.run(Unknown Source)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    | Caused by: io.r2dbc.postgresql.client.ReactorNettyClient$PostgresConnectionException: null
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient.lambda$handleConnectionError$12(ReactorNettyClient.java:470)
core_transformer.1.ym4eg272i3vm@UCI-Docker-Swarm-Master    |    ... 29 common frames omitted
  1. Error 
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    | reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: [53300] sorry, too many clients already
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    | Caused by: org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: [53300] sorry, too many clients already
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3733)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoDelayUntil$DelayUntilTrigger.onError(MonoDelayUntil.java:514)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:203)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:125)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleConditionalSubscriber.onNext(FluxHandle.java:327)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:793)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:718)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:154)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$Conversation.emit(ReactorNettyClient.java:700)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.emit(ReactorNettyClient.java:888)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onNext(ReactorNettyClient.java:822)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onNext(ReactorNettyClient.java:731)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:119)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:854)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:279)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:388)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at java.base/java.lang.Thread.run(Unknown Source)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    | Caused by: io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: sorry, too many clients already
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.ExceptionFactory.createException(ExceptionFactory.java:98)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.ExceptionFactory.handleErrorResponse(ExceptionFactory.java:110)
core_transformer.1.uyde5q076z73@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:103)
pankajjangid05 commented 1 year ago

Yesterday, we targeted over 112,000 users to trigger notifications on prod. As a result, this error did not occur again. But I want to test the load on the transformer and test how it performs.

pankajjangid05 commented 1 year ago

Friday we got this error : 

core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    | 2023-07-21 10:32:37,801 ERROR [reactor-tcp-epoll-16] com.uci.transformer.odk.ODKConsumerReactive:533: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: [53300] sorry, too many clients already
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    | 2023-07-21 10:32:37,801 ERROR [reactor-tcp-epoll-16] reactor.util.Loggers$Slf4JLogger:315: Operator called default onErrorDropped
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    | reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: [53300] sorry, too many clients already
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    | Caused by: org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: [53300] sorry, too many clients already
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3733)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Operators.error(Operators.java:198)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.Mono.subscribe(Mono.java:4400)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.MonoDelayUntil$DelayUntilTrigger.onError(MonoDelayUntil.java:514)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:203)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:125)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleConditionalSubscriber.onNext(FluxHandle.java:327)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:793)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:718)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:154)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$Conversation.emit(ReactorNettyClient.java:700)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.emit(ReactorNettyClient.java:888)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onNext(ReactorNettyClient.java:822)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onNext(ReactorNettyClient.java:731)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:119)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:854)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:279)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:388)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at java.base/java.lang.Thread.run(Unknown Source)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    | Caused by: io.r2dbc.postgresql.ExceptionFactory$PostgresqlNonTransientResourceException: sorry, too many clients already
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.ExceptionFactory.createException(ExceptionFactory.java:98)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at io.r2dbc.postgresql.ExceptionFactory.handleErrorResponse(ExceptionFactory.java:110)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:103)
core_transformer.1.po4knwou2uqe@UCI-Docker-Swarm-Master    |    ... 35 common frames omitted
pankajjangid05 commented 1 year ago

Connection pooling added and deployed on staging and prod .

aakashyadav-kgp commented 1 year ago

Waiting for a few more bot triggering to double check resolution.