Why does Fabric use Consistency level ALL during batch process

We have batches failing with below error. Does fabric use Consistency ALL for managing batches?

ERROR 2024-06-11 15:48:21,436 [LIDddc000000000064] [JobPoolExecutorThread_BROADWAY_JOB_214] c.k.c.b.BatchProcessAPI - Failure while running batch process d1e4f61f-4155-4a88-a0ca-1d0309c69035 com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:96) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:22) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58) at com.k2view.cdbms.shared.dao.CassandraDaoInterface.execute(CassandraDaoInterface.java:81) at com.k2view.cdbms.shared.dao.CassandraDaoInterface.execute(CassandraDaoInterface.java:23) at com.k2view.cdbms.shared.dao.DaoInterface.executePrepared(DaoInterface.java:19) at com.k2view.cdbms.shared.dao.BaseDao.executePrepared(BaseDao.java:46) at com.k2view.cdbms.shared.dao.BaseDao.executePrepared(BaseDao.java:50) at com.k2view.cdbms.batchProcess.BatchProcessDaoCassImpl.executeQuery(BatchProcessDaoCassImpl.java:196) at com.k2view.cdbms.batchProcess.BatchProcessDaoCassImpl.getBatchProcessNodesData(BatchProcessDaoCassImpl.java:165) at com.k2view.cdbms.batchProcess.BatchProcessDetails.getAllNodesData(BatchProcessDetails.java:115) at com.k2view.cdbms.batchProcess.BatchProcessDetails.getTotalHandled(BatchProcessDetails.java:369) at com.k2view.cdbms.batchProcess.BatchProcessSteps.done(BatchProcessSteps.java:114) at com.k2view.cdbms.batchProcess.BatchProcessSteps.inProcess(BatchProcessSteps.java:95) at com.k2view.cdbms.batchProcess.BatchProcessSteps.generateInstances(BatchProcessSteps.java:79) at com.k2view.cdbms.batchProcess.BatchProcessSteps.startNew(BatchProcessSteps.java:27) at com.k2view.cdbms.batchProcess.BatchProcessStatus.run(BatchProcessStatus.java:22) at com.k2view.cdbms.batchProcess.BatchProcessAPI.startBatchProcess(BatchProcessAPI.java:88) at com.k2view.cdbms.batchProcess.BatchProcessAPI.startBatchProcess(BatchProcessAPI.java:57) at com.k2view.fabric.session.command.BatchProcessCommand.process(BatchProcessCommand.java:212) at com.k2view.fabric.session.command.BatchProcessCommand.boundExecute(BatchProcessCommand.java:69) at com.k2view.fabric.session.command.core.SimpleFabricCommand.doExecute(SimpleFabricCommand.java:43) at com.k2view.fabric.session.command.core.FabricCommand$BoundedFabricCommand.execute(FabricCommand.java:63) at com.k2view.fabric.session.sql.FabricStatement.lambda$doExecute$3(FabricStatement.java:247) at com.k2view.fabric.common.jdbc.JdbcUtil.sqlException(JdbcUtil.java:167) at com.k2view.fabric.session.sql.FabricStatement.doExecute(FabricStatement.java:244) at com.k2view.fabric.session.sql.FabricPreparedStatement.lambda$execute$1(FabricPreparedStatement.java:169) at com.k2view.fabric.common.jdbc.JdbcUtil.sqlException(JdbcUtil.java:167) at com.k2view.fabric.session.sql.FabricPreparedStatement.execute(FabricPreparedStatement.java:169) at com.k2view.fabric.common.jdbc.delegates.PreparedStatementDelegate.execute(PreparedStatementDelegate.java:216) at com.k2view.fabric.session.sql.SessionStatementWrapper$PrepareStatementDelegateWrapper.lambda$execute$0(SessionStatementWrapper.java:191) at com.k2view.fabric.session.sql.SessionStatementWrapper$StatementDelegateWrapper.postExecute(SessionStatementWrapper.java:105) at com.k2view.fabric.session.sql.SessionStatementWrapper$PrepareStatementDelegateWrapper.execute(SessionStatementWrapper.java:191) at com.k2view.fabric.common.io.basic.IoJdbc.sqlEx(IoJdbc.java:337) at com.k2view.fabric.common.io.basic.IoJdbc.access$100(IoJdbc.java:18) at com.k2view.fabric.common.io.basic.IoJdbc$1.execute(IoJdbc.java:80) at com.k2view.broadway.actors.builtin.DbCommand.execute(DbCommand.java:97) at com.k2view.broadway.actors.builtin.DbCommand.action(DbCommand.java:70) at com.k2view.broadway.model.Actor.action(Actor.java:71) at com.k2view.broadway.model.ActorWrapper.act(ActorWrapper.java:194) at com.k2view.broadway.model.StageImp.lambda$runActor$5(StageImp.java:304) at com.k2view.broadway.trace.TraceReport.actor(TraceReport.java:112) at com.k2view.broadway.model.StageImp.runActor(StageImp.java:303) at com.k2view.broadway.model.StageImp.tryRunningActor(StageImp.java:293) at com.k2view.broadway.model.StageImp.lambda$action$2(StageImp.java:227) at com.k2view.broadway.trace.TraceReport.stage(TraceReport.java:85) at com.k2view.broadway.model.StageImp.action(StageImp.java:186) at com.k2view.broadway.model.FlowImp.runStage(FlowImp.java:531) at com.k2view.broadway.model.FlowImp.runLevelStages(FlowImp.java:487) at com.k2view.broadway.model.FlowImp.runLevel(FlowImp.java:369) at com.k2view.broadway.model.FlowImp.runFlow(FlowImp.java:316) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:233) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:983) at com.k2view.broadway.model.FlowImp.lambda$act$19(FlowImp.java:965) at com.k2view.broadway.trace.TraceReport.flow(TraceReport.java:56) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:960) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:950) at com.k2view.broadway.BroadwayImpPool$PooledFlow.lambda$act$1(BroadwayImpPool.java:75) at com.k2view.broadway.BroadwayImpPool$PooledFlow.actAndReturn(BroadwayImpPool.java:85) at com.k2view.broadway.BroadwayImpPool$PooledFlow.act(BroadwayImpPool.java:75) at com.k2view.broadway.actors.builtin.InnerFlow.action(InnerFlow.java:19) at com.k2view.broadway.model.Actor.action(Actor.java:71) at com.k2view.broadway.model.ActorWrapper.act(ActorWrapper.java:194) at com.k2view.broadway.model.StageImp.lambda$runActor$5(StageImp.java:304) at com.k2view.broadway.trace.TraceReport.actor(TraceReport.java:112) at com.k2view.broadway.model.StageImp.runActor(StageImp.java:303) at com.k2view.broadway.model.StageImp.tryRunningActor(StageImp.java:293) at com.k2view.broadway.model.StageImp.lambda$action$2(StageImp.java:227) at com.k2view.broadway.trace.TraceReport.stage(TraceReport.java:85) at com.k2view.broadway.model.StageImp.action(StageImp.java:186) at com.k2view.broadway.model.FlowImp.runStage(FlowImp.java:531) at com.k2view.broadway.model.FlowImp.runLevelStages(FlowImp.java:487) at com.k2view.broadway.model.FlowImp.runLevel(FlowImp.java:369) at com.k2view.broadway.model.FlowImp.runFlow(FlowImp.java:316) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:233) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:983) at com.k2view.broadway.model.FlowImp.lambda$act$19(FlowImp.java:965) at com.k2view.broadway.trace.TraceReport.flow(TraceReport.java:56) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:960) at com.k2view.broadway.model.FlowImp.act(FlowImp.java:931) at com.k2view.broadway.BroadwayImpPool$PooledFlow.lambda$act$0(BroadwayImpPool.java:70) at com.k2view.broadway.BroadwayImpPool$PooledFlow.actAndReturn(BroadwayImpPool.java:85) at com.k2view.broadway.BroadwayImpPool$PooledFlow.act(BroadwayImpPool.java:70) at com.k2view.fabric.session.broadway.execution.BroadwayExecution.process(BroadwayExecution.java:34) at com.k2view.fabric.session.broadway.execution.BroadwayExecution.process(BroadwayExecution.java:20) at com.k2view.cdbms.execution.Execution.execute(Execution.java:41) at com.k2view.fabric.session.command.BroadwayCommand.simpleExecute(BroadwayCommand.java:79) at com.k2view.fabric.session.command.core.SimpleFabricCommand.boundExecute(SimpleFabricCommand.java:47) at com.k2view.fabric.session.command.core.SimpleFabricCommand.doExecute(SimpleFabricCommand.java:43) at com.k2view.fabric.session.command.core.FabricCommand$BoundedFabricCommand.execute(FabricCommand.java:63) at com.k2view.fabric.session.sql.FabricStatement.lambda$doExecute$3(FabricStatement.java:247) at com.k2view.fabric.common.jdbc.JdbcUtil.sqlException(JdbcUtil.java:167) at com.k2view.fabric.session.sql.FabricStatement.doExecute(FabricStatement.java:244) at com.k2view.fabric.session.sql.FabricStatement.executeFabricCmd(FabricStatement.java:206) at com.k2view.fabric.session.sql.FabricStatement.execute(FabricStatement.java:80) at com.k2view.fabric.common.jdbc.delegates.StatementDelegate.execute(StatementDelegate.java:87) at com.k2view.fabric.session.sql.SessionStatementWrapper$StatementDelegateWrapper.lambda$execute$0(SessionStatementWrapper.java:82) at com.k2view.fabric.session.sql.SessionStatementWrapper$StatementDelegateWrapper.postExecute(SessionStatementWrapper.java:105) at com.k2view.fabric.session.sql.SessionStatementWrapper$StatementDelegateWrapper.execute(SessionStatementWrapper.java:82) at com.k2view.fabric.session.sql.SessionStatementWrapper$StatementDelegateWrapper.executeQuery(SessionStatementWrapper.java:87) at com.k2view.fabric.common.jdbc.JdbcUtil.resultSet(JdbcUtil.java:43) at com.k2view.fabric.common.jdbc.JdbcUtil.lambda$query$1(JdbcUtil.java:115) at com.k2view.cdbms.jobs.BroadwayJob.process(BroadwayJob.java:64) at com.k2view.cdbms.jobs.Job.execute(Job.java:49) at com.k2view.cdbms.jobs.JobExecutor.run(JobExecutor.java:95) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:646) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:91) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:66) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted INFO 2024-06-11 15:48:21,442 [LIDddc000000000064] [JobPoolExecutorThread_BROADWAY_JOB_214] c.k.c.b.BatchProcessStatisticsCollector - Batch process id 'd1e4f61f-4155-4a88-a0ca-1d0309c69035' was changed to 'FAILED' INFO 2024-06-11 15:48:21,442 [LIDddc000000000064] [JobPoolExecutorThread_BROADWAY_JOB_214] c.k.c.b.BatchProcessAPI - Batch process d1e4f61f-4155-4a88-a0ca-1d0309c69035 was completed

k2v-academy / K2View-Academy

Why does Fabric use Consistency level ALL during batch process #1078