search

dyrnq / cdc-vagrant

CDC(Change Data Capture) is made up of two components, the CDD and the CDT. CDD is stand for Change Data Detection and CDT is stand for Change Data Transfer.
https://nightlies.apache.org/flink/flink-cdc-docs-stable/
0 stars 0 forks source link

java.io.IOException: Could not perform checkpoint 2 for operator SinkMaterializer[36] -> Sink: enriched_orders[36] (2/2)#0 #15

Open dyrnq opened 1 year ago

dyrnq commented 1 year ago 
2022-12-01 23:00:04,595 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 1 as completed for source Source: orders[25].
2022-12-01 23:00:04,595 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 1 as completed for source Source: products[27].
2022-12-01 23:00:06,519 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1669906806494 for job 669fd3809635a13a875e92c90d5ada67.
2022-12-01 23:00:07,329 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - SinkMaterializer[36] -> Sink: enriched_orders[36] (2/2) (d895619b5a7986dffbe3bc1df0522100_5b5d46caf7fabdb9e789f1f4dac466a5_1_0) switched from RUNNING to FAILED on 192.168.56.116:9267-93cc4c @ vm116 (dataPort=15041).
java.io.IOException: Could not perform checkpoint 2 for operator SinkMaterializer[36] -> Sink: enriched_orders[36] (2/2)#0.
    at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1238) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:488) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) ~[flink-dist-1.16.0.jar:1.16.0]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]
    Suppressed: java.lang.RuntimeException: An error occurred in ElasticsearchSink.
        at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.checkErrorAndRethrow(ElasticsearchSinkBase.java:426) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.close(ElasticsearchSinkBase.java:365) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:41) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.close(AbstractUdfStreamOperator.java:114) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1025) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:943) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$0(Task.java:917) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:917) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) ~[flink-dist-1.16.0.jar:1.16.0]
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) ~[flink-dist-1.16.0.jar:1.16.0]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
    Caused by: org.apache.flink.elasticsearch7.shaded.org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=mapper [price] cannot be changed from type [long] to [float]]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:139) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:188) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$10(RestHighLevelClient.java:1699) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1781) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:636) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestClient$1.completed(RestClient.java:376) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestClient$1.completed(RestClient.java:370) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
        ... 1 more
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 2 for operator SinkMaterializer[36] -> Sink: enriched_orders[36] (2/2)#0. Failure reason: Checkpoint was declined.
    at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:269) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:173) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:345) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:726) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:363) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1281) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1269) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1226) ~[flink-dist-1.16.0.jar:1.16.0]
    ... 22 more
Caused by: java.lang.RuntimeException: An error occurred in ElasticsearchSink.
    at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.checkErrorAndRethrow(ElasticsearchSinkBase.java:426) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.checkAsyncErrorsAndRequests(ElasticsearchSinkBase.java:431) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.snapshotState(ElasticsearchSinkBase.java:344) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:173) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:345) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:726) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:363) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1281) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1269) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1226) ~[flink-dist-1.16.0.jar:1.16.0]
    ... 22 more
Caused by: org.apache.flink.elasticsearch7.shaded.org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=mapper [price] cannot be changed from type [long] to [float]]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:139) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:188) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$10(RestHighLevelClient.java:1699) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1781) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:636) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestClient$1.completed(RestClient.java:376) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.RestClient$1.completed(RestClient.java:370) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    at org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[flink-sql-connector-elasticsearch7-1.16.0.jar:1.16.0]
    ... 1 more
2022-12-01 23:00:07,380 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - 11 tasks will be restarted to recover the failed task d895619b5a7986dffbe3bc1df0522100_5b5d46caf7fabdb9e789f1f4dac466a5_1_0.
2022-12-01 23:00:07,380 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job insert-into_default_catalog.default_database.enriched_orders (669fd3809635a13a875e92c90d5ada67) switched from state RUNNING to RESTARTING.
2022-12-01 23:00:07,381 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 2 for job 669fd3809635a13a875e92c90d5ada67. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator is suspending.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:1926) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1566) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1161) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1133) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:571) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:362) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:334) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:305) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:247) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:240) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:738) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:715) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78) ~[flink-dist-1.16.0.jar:1.16.0]
    at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:477) ~[flink-dist-1.16.0.jar:1.16.0]
    at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:309) ~[flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) ~[flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:307) ~[flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:222) ~[flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84) ~[flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:168) ~[flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_0aad32e2-2924-43db-8369-4b5052950f45.jar:1.16.0]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
    at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
2022-12-01 23:00:07,395 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#0) of source Source: orders[25].
2022-12-01 23:00:07,394 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: orders[25] (1/2) (d895619b5a7986dffbe3bc1df0522100_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,402 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 1 (#0) of source Source: orders[25].
2022-12-01 23:00:07,402 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: orders[25] (2/2) (d895619b5a7986dffbe3bc1df0522100_bc764cd8ddf7a0cff126f51c16239658_1_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,403 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: products[27] (1/2) (d895619b5a7986dffbe3bc1df0522100_feca28aff5a3958840bee985ee7de4d3_0_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,403 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Join[29] -> Calc[30] (2/2) (d895619b5a7986dffbe3bc1df0522100_f3c52ad168ea5842a0be53deff739f6c_1_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,403 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: shipments[32] (1/1) (d895619b5a7986dffbe3bc1df0522100_605b35e407e90cda15ad084365733fdd_0_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,403 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Join[34] -> Calc[35] -> ConstraintEnforcer[36] (1/2) (d895619b5a7986dffbe3bc1df0522100_4bc97eb6511780b518f989f149963722_0_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,404 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Join[34] -> Calc[35] -> ConstraintEnforcer[36] (2/2) (d895619b5a7986dffbe3bc1df0522100_4bc97eb6511780b518f989f149963722_1_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,404 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 (#0) of source Source: products[27].
2022-12-01 23:00:07,404 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Join[29] -> Calc[30] (1/2) (d895619b5a7986dffbe3bc1df0522100_f3c52ad168ea5842a0be53deff739f6c_0_0) switched from RUNNING to CANCELING.
2022-12-01 23:00:07,404 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: products[27] (2/2) (d895619b5a7986dffbe3bc1df0522100_feca28aff5a3958840bee985ee7de4d3_1_0) switched from RUNNING to CANCELING.