Import of cancelled instances is not blocked when batch operation is absent

Description

(This is a follow-up for #5984)

When PROCESS_INSTANCE ELEMENT_TERMINATED record is being imported it searched for corresponding operations and batch operation to update the progress. It may happen that batch operation does not exist (e.g. after data loss). In such case exception is thrown and import is blocked:

io.camunda.operate.exceptions.PersistenceException: org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: index is missing;2: index is missing;3: index is missing;4: index is missing;5: index is missing;6: index is missing;7: index is missing;8: index is missing;9: index is missing;10: index is missing;11: index is missing;12: index is missing;13: index is missing;14: index is missing;15: index is missing;16: index is missing;17: index is missing;
    at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.performImport(AbstractImportBatchProcessor.java:43) ~[operate-importer-common-8.3.3.jar!/:8.3.3]

Mitigation

One solution is to delete operations for which batch operations do not exist anymore. See Support KB

TODO

ignore missing batch operation when importing
improve logging when possible to see which specific request from bulk request caused the issue

Additional Information

Related support case: https://jira.camunda.com/browse/SUPPORT-19750 https://camunda.slack.com/archives/C06A7UCA2BE
Here is the problem located: https://github.com/camunda/operate/blob/71d87a6e6a5a9976cdbc0f073e2b6d576f3800fa/els-schema/src/main/java/io/camunda/operate/util/OperationsManager.java#L50

the full stacktrace:

io.camunda.operate.exceptions.PersistenceException: org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: index is missing;
at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.performImport(AbstractImportBatchProcessor.java:43) ~[operate-importer-common-8.3.3.jar!/:8.3.3]
at io.camunda.operate.zeebeimport.ImportJob.processOneIndexBatch(ImportJob.java:142) ~[operate-importer-8.3.3.jar!/:8.3.3]
at io.camunda.operate.zeebeimport.ImportJob.call(ImportJob.java:84) ~[operate-importer-8.3.3.jar!/:8.3.3]
at io.camunda.operate.zeebeimport.elasticsearch.ElasticsearchRecordsReader.lambda$executeJob$5(ElasticsearchRecordsReader.java:462) ~[operate-importer-8.3.3.jar!/:8.3.3]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: index is missing;
at org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:486) ~[elasticsearch-7.17.15.jar!/:7.17.15]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2133) ~[elasticsearch-rest-high-level-client-7.17.15.jar!/:7.17.15]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105) ~[elasticsearch-rest-high-level-client-7.17.15.jar!/:7.17.15]
at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:620) ~[elasticsearch-rest-high-level-client-7.17.15.jar!/:7.17.15]
at io.camunda.operate.util.ElasticsearchUtil.processLimitedBulkRequest(ElasticsearchUtil.java:336) ~[operate-els-schema-8.3.3.jar!/:8.3.3]
at io.camunda.operate.util.ElasticsearchUtil.processBulkRequest(ElasticsearchUtil.java:291) ~[operate-els-schema-8.3.3.jar!/:8.3.3]
at io.camunda.operate.util.ElasticsearchUtil.processBulkRequest(ElasticsearchUtil.java:284) ~[operate-els-schema-8.3.3.jar!/:8.3.3]
at io.camunda.operate.store.elasticsearch.ElasticsearchBatchRequest.execute(ElasticsearchBatchRequest.java:169) ~[operate-els-schema-8.3.3.jar!/:8.3.3]
at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.lambda$performImport$1(AbstractImportBatchProcessor.java:38) ~[operate-importer-common-8.3.3.jar!/:8.3.3]
at io.micrometer.core.instrument.AbstractTimer.recordCallable(AbstractTimer.java:175) ~[micrometer-core-1.11.5.jar!/:1.11.5]
at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.withTimer(AbstractImportBatchProcessor.java:59) ~[operate-importer-common-8.3.3.jar!/:8.3.3]
at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.withImportIndexQueryTimer(AbstractImportBatchProcessor.java:52) ~[operate-importer-common-8.3.3.jar!/:8.3.3]
at io.camunda.operate.zeebeimport.AbstractImportBatchProcessor.performImport(AbstractImportBatchProcessor.java:37) ~[operate-importer-common-8.3.3.jar!/:8.3.3]
... 7 more

Acceptance Criteria

[ ] Checked if the specific data needs to be provided for the QA process

Definition of Ready - Checklist

[ ] The issue has a meaningful title, description and testable acceptance criteria

^{:robot: This issue is automatically synced from: source}

camunda / issues