search

camunda / camunda

Process Orchestration Framework
https://camunda.com/platform/
3.32k stars 604 forks source link

Operate not importing incidents, Post Importer #21407

Open klidio opened 2 months ago

klidio commented 2 months ago

Describe the bug

Camunda Operate stops working randomly, it does not import incidents and throws an exception, events no longer imported after exception. Related to: https://github.com/camunda/camunda/issues/13189

Current behavior

ERROR 7 --- [ postimport_1] c.o.z.p.AbstractIncidentPostImportAction : Exception occurred when performing post import for partition 3: Process instance is not yet imported for incident> io.camunda.operate.exceptions.OperateRuntimeException: Process instance is not yet imported for incident processing. Incident id: 6755399487015201, process instance id: 6755399468092024 at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.checkDataAndCollectParentTreePaths(OpensearchIncidentPostImportAction.java:445) ~[operate-importer-8.4.5.jar!/:8.4.5] at io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.searchForInstances(OpensearchIncidentPostImportAction.java:404) ~[operate-importer-8.4.5.jar!/:8.4.5] at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:111) ~[operate-importer-8.4.5.jar!/:8.4.5] at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:58) ~[operate-importer-8.4.5.jar!/:8.4.5] at io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:72) [operate-importer-8.4.5.jar!/:8.4.5] at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.0.17.jar!/:6.0.17] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.lang.Thread.run(Thread.java:840) [?:?]

Expected behavior

IncidentPostImportAction imports incidents to Operate as usually.

Log/Stacktrace

Error keeps repeating every five seconds

Environment

psmarques commented 4 days ago

We are facing the same issue... after a while, some instances are not processing as expected..

Checking for zeebe logs, we found some errors:

{"severity":"ERROR","logging.googleapis.com/sourceLocation":{"function":"run","file":"AbstractIncidentPostImportAction.java","line":85},"message":"Exception occurred when performing post import for partition 2: For input string: \"4503599627493503\". Will be retried...","serviceContext":{"service":"operate","version":"development"},"context":{"threadId":108,"threadPriority":5,"loggerName":"io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction","threadName":"postimport_1"},"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","exception":"io.camunda.operate.exceptions.OperateRuntimeException: For input string: \"4503599627493503\"\n\tat io.camunda.operate.util.ExceptionHelper.withOperateRuntimeException(ExceptionHelper.java:47) ~[operate-schema-8.5.6.jar!/:8.5.6]\n\tat io.camunda.operate.store.opensearch.client.sync.OpenSearchBatchOperations.bulk(OpenSearchBatchOperations.java:79) ~[operate-schema-8.5.6.jar!/:8.5.6]\n\tat io.camunda.operate.zeebeimport.post.opensearch.OpensearchPostImporterRequests.execute(OpensearchPostImporterRequests.java:80) ~[operate-importer-8.5.6.jar!/:8.5.6]\n\tat io.camunda.operate.zeebeimport.post.opensearch.OpensearchIncidentPostImportAction.processIncidents(OpensearchIncidentPostImportAction.java:283) ~[operate-importer-8.5.6.jar!/:8.5.6]\n\tat io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.processPendingIncidents(AbstractIncidentPostImportAction.java:129) ~[operate-importer-8.5.6.jar!/:8.5.6]\n\tat io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.performOneRound(AbstractIncidentPostImportAction.java:64) ~[operate-importer-8.5.6.jar!/:8.5.6]\n\tat io.camunda.operate.zeebeimport.post.AbstractIncidentPostImportAction.run(AbstractIncidentPostImportAction.java:79) ~[operate-importer-8.5.6.jar!/:8.5.6]\n\tat org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.1.10.jar!/:6.1.10]\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]\n\tat java.base/java.lang.Thread.run(Thread.java:1583) [?:?]\n","timestampSeconds":1729888792,"timestampNanos":82165605}

klidio commented 3 days ago

Check if the processDefinitionKey for this indident exists, check operate processes, usually when incident its not imported its due to missing process, check data in index operate-process-8.*.0_