Increase process instance throughput and reduce latency of changes appearing in Operate and Tasklist webapps, by cutting off the importer from the data pipeline
User Problem 🤦
The importer is a common source of errors. For example, if it gets stuck, Zeebe will continue executing processes, but the changes will not reflect in Operate. End users will perceive the overall system as unhealthy.
The Zeebe exporter and Operate/Tasklist importer are on the hot data path of the system. It needs to work at a large scale (=> process instances) and efficiently.
By writing all data first to Zeebe indexes and then moving it from there to Operate/Tasklist indexes, we are copying all exported data around unnecessarily. Furthermore, this pattern of deleting all documents in a high-volume index does not align with Elasticsearch / Opensearch best practices and creates unnecessary housekeeping overhead for Elasticsearch / Opensearch.
User Stories 🧑🚀
As an operations engineer, I can easily migrate from the Camunda version with importer + exporter to the version with exporter only (8.7). No manual steps are required from me when upgrading to new version of Camunda
As an operations engineer, I can read documentation how the exporter works
Implementation 👷
The following assumes that 8.6 is the version that ships this epic and 8.5 is the last version that does not (if we do not ship this epic in 8.6, the versions adapt to the new release target). Some aspects will be implemented in 8.5 to prepare Zeebe/Operate for a smooth migration. There is no end-user value in 8.5 that contributes to the value proposition.
This is a rather central architectural change in how data flows in the hot data path. It requires us revisiting features and behavior that we have built-in assumptions of the current architecture, such as: Operate index migration, backups, performance goals of import/export, migration of in-flight data when switching from one approach to the other
:robot: This issue is automatically synced from: source
Value Proposition Statement 🚀
Increase process instance throughput and reduce latency of changes appearing in Operate and Tasklist webapps, by cutting off the importer from the data pipeline
User Problem 🤦
User Stories 🧑🚀
Implementation 👷
:robot: This issue is automatically synced from: source