search

camunda-community-hub / camunda-8-helm-profiles

A collection of Helm values files for the Camunda Platform 8 Helm Chart
Apache License 2.0
19 stars 21 forks source link

Multi-Region: Conflicting exporter names result in lost data #82

Open lenaschoenburg opened 9 months ago

lenaschoenburg commented 9 months ago

The Multi-Region profiles configure each region's Zeebe to use their local ES as exporter named "elasticsearch" and the remote ES as exporter named "elasticsearch2".

As a result, Zeebe will distribute conflicting metadata about the exporters. Zeebe brokers from region 0 share something like es1={pos=15, sequence=10},es2={pos=30, sequence=27} while Zeebe brokers from region 1 flip it and distribute es1={pos=30, sequence=27},es2={pos=15, sequence=10}.

Depending on the ordering in which updates are distributed, some leaders will run with incorrect metadata which can result in missing data in ES (i.e. dataloss) or wrong sequence numbers (i.e. data corruption).

hamza-m-masood commented 9 months ago

Thanks for creating the issue. Here is the fix within this commit