search

berndruecker / flowing-retail

Sample application demonstrating an order fulfillment system decomposed into multiple independant components (e.g. microservices). Showing concrete implementation alternatives using e.g. Java, Spring Boot, Apache Kafka, Camunda, Zeebe, ...
Apache License 2.0
1.31k stars 468 forks source link

Kafka Docker Example Performance #82

Closed 1123 closed 10 months ago

1123 commented 1 year ago

Great Tutorial and Use Case. This shows clearly the benefit of a distributed workflow engine.

While testing the performance in the local setup, I noticed that the system is already at a limit with 10 orders per second. Clearly in a production environment, services would be distributed across separate VMs, and possibly also scaled horizontally for fault tolerance. Still it seems like there is a clear bottleneck in the Zeebe service writing data to ElasticSearch, and one would expect a performance of more than 10 workflows / second even in a sample setup. While all other services have done their job, the logs of Zeebe still show write requests to Elastic.

image

This results in Camunda Operate showing out of date information when it comes to the state of the workflow instances:

image

Wouldn't it be possible to batch the writes from Zeebe to ElasticSearch to improve the performance by quite a bit? Or Is batching already taking place? Why are the writes so slow?

berndruecker commented 10 months ago

I am closing this if you don't mind - I think if you want to look at performance you should use a normal Camunda 8 distro and test there (and ideally ask in the forum then if there are any issues: https://forum.camunda.io/.

OK?