search

numaproj / numaflow-java

Numaflow Java SDK
Apache License 2.0
22 stars 10 forks source link

feat: using akka actors instead of threads #19

Closed yhl25 closed 1 year ago

vigith commented 1 year ago

@KeranYang @yhl25, we should merge #18 before this PR, right?

yhl25 commented 1 year ago

@KeranYang @yhl25, we should merge #18 before this PR, right?

yes

KeranYang commented 1 year ago

@KeranYang @yhl25, we should merge #18 before this PR, right?

Yes, and we should add coverage check to CI as part of #14 . Currently we rely on developers to run build locally to trigger coverage check.

KeranYang commented 1 year ago

Thanks for introducing akka actors. LGTM. I have no doubt it helps achieve great performance improvement compared to using threads. Could you share with me how you tested this change and how the performance improvement looks like? @yhl25

yhl25 commented 1 year ago

Thanks for introducing akka actors. LGTM. I have no doubt it helps achieve great performance improvement compared to using threads. Could you share with me how you tested this change and how the performance improvement looks like? @yhl25

We have tested this in an actual prod pipeline which was before running without Akka actors.

observations: With Akka actors in the new pipeline we are seeing almost 5x improvement in memory usage compared to the old pipeline. Java threads are costly, with a cached thread pool we used to create too many threads which consumed a lot of memory. With Akka actors we are able to achieve very high parallelism with few java threads.