Add new option for parallel publish send.

dotnetcore / CAP

Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern

http://cap.dotnetcore.xyz

MIT License

6.61k stars 1.28k forks source link

Add new option for parallel publish send. #1540

Closed maisa92 closed 2 months ago

maisa92 commented 3 months ago

When we enable the EnablePublishParallelSend option, our application crashes under high load.

Is it possible to add an option to control the thread count for parallel sending? like it is done on the subscriber side SubscriberParallelExecuteThreadCount.

I am referring to issue #1513 as a benchmark, and I believe a similar improvement would help us resolve this issue.

yang-xiaodong commented 3 months ago

Hello @maisa92 ,

I need some time to determine the cause of the application crash because I didn't observe this issue in my previous benchmarks. Unlike consumption, publishing messages is a short-term task, which is well-suited for .NET thread pool tasks.

Could you share more information? For example, was it running in Docker and got killed due to exceeding memory or CPU limits, or was there another reason? Any details you can share would be greatly appreciated.

yang-xiaodong commented 2 months ago

Fixed in version 8.2.0.

Place tasks batches into thread pool when turning on EnablePublishParallelSend