search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.28k stars 3.59k forks source link

Add batching properties to the ProducerConfig for Pulsar Functions #21758

Open david-streamlio opened 11 months ago

david-streamlio commented 11 months ago

Search before asking

Motivation

When deploying a Pulsar Function, a user might need to override the batching behavior of the functions producer. Currently, the ProducerConfig configuration only provides two options; "DEFAULT", and "KEY_BASED".

It would be worthwhile to allow users to disable batching entirely, or have more granular control of the batching behavior, in terms of batchingMaxBytes,batchingMaxMessages, and batchingMaxPublishDelay

Solution

Add the above three fields to the ProducerConfig definition and have them passed through to the ProducerBuilder class when the Pulsar Function is instantiated.

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

jsteggink commented 11 months ago

Great idea! This is something my team is also very interested in. In the past @devinbost made a PR for similar configuration options, but was never accepted: https://github.com/apache/pulsar/pull/10154

Technoboy- commented 11 months ago

Could you help check if we can improve this ? @freeznet @jiangpengcheng