[Bug] Pulsar Functions ignores compressionType and crypto config for producers created with Context produce/newOutputMessage methods - Githubissues

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system

https://pulsar.apache.org/

Apache License 2.0

13.94k stars 3.53k forks source link

[Bug] Pulsar Functions ignores compressionType and crypto config for producers created with Context produce/newOutputMessage methods #22948

Closed lhotari closed 1 week ago

lhotari commented 1 week ago

Search before asking

[X] I searched in the issues and found nothing similar.

Read release policy

[X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

any released version

Minimal reproduce step

-

What did you expect to see?

I'd expect that the configured compression type would be used. compression type is handled in the other location where Pulsar Sources create producers: https://github.com/apache/pulsar/blob/5fc0eafab9ea2a4ece7b87218404489c270b64e6/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L123-L127

crypto config is handled for Pulsar Sources: https://github.com/apache/pulsar/blob/5fc0eafab9ea2a4ece7b87218404489c270b64e6/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L134-L140

What did you see instead?

compression type is hard coded to LZ4: https://github.com/apache/pulsar/blob/5fc0eafab9ea2a4ece7b87218404489c270b64e6/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java#L565 There's no crypto config configuration.

Anything else?

Code to create producers shouldn't be duplicated in Pulsar Functions since it leads to inconsistencies.

Are you willing to submit a PR?

[X] I'm willing to submit a PR!