search

PostHog / posthog

🦔 PostHog provides open-source product analytics, session recording, feature flagging and A/B testing that you can self-host.
https://posthog.com
Other
20.56k stars 1.22k forks source link

Allow Kafka compression to be set by configuration #14862

Open fierzdev opened 1 year ago

fierzdev commented 1 year ago

Is your feature request related to a problem?

We tried to use the posthog hobby version and use the Azure Event Hub instead of a local Kafka installation. This leads to the plugin-server not being able to write to Kafka, as it uses snappy compression on default. This is (not yet) supported by the event hub (https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-troubleshooting-guide#compressionmessage-format-version-issue). When removing the compression, everything works fine.

Describe the solution you'd like

Currently this needs to be adjusted in Code. In the kafka producer wrapper, the compression type is hardcoded (https://github.com/PostHog/posthog/blob/master/plugin-server/src/utils/db/kafka-producer-wrapper.ts): compression: CompressionTypes.Snappy, I think it could make sense to change that, so that we can set an environment variable with the specified compression. This could also be introduced for the Python Producer for consistency. At the moment there is no compression used in Python.

As the option for compression is definitely useful, I'd suggest to try to handle this with a environment variable. If this is something that you would find useful, I might find the time to solve it and would open a pull request in the next few weeks. Or are there other solutions which would be easier?

Twixes commented 1 year ago

Hey! We'd definitely accept a PR adding KAFKA_COMPRESSION_CODEC as a plugin-server/src/config/config.ts setting.