Currently this needs to be adjusted in Code. In the kafka producer wrapper, the compression type is hardcoded (https://github.com/PostHog/posthog/blob/master/plugin-server/src/utils/db/kafka-producer-wrapper.ts):
compression: CompressionTypes.Snappy,
I think it could make sense to change that, so that we can set an environment variable with the specified compression. This could also be introduced for the Python Producer for consistency. At the moment there is no compression used in Python.
As the option for compression is definitely useful, I'd suggest to try to handle this with a environment variable. If this is something that you would find useful, I might find the time to solve it and would open a pull request in the next few weeks. Or are there other solutions which would be easier?
Is your feature request related to a problem?
We tried to use the posthog hobby version and use the Azure Event Hub instead of a local Kafka installation. This leads to the plugin-server not being able to write to Kafka, as it uses snappy compression on default. This is (not yet) supported by the event hub (https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-troubleshooting-guide#compressionmessage-format-version-issue). When removing the compression, everything works fine.
Describe the solution you'd like
Currently this needs to be adjusted in Code. In the kafka producer wrapper, the compression type is hardcoded (https://github.com/PostHog/posthog/blob/master/plugin-server/src/utils/db/kafka-producer-wrapper.ts):
compression: CompressionTypes.Snappy,
I think it could make sense to change that, so that we can set an environment variable with the specified compression. This could also be introduced for the Python Producer for consistency. At the moment there is no compression used in Python.As the option for compression is definitely useful, I'd suggest to try to handle this with a environment variable. If this is something that you would find useful, I might find the time to solve it and would open a pull request in the next few weeks. Or are there other solutions which would be easier?