[Feature Request] Allow for arbitrary consumer attribute mappings

[jrnt30]

There are a bunch of attributes that can be set for Kafka consumers. Our particular case has to do with adding in support for TLS.

It would be handy if we could:

• Provide general KAFKA_ environment variables that would map to the KafkaConsumer's configuration map

• BONUS: Add in multiple --consumer-config arguments that would be merged in

[braedon]

@jrnt30 I'm not sure the best way to handle environment variables in this case - while they certainly would be handy for the docker image, it seems problematic to pull config from random KAFKA_ variables when running "natively"...

I might add some form of support in the future, but for now I've gone with the BONUS implementation (in the form of .properties files - not sure if that was what you were suggesting?), which I hope will cover your use case. Let me know if not!

[jrnt30]

I understand the point and the way I've seen this addressed is that the actual environment variable prefix itself would be configurable. So as an example, the default env prefix could be something like PKC_ for Prometheus Kafka Consumer but there could be a --env-prefix argument that would allow you to configure the prefix that the code uses to make it more customizable.

We implemented something similar for a client that essentially just looked the os.environ, stripped off the prefix and then lowered/replaced _ with . so that you could have things like PKC_SECURITY_PROTOCOL=SSL PKC_CA_LOCATION=<file_path> that mapped to the security.protocol=ssl and ca.location=<filepath> for the consumer config

That being said, an external properties file would still be useful so thanks