Ability to disable SSL verification for Kafka connection

[sgaragan]

In our lab environment, the certificate being used by Kafka is not always a valid one (either self-signed or the CN/SAN does not match the hostname). Kafka has a property that can be set, "enable.ssl.certificate.verification" but this is not available to the gRPC Kafka configuration. We are looking for something like the following for the pmtelemetryd-grpc-dialout.conf

## Kafka configuration ###
bootstrap_servers = "jedha-kafka-kafka-mtlsext-0.kafka.svc:9096,jedha-kafka-kafka-mtlsext-1.kafka.svc:9096,jedha-kafka-kafka-mtlsext-2.kafka.svc:9096";
topic = "matterhorn.ietf.grpc-raw";
enable_idempotence = "true";
client_id = "mdt-dialout-collector";
# valid options are either plaintext or ssl
security_protocol = "ssl";
ssl_key_location = "/etc/pmacct/ssl/kafka.key";
ssl_certificate_location = "/etc/pmacct/ssl/kafka.crt";
ssl_ca_location = "/etc/pmacct/ssl/ca.crt";
enable_ssl_certificate_verification = "false";  <---- New Kafka config property

Thanks, Sean

[scuzzilla]

@sgaragan many thanks for your request. For clarity, which of the available "Deployment Options" are you planning to use? More specifically, are you considering the standalone version of the collector, or would you prefer to integrate it with pmtelemetryd?

[sgaragan]

We are integrating as part of pmtelemetryd to allow us to use this binary for multiple telemetry input options

[scuzzilla]

@sgaragan, in this instance, ZMQ is used for the communication between the gRPC dial-out collector and pmtelemetryd, which directly handles data delivery to Kafka. Therefore, the Kafka option you are inquiring about is already available. For reference, see: https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L823.

[sgaragan]

@sgaragan, in this instance, ZMQ is used for the communication between the gRPC dial-out collector and pmtelemetryd, which directly handles data delivery to Kafka. Therefore, the Kafka option you are inquiring about is already available. For reference, see: https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L823.

We have that set in the librdkafka.conf file but the pmtelemetryd-grpc.conf where we have the gRPC configuration does not seem to use that, it uses the configuration keys defined in mdt-dialout-collector. I have attached a zip file with our config for pmtelemetryd (with some redactions of course)

pmtelemetryd-conf.zip

[scuzzilla]

@sgaragan, if you haven't already, you might want to check out this link where I describe the integration between gRPC dial-out and pmtelemetryd schematically. As you will see, it is required to set data_delivery_method = "zmq", which is mutually exclusive with Kafka.

[sgaragan]

@sgaragan, if you haven't already, you might want to check out this link where I describe the integration between gRPC dial-out and pmtelemetryd schematically. As you will see, it is required to set data_delivery_method = "zmq", which is mutually exclusive with Kafka.

That worked. We must have missed the requirement to use zmq when it was originally set up in our lab environment.

[scuzzilla]

@sgaragan in case you would like to give a try to the standalone version of the collector I added the kafka options you requested earlier: "enable_ssl_certificate_verification"