[bug] tls error when configuring a kafka client

[bb-ha-melech]

hey, i am experiencing some issues with the kafka client connection with centrifugo, while configuring it via image.

The kafka itself if secured with SSL (there are self signed certificates that need to be sent), and the trusted root ca on the pod is updated. The kafka certificates themselves are also correct (i am able to connect with them if i try to do it using python kafka library). When i am trying to run the centrifugo serer with the kafka i get:

error initializing consumers: error (consumer group): error init Kafka Client: error ping Kafka: remote error: tls: bad certificate

It could be because maybe i have some misunderstanding of the config.json structure :)

Is there maybe an example to file types/expected content for each key in the tls kafka config that are expected in the config.json? ...

Versions

Centrifugo version is 5.4.6 Operating system is docker alpine linux 3.18

Could there be any problem with self signed certificates in golang? we are using it as a deployed service and do not usually develop with go, so any help figuring this out will be amazing

  "consumers": [
    {
      "name": "kafka_consumer",
      "type": "kafka",
      "kafka": {
        "brokers": ["broker-address"],
        "topics": ["..."],
        "consumer_group": "consumer-group", 
        "tls": true,
        "tls_key": "path-to-key/key-file.key",
        "tls_cert_pem":"pem-content",
         "tls_root_ca":"path-to-ca/ca.crt",
         "tls_insecure_skip_verify": true
      }
    }
  ]
}

[FZambia]

Hello @bb-ha-melech , I think this may be caused by how Centrifugo now resolves TLS configuration: you have tls_key from file but tls_cert_pem from memory – and eventually no TLS config created in such a case.

Try using whether tls_key and tls_cert (i.e. both from PEM files on disk), or tls_key_pem and tls_cert_pem (both from pem strings directly set in config)

[bb-ha-melech]

thank you for answering so quickly can the tls_cert be in .crt format?

[FZambia]

I think no.. looking at Go function which is used by Centrifugo it seems it expects content to contain PEM-encoded cert.

I suppose you can try converting your .crt if it's not PEM yet using sth like:

openssl x509 -in yourfile.crt -out yourfile.crt.pem -outform PEM

[bb-ha-melech]

ok, i thing i got it. I got from my kafka provider .key and .pem certificates so i understand correctly then: i pass the .pem anf the bundle.pem as the required verification

[bb-ha-melech]

we tried what you said and we still get the bad certificate error is there a way to debug this error? we can't really know which file causes the error, the logs dont show more information

even when we put tls true, and then dont put anything else, then we still get the bad certificate error

[bb-ha-melech]

We tried connecting using openssl s_client -connect <OUR BROKER> -key -cert .... and we connected successfully ! But the same certificates just don't seem to work when using the Centrifugo configuration :(

We tried every combination, using both the string and the file methods and it doesn't seem to change the end result Is there any information we could give you to help debugging the process ? Our kafka is version is 3.5.1 if that could help diagnose the problem

Thank you very much for the help ! we really want to use centrifugo in our project but this is a serious blocker for us :(

[FZambia]

I think we can figure it out eventually, just not very handy since I don't have the local reproducer. And have only part of the information about your certs.

I added TLS configuration debug logs to v6 branch (it was much easier for me at this stage), maybe you can try it? It uses a bit different configuration from v5 due to https://github.com/centrifugal/centrifugo/issues/832 but here are the instructions:

You need Go installed.

Then:

git clone -b v6_dev https://github.com/centrifugal/centrifugo.git
cd centrifugo
go build
./centrifugo --log_level debug -c kafka.json

Where kafka.json is:

{
  "log_level": "debug",
  "consumers": [
    {
      "enabled": true,
      "name": "mykafka",
      "type": "kafka",
      "kafka": {
        "brokers": ["localhost:29092"],
        "topics": ["postgres.public.chat_cdc"],
        "consumer_group": "centrifugo",
        "tls": {
          "enabled": true,
          "cert_pem_file": "/path/to/cert.pem",
          "key_pem_file": "/path/to/key.pem",
          "insecure_skip_verify": true
        }
      }
    }
  ]
}

Replace values in configuration above with your Kafka's. In v6 TLS for Kafka is configured using this object https://centrifugal.dev/docs/server/tls#unified-tls-config-object which will be used for all TLS configs in Centrifugo.

After that on DEBUG log level you will see some information what is used for TLS config construction. Maybe it can give us some glue and we can proceed from there. At least if it does not work – we will see in logs that all necessary files were really used to create TLS config.

[bb-ha-melech]

We will try it with V6 and debug. Thank you for the help :)