[Question] Out of curiosity, is `ssl.ca.location` now necessary to connect with Event Streams?

[ericqqqqq]

I used kafka-node(now does not work) to connect with IBM Event Streams. The full configuration was

{
      kafkaHost: kafka_brokers_sasl,
      sslOptions: { rejectUnauthorized: false },
      sasl: {
        mechanism: "plain",
        username: user,
        password: password
      }
};

However, when looking at the nodejs sample that uses node-rdkafka, we need to pass in an extra value: ssl.ca.location comparing to kafka-node.

Saw this answer and saw that ssl.ca.location is needed to enable the Kafka client to verify the broker keys.

Was cert unnecessary for Event Streams before then becomes necessary recently?

Out of curiosity, I also wondering why kafka-node stopped working with Event Streams.

Thanks

[mimaison]

Our recommended Node.js client is node-rdkafka. We don't test explicitly with kafka-node. That said, we have some users running kafka-node and it used to work. I'm also not aware of a recent change that could have broken it. Can you share some logs to help us understand what's wrong with kafka-node?

To go back to your question, yes ssl.ca.location is required with node-rdkafka and always have been. We also don't recommend disabling any TLS validations (rejectUnauthorized: false) as these can allow a range of attacks that can compromise the security of your data.

[ericqqqqq]

Hi @mimaison . Thanks for the reply.

I was asking about kafka-node, as I remembered that kafka-node was recommended by IBM Event Stream before(if I recall it right). These days, I found the kafka-node stopped working and I found this documentation, but not sure why these unsupported clients section is commented out.

With that said, when I have configuration:

{
      kafkaHost: kafka_brokers_sasl,
      sslOptions: { rejectUnauthorized: false },
      sasl: {
        mechanism: "plain",
        username: user,
        password: password
      }
};

I get the errors:

{ TimeoutError: Request timed out after 30000ms
    at new TimeoutError (/path/to/project/node_modules/kafka-node/lib/errors/TimeoutError.js:6:9)
    at Timeout.timeoutId._createTimeout [as _onTimeout] (/path/to/project/node_modules/kafka-node/lib/kafkaClient.js:1012:16)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10) message: 'Request timed out after 30000ms' }

However, I found a workaround here. And what we discussed there is the error we are facing. If I pass servername under sslOptions, I can successfully connect to IBM Event Streams

{
      kafkaHost: kafka_brokers_sasl,
      sslOptions: {
           rejectUnauthorized: false,
           servername: one of the kafka broker server name
      },
      sasl: {
        mechanism: "plain",
        username: user,
        password: password
      }
};

However, I need the luck to have a leader broker passed in. Therefore, I suppose it is not a workaround for the production environment. Any ideas what is going wrong with the kafka-node?

[mimaison]

I guess you were previously using the Classic plan. This plan was retired earlier this year.

All our current plans (since early 2019) require Kafka clients to use SNI (Server Name Indication). In kafka-node this is enabled via the servername setting. This is not a workaround but the correct configuration. The Classic plan did not require SNI.

[ericqqqqq]

Thanks @mimaison

You are right, I was using the Classic plan. I am trying to migrate the project to use the standard plan. I thought using node-rdkafka is the proper way until you mentioned that adding servername is the correct configuration instead of a workaround.

However, I have a concern, does the leader broker change? If I do not provide a leader broker, I would get NotLeaderForPartition. If that is the case, I will need to carefully handle that situation.

Thanks

[mimaison]

I don't know kafka-node very well, but yes the fact that you can only provide a single servername is likely to be problematic. I see there's an open PR to address this: https://github.com/SOHU-Co/kafka-node/pull/1375.

Our recommendation is still to use node-rdkafka. This is a client we continuously test and know works well.

[ericqqqqq]

Thank you very much, I think I will implement another API with node-rdkafka for my project.

[mimaison]

If I've answered your questions, can we close this ticket?

[ericqqqqq]

Yes.