Kerberized Kafka Broker transport failure

[tony-cocco]

After having successfully integrated with a kafka cluster, I am migrating to a kerberized one. When installing node-rdkafka I verified the following:

checking for libsasl2 (by pkg-config)... ok
checking for libsasl2 (by compile)... ok

I updated my configuration object to be (I explicitly set the kinit cmd):

{
    "client.id": "myid",
    "metadata.broker.list": "broker1.domain:6667,broker2.domain:6667",
    "security.protocol": "sasl_plaintext",
    "sasl.kerberos.keytab": "path/to/my/keytab",
    "sasl.kerberos.principal": "principalname",
    "sasl.kerberos.kinit.cmd": "kinit -S kafka/domain -k -t path/to/my/keytab principalname",
    "dr_cb": true
  }

After attempting to post to a topic I get the following:

{ Error: Local: Broker transport failure
    at Function.createLibrdkafkaError [as create] (/opt/arrow/dev/node_modules/node-rdkafka/lib/error.js:254:10)
    at /opt/arrow/dev/node_modules/node-rdkafka/lib/client.js:340:42
  message: 'Local: Broker transport failure',
  code: -195,
  errno: -195,
  origin: 'kafka' }

I can successfully run my kinit command and use klist to verify a valid ticket. Setting the config option debug: true gave me no additional insight.

Any help would be appreciated.

[webmakersteve]

When you set the debug option you need to make sure to specify string options for the type of debug information you want. If you specify 'all', you will get all of the info. However, you then need to make sure you are consuming the information it is telling you.

Here is a little snippet I usually use (I use bunyan for logging)

producer
  .on('event.log', function(event) {
      const loggedEvent = {
        severity: event.severity,
        fac: event.fac
      };

      if (event.severity >= 7) {
        logger.debug(loggedEvent, event.message);
      } else if (event.severity === 6 || event.severity === 5) {
        logger.info(loggedEvent, event.message);
      } else if (event.severity === 4) {
        logger.warn(loggedEvent, event.message);
      } else if (event.severity > 0) {
        logger.error(loggedEvent, event.message);
      } else {
        logger.fatal(loggedEvent, event.message);
      }
    })

See if that gives any additional insight.

Additionally, does your code work when you do not use kerberos?

[tony-cocco]

My code works when I point it to a non-kerberized broker. I can add a handler in for the event log to see what the output is.

[tony-cocco]

Well, the callback got me some output!

I see some other stuff, but notably:

{ severity: 7, fac: 'SASL' } '[thrd:sasl_plaintext://mydomain:6667/bootstrap]: sasl_plaintext://mydomain:6667/bootstrap: My supported SASL mechanisms: EXTERNAL ANONYMOUS'
{ severity: 2, fac: 'LIBSASL' } '[thrd:sasl_plaintext://mydomain:6667/bootstrap]: sasl_plaintext://mydomain:6667/bootstrap: No worthy mechs found'
{ severity: 7, fac: 'BROKERFAIL' } '[thrd:sasl_plaintext://mydomain:6667/bootstrap]: sasl_plaintext://mydomain:6667/bootstrap: failed: err: Local: Authentication failure: (errno: Invalid argument)'
{ severity: 3, fac: 'FAIL' } '[thrd:sasl_plaintext://mydomain:6667/bootstrap]: sasl_plaintext://mydomain:6667/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start (-4)): SASL(-4): no mechanism available: No worthy mechs found'

Something is most definitely not configured correctly.

[webmakersteve]

I'm not sure how librdkafka will do path resolution running as a node addon actually. I would start there first, maybe. Perhaps try using full absolute paths for your keytab and see if that helps?

[RainofTerra]

https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka can you do the kafkacat example and see if that works?

[tony-cocco]

OK, so a few updates.

I installed 2 more packages: cyrus-sasl-plain and cyrus-sasl-gssapi. I thought these were deps from the already installed cyrus-sasl-devel but they weren't.

Now I'm getting:

{ Error: Local: Authentication failure
    at Error (native)
  origin: 'local',
  message: 'authentication failure',
  code: -1,
  errno: -1,
  stack: 'Error: Local: Authentication failure\n    at Error (native)' }

More detailed debug: { severity: 2, fac: 'LIBSASL' } '[thrd:sasl_plaintext://mydomain:6667/bootstrap]: sasl_plaintext://mydomain:6667/1003: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Matching credential not found (filename: /tmp/krb5cc_990))'

I just installed kafkacat and will try that.

[tony-cocco]

Kafkacat has also stopped complaining about no sasl mechanisms. However, I get:

Broker: Topic authorization failed

Is this gonna be a kafka configuration issue?

[RainofTerra]

Do you have ACLs configured on the broker and is the principal you're kinit'd as in those ACLs?

[tony-cocco]

One of my coworkers informed me my kinit was incorrect. I had to remove the -S flag and param. After that, both node-rdkafka and kafkacat are giving me the same topic authorization failed.

We looked at the ACLs and they appear to be open. We even created a new one, albeit with a stacktrace that might be a misconfiguration...

My team doesn't have admin so we will have to check back tomorrow when they are in the office. It would seem all signs now point to a bad kafka configuration and not an issue with node-rdkafka itself, so I will close this ticket. Thanks for all the help! I'll report back when I get an update.

Update: Server admin addressed a security configuration issue. He wouldn't provide details, but I am successfully talking to kerberized kafka.

[samatree]

Tony-cocco,

What did you admin change on server side?

We are getting the same error...

bootstrap: failed: err: Local: Authentication failure: (errno: Invalid argument)

The kafka kerberos logs show successful authentication but kafkacat/librdkafka still reports error.

Thanks.

[tony-cocco]

@raneprashant I wasn't getting an invalid argument. That to me should mean the kerberos log should not be showing success.

All I can tell you is he adjusted the kafka configuration. For ~stupid~ political reasons he wouldn't share the details.