Kafka source with GSSAPI SASL unexpected error.

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hi, everyone, hope you're doing great :)

I've faced the strange error about Kerberos authentication in Kafka using librdkafka_options with the config provided below. Keytab and username are 100% verified and work properly. Klist, kinit also reply correctly. Can you suggest, please? Vector is 0.35.0 x86_64 on Linux. Error: ``` 2024-03-19T13:56:50.439123Z ERROR librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka.domain.ru:9092/bootstrap]: sasl_plaintext://kafka.domain.ru:9092/bootstrap: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0)) 2024-03-19T13:56:50.439193Z ERROR librdkafka: librdkafka: FAIL [thrd:sasl_plaintext://kafka.domain.ru:9092/bootstrap]: sasl_plaintext://kafka.domain.ru:9092/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start (-1)): SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0)) (after 1ms in state AUTH_REQ)


### Configuration

```text
sources:
  src_kafka:
    type: "kafka"
    bootstrap_servers: "kafka1.domain.ru:9092, kafka2.domain.ru:9092, kafka3.domain.ru:9092"
    group_id: "kafka-consumer"
    topics:
      - test-topic
    librdkafka_options:
      debug: "all"
      security.protocol: "SASL_PLAINTEXT"
      sasl.mechanism: "GSSAPI"
      sasl.kerberos.service.name: "srv_kfk"
      sasl.kerberos.keytab: "/etc/vector/srv.keytab"
      sasl.kerberos.principal: "srv_kfk@DOMAIN.RU"

Version

vector 0.36.1 (x86_64-unknown-linux-gnu 2857180 2024-03-11 14:32:52.417737479)

Debug Output

2024-03-19T15:16:44.988103Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-03-19T15:16:44.988138Z  INFO vector::app: Log level is enabled. level="trace"
2024-03-19T15:16:44.988189Z DEBUG vector::app: messaged="Building runtime." worker_threads=2
2024-03-19T15:16:44.988263Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE
2024-03-19T15:16:44.989568Z  INFO vector::app: Loading configs. paths=["vector.yaml"]
2024-03-19T15:16:44.990406Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2024-03-19T15:16:44.991537Z DEBUG vector::topology::builder: Building new source. component=src_kafka
2024-03-19T15:16:44.991966Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: librdkafka: librdkafka: SASL [thrd:app]: Selected provider Cyrus for SASL mechanism GSSAPI
2024-03-19T15:16:44.993097Z DEBUG librdkafka: librdkafka: SASLREFRESH [thrd:main]: Refreshing Kerberos ticket with command: kinit -R -t "/etc/vector/srv_kafka.keytab" -k srv_kafka@DOMAIN.RU || kinit -t "/etc/vector/srv_kafka.keytab" -k srv_kafka@DOMAIN.RU
2024-03-19T15:16:44.993176Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: librdkafka: librdkafka: INIT [thrd:app]: librdkafka v2.3.0 (0x20300ff) vector#consumer-1 initialized (builtin.featuresgzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CMAKE GNU GNU PKGCONFIG HDRHISTOGRAM ZLIB ZSTD LIBDL PLUGINS SSL SASL_SCRAM SASL_OAUTHBEARER SASL_CYRUS CRC32C_HW SNAPPY SOCKEM, debug 0x200)
2024-03-19T15:16:44.993196Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::client: Create new librdkafka client 0x7fe83df11400
2024-03-19T15:16:44.993283Z TRACE rdkafka::consumer::stream_consumer: Starting stream consumer wake loop: 0x7fe83df11400
2024-03-19T15:16:45.019150Z DEBUG librdkafka: librdkafka: SASLREFRESH [thrd:main]: First kinit command finished: waking up broker threads
2024-03-19T15:16:45.019198Z DEBUG librdkafka: librdkafka: SASLREFRESH [thrd:main]: Kerberos ticket refreshed in 25ms
2024-03-19T15:16:45.019259Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroying topic partition list: 0x7fe83dfd0730
2024-03-19T15:16:45.019271Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroyed topic partition list: 0x7fe83dfd0730
2024-03-19T15:16:45.019319Z DEBUG vector::topology::builder: Building new transform. component=parse_logs
2024-03-19T15:16:45.020055Z DEBUG vector::topology::builder: Building new sink. component=print
2024-03-19T15:16:45.020318Z  INFO vector::topology::running: Running healthchecks.
2024-03-19T15:16:45.020329Z DEBUG vector::topology::running: Connecting changed/added component(s).
2024-03-19T15:16:45.020339Z DEBUG vector::topology::running: Configuring outputs for source. component=srv_kafka
2024-03-19T15:16:45.020355Z DEBUG vector::topology::running: Configuring output for component. component=srv_kafka output_id=None
2024-03-19T15:16:45.020363Z DEBUG vector::topology::running: Configuring outputs for transform. component=parse_logs
2024-03-19T15:16:45.020368Z DEBUG vector::topology::running: Configuring output for component. component=parse_logs output_id=None
2024-03-19T15:16:45.020377Z DEBUG vector::topology::running: Connecting inputs for transform. component=parse_logs
2024-03-19T15:16:45.020388Z DEBUG vector::topology::running: Adding component input to fanout. component=parse_logs fanout_id=srv_kafka
2024-03-19T15:16:45.020397Z DEBUG vector::topology::running: Connecting inputs for sink. component=print
2024-03-19T15:16:45.020405Z DEBUG vector::topology::running: Adding component input to fanout. component=print fanout_id=parse_logs
2024-03-19T15:16:45.020425Z DEBUG vector::topology::running: Spawning new source. key=srv_kafka
2024-03-19T15:16:45.020441Z DEBUG vector::topology::running: Spawning new transform. key=parse_logs
2024-03-19T15:16:45.020449Z TRACE vector::topology::running: Spawning new sink. key=print
2024-03-19T15:16:45.020486Z  INFO vector: Vector has started. debug="false" version="0.36.1" arch="x86_64" revision="2857180 2024-03-11 14:32:52.417737479"
2024-03-19T15:16:45.020497Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-03-19T15:16:45.021042Z  INFO vector::topology::builder: Healthcheck passed.
2024-03-19T15:16:45.021066Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: vector::topology::builder: Source pump supervisor starting.
2024-03-19T15:16:45.021085Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: vector::topology::builder: Source pump starting.
2024-03-19T15:16:45.021092Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: vector::topology::builder: Source starting.
2024-03-19T15:16:45.021219Z DEBUG transform{component_kind="transform" component_id=parse_logs component_type=remap}: vector::topology::builder: Synchronous transform starting.
2024-03-19T15:16:45.021229Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::topology::builder: Sink starting.
2024-03-19T15:16:45.022929Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: vector: Beep.
2024-03-19T15:16:45.022966Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::utilization: utilization=0.009277043927337858
2024-03-19T15:16:45.023707Z DEBUG librdkafka: librdkafka: AUTH [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
2024-03-19T15:16:45.024303Z DEBUG librdkafka: librdkafka: SASLMECHS [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Broker supported SASL mechanisms: PLAIN,GSSAPI
2024-03-19T15:16:45.024332Z DEBUG librdkafka: librdkafka: AUTH [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
2024-03-19T15:16:45.024339Z DEBUG librdkafka: librdkafka: SASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Initializing SASL client: service name srv_kfk, hostname kafka1.domain.ru, mechanisms GSSAPI, provider Cyrus
2024-03-19T15:16:45.024395Z DEBUG librdkafka: librdkafka: SASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: My supported SASL mechanisms: GSSAPI EXTERNAL
2024-03-19T15:16:45.024407Z DEBUG librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: GSSAPI client step 1
2024-03-19T15:16:45.025786Z ERROR librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0))
2024-03-19T15:16:45.025826Z ERROR librdkafka: librdkafka: FAIL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start (-1)): SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0)) (after 1ms in state AUTH_REQ)
2024-03-19T15:16:45.026047Z ERROR source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: Authentication (Local: Authentication failure): sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start (-1)): SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0)) (after 1ms in state AUTH_REQ)
2024-03-19T15:16:45.026084Z ERROR source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2024-03-19T15:16:45.280511Z DEBUG librdkafka: librdkafka: AUTH [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
2024-03-19T15:16:45.281009Z DEBUG librdkafka: librdkafka: SASLMECHS [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Broker supported SASL mechanisms: PLAIN,GSSAPI
2024-03-19T15:16:45.281019Z DEBUG librdkafka: librdkafka: AUTH [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
2024-03-19T15:16:45.281025Z DEBUG librdkafka: librdkafka: SASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Initializing SASL client: service name kafka_service, hostname kafka1.domain.ru, mechanisms GSSAPI, provider Cyrus
2024-03-19T15:16:45.281086Z DEBUG librdkafka: librdkafka: SASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: My supported SASL mechanisms: GSSAPI EXTERNAL
2024-03-19T15:16:45.281095Z DEBUG librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: GSSAPI client step 1
2024-03-19T15:16:45.282282Z ERROR librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0))
2024-03-19T15:16:45.282329Z ERROR librdkafka: librdkafka: FAIL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start (-1)): SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0)) (after 1ms in state AUTH_REQ, 1 identical error(s) suppressed)
2024-03-19T15:16:45.282364Z ERROR source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: Authentication (Local: Authentication failure): sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start (-1)): SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0)) (after 1ms in state AUTH_REQ, 1 identical error(s) suppressed)
2024-03-19T15:16:46.022153Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: vector: Beep.
2024-03-19T15:16:46.365288Z DEBUG librdkafka: librdkafka: AUTH [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
2024-03-19T15:16:46.365932Z DEBUG librdkafka: librdkafka: SASLMECHS [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Broker supported SASL mechanisms: PLAIN,GSSAPI
2024-03-19T15:16:46.365947Z DEBUG librdkafka: librdkafka: AUTH [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
2024-03-19T15:16:46.365973Z DEBUG librdkafka: librdkafka: SASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: Initializing SASL client: service name srv_kfk, hostname kafka1.domain.ru, mechanisms GSSAPI, provider Cyrus
2024-03-19T15:16:46.366053Z DEBUG librdkafka: librdkafka: SASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: My supported SASL mechanisms: GSSAPI EXTERNAL
2024-03-19T15:16:46.366066Z DEBUG librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: GSSAPI client step 1
2024-03-19T15:16:46.369137Z ERROR librdkafka: librdkafka: LIBSASL [thrd:sasl_plaintext://kafka1.domain.ru:9092/bootstrap]: sasl_plaintext://kafka1.domain.ru:9092/bootstrap: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KEYRING:persistent:0))
2024-03-19T15:16:47.022295Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: vector: Beep.
2024-03-19T15:16:48.021497Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: vector: Beep.
2024-03-19T15:16:48.690880Z  INFO source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: vector::signal: Signal received. signal="SIGINT"
2024-03-19T15:16:48.690933Z  INFO vector: Vector has stopped.
2024-03-19T15:16:48.691051Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: rdkafka::util: Destroying topic partition list: 0x7fe83a4e0120
2024-03-19T15:16:48.691062Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}:kafka_source: rdkafka::util: Destroyed topic partition list: 0x7fe83a4e0120
2024-03-19T15:16:48.691109Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: vector::topology::builder: Source pump finished normally.
2024-03-19T15:16:48.691142Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: vector::topology::builder: Source pump supervisor task finished normally.
2024-03-19T15:16:48.691185Z DEBUG transform{component_kind="transform" component_id=parse_logs component_type=remap}: vector::topology::builder: Synchronous transform finished normally.
2024-03-19T15:16:48.691201Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::topology::builder: Sink finished normally.
2024-03-19T15:16:48.691459Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroying queue: 0x7fe83e041560
2024-03-19T15:16:48.691468Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroyed queue: 0x7fe83e041560
2024-03-19T15:16:48.691471Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::consumer::base_consumer: Destroying consumer: 0x7fe83df11400
2024-03-19T15:16:48.691482Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroying topic partition list: 0x7fe83dfd0680
2024-03-19T15:16:48.691485Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroyed topic partition list: 0x7fe83dfd0680
2024-03-19T15:16:48.691912Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::consumer::base_consumer: Consumer destroyed: 0x7fe83df11400
2024-03-19T15:16:48.691920Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroying client: 0x7fe83df11400
2024-03-19T15:16:48.691929Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: librdkafka: librdkafka: DESTROY [thrd:app]: Terminating instance (destroy flags none (0x0))
2024-03-19T15:16:48.691981Z DEBUG librdkafka: librdkafka: DESTROY [thrd:main]: Destroy internal
2024-03-19T15:16:48.691992Z DEBUG librdkafka: librdkafka: DESTROY [thrd:main]: Removing all topics
2024-03-19T15:16:48.692288Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="srv_kafka" time_remaining="59 seconds left"
2024-03-19T15:16:48.692399Z TRACE source{component_kind="source" component_id=srv_kafka component_type=kafka}: rdkafka::util: Destroyed client: 0x7fe83df11400
2024-03-19T15:16:48.692415Z DEBUG source{component_kind="source" component_id=srv_kafka component_type=kafka}: vector::topology::builder: Source finished normally.
2024-03-19T15:16:48.692438Z TRACE rdkafka::consumer::stream_consumer: Shut down stream consumer wake loop: 0x7fe83df11400

Example Data

No response

Additional Context

No response

References

No response

vectordotdev / vector