Support Kafka component GSSAPI authentication on musl and aarch64 platforms

cshimegi commented 1 year ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm trying to use Kafka component GSSAPI authentication which is supported according to the librdkafka document. However, I get an error message saying that it is unsupported.

sasl.mechanism document

Screen Shot 2023-03-09 at 15 59 31

Error message

2023-03-08T08:45:41.961871Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-03-08T08:45:41.961948Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-03-08T08:45:41.961989Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-03-08T08:45:41.963314Z ERROR vector::topology: Configuration error. error=Source "test_kafka": Could not create Kafka consumer: Client creation error: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER

Configuration

[sources.test_kafka]
  type = "kafka"
  auto_offset_reset = "latest"
  bootstrap_servers = "server1:9092;..."
  commit_interval_ms = 5_000
  fetch_wait_max_ms = 100
  group_id = "groupId"
  session_timeout_ms = 120_000
  socket_timeout_ms = 120_000
  topic_key = "topic"
  topics = ["topic1"]

    [sources.test_kafka.librdkafka_options]
    "security.protocol" = "SASL_SSL"
    "ssl.keystore.location" = "/etc/truststore/keystore1"
    "ssl.keystore.password" = "password"
    "sasl.kerberos.service.name" = "service_name"
    "sasl.kerberos.keytab" = "/etc/keytabs/keytab1"
    "sasl.kerberos.principal" = "principle1"
    "sasl.mechanism" = "GSSAPI"

[sinks.print]
type = "console"
inputs = ["test_kafka"]
encoding.codec = "json"

[api]
enabled = true
address = "0.0.0.0:8686"

Version

vector 0.28.1 (aarch64-unknown-linux-musl ff15924 2023-03-06)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

zamazan4ik commented 1 year ago

First guess: this one could be implemented just with enabling gssapi feature in rdkafka dependency.

jszwedko commented 1 year ago

Indeed, it looks like we currently only enable GSSAPI for GNU x86_64 builds but not for aarch64 or musl builds. I assume there was some build issue blocking enabling on these other platforms, but I'm not sure what it might be. I'll open a PR adding the feature to the other platforms and see if it fails and, if so, why.

jszwedko commented 1 year ago

Opened a draft PR, https://github.com/vectordotdev/vector/pull/16738, to see what happens.

jszwedko commented 1 year ago

Welp, it looks like every build except x86_64-unknown-linux-gnu fails. Including the log files here for posterity since GHA will delete them after some time period.

x86_64-unknown-linux-musl seems to fail due to undefined symbols. x86_64-unknown-linux-musl.log
aarch64-unknown-linux-gnu seems to fail because libsasl2_sys is not correctly building for aarch64, but is instead still building for x86_64. aarch64-unknown-linux-gnu.log
aarch64-unknown-linux-musl fails for both the arch and musl reasons. aarch64-unknown-linux-musl.log
armv7-unknown-linux-gnueabihf fails for same reason the aarch64 gnu build fails. armv7-unknown-linux-gnueabihf.log
armv7-unknown-linux-musleabihf fails for the same reason the aarch64 musl build fails. armv7-unknown-linux-musleabihf.log

If anyone digs into this, I think the place to start is with investigating why the sasl2_sys crate doesn't correctly cross-compile on other architectures than x86_64. It seems to be due to https://github.com/MaterializeInc/rust-sasl/issues/46 which does actually have a PR already so we may just need to have that be merged in.

KareNorthter commented 6 months ago

try complie librdkafka （with ssl and sasl）

export CC=aarch64-linux-gnu-gcc export CXX=aarch64-linux-gnu-g++ ./configure --arch=aarch64 --prefix=/usr/aarch64-linux-gnu --enable-ssl --enable-sasl

please see： https://github.com/confluentinc/librdkafka/wiki/Using-SSL-with-librdkafka https://github.com/confluentinc/librdkafka/wiki/Using-SASL-with-librdkafka

vectordotdev / vector