Enable OIDC use authentication with the kafka source

rightly commented 1 day ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When I use kafka with OAUTHBEARER mechanism. I faced this errors

ERROR vector::topology::builder: Configuration error. error=Sink "kafka": creating kafka producer failed: Client config error: Configuration property "sasl.oauthbearer.token.endpoint.url" not supported in this build: OAuth/OIDC depends on libcurl and OpenSSL which were not available at build time sasl.oauthbearer.token.endpoint.url
ERROR vector::topology::builder: Configuration error. error=Sink "kafka": creating kafka producer failed: Client config error: Configuration property "sasl.oauthbearer.client.id" not supported in this build: OAuth/OIDC depends on libcurl and OpenSSL which were not available at build time sasl.oauthbearer.client.id
ERROR vector::topology::builder: Configuration error. error=Sink "kafka": creating kafka producer failed: Client config error: Configuration property "sasl.oauthbearer.method" not supported in this build: OAuth/OIDC depends on libcurl and OpenSSL which were not available at build time sasl.oauthbearer.method

I think there was a patch to use oauthbearer, how do I fix this?

Configuration

kafka:
    bootstrap_servers: kafka-domain:10992
    librdkafka_options:
      sasl.mechanism: "OAUTHBEARER"
      sasl.oauthbearer.token.endpoint.url: "https://oauth-url"
      sasl.oauthbearer.client.id: "client"
      sasl.oauthbearer.client.secret: "secrets"
      sasl.oauthbearer.scope: "domain"
      sasl.oauthbearer.method: "OIDC"
      ssl.ca.location: "/etc/vector/ca.crt"
      security.protocol: "SASL_SSL"

Version

0.42.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

jszwedko commented 1 day ago

https://github.com/vectordotdev/vector/pull/21103 was intended to enable this, but it looks like there are some more changes needed. I can confirm that I receive the same error with the above configuration.

cc/ @zapdos26 in case you have any thoughts.

zapdos26 commented 1 day ago

So, I've been using the one I built for a while now. Let me check what possibly changed.

zapdos26 commented 1 day ago

Okay, so after testing, its due to the fact curl was changed to curl-sys. Apparently curl-sys is not a valid to enable to OAUTH/OIDC.

Building with curl should work.

jszwedko commented 1 day ago

Ah, I see, interesting. This may be a bug with rust-rdkafka or librdkafka that including curl-static is insufficient since statically compiling in libcurl should make it available. We try to avoid any dynamic linking in Vector to reduce dependencies on the host system.

It sounds like a workaround would be to build Vector yourself with the curl feature rather than curl-static for rust-rdkafka.

jszwedko commented 1 day ago

Thanks for looking at that quickly @zapdos26 !

rightly commented 1 day ago

@zapdos26 , @jszwedko Thanks!!

Could you provide me with a package(build?)?

vectordotdev / vector