search

flipp-oss / deimos

Framework to work with Kafka, Avro and ActiveRecord
Other
59 stars 22 forks source link

Confluent Cloud authentication #133

Closed iMacTia closed 2 years ago

iMacTia commented 2 years ago

I feel like this might a dummy question considering I'm the first one asking 😄... but we have a Confluent Cloud cluster and I'm struggling to find the correct configuration to instruct Deimos to connect to it.

First of all, the seed_brokers URL, I suspect that's what CC calls the "Boostrap Server"? And regarding authentication, I got a pair of ApiKey and ApiSecret for both the schema registry and the bootstrap server, but I can't find a configuration where to put these. It seems like these are supposed to go as Basic authentication, but all I can find is SSL certificate-based authentication in the documentation.

And should the support for such authentication not be a thing, would you be open to a PR that adds that?

Thanks in advance for the help 🙏!

dorner commented 2 years ago

Hiya!

I think the API key and secret represent SASL username and password based on this page: https://github.com/wishabi/ruby-kafka/blob/master/lib/kafka/client.rb#L57

This is supported in the underlying RubyKafka client:

https://github.com/wishabi/ruby-kafka/blob/master/lib/kafka/client.rb#L57

So yeah, definitely put in a PR and I'd be glad to add it in! You can look at the following two files:

https://github.com/flipp-oss/deimos/blob/master/lib/deimos/config/configuration.rb#L118 https://github.com/flipp-oss/deimos/blob/master/lib/deimos/config/phobos_config.rb#L36

It should pass everything through to the RubyKafka client correctly.

And yes I believe that seed_brokers corresponds to the bootstrap server.

iMacTia commented 2 years ago

Amazing, thanks for pointing me in the right direction 🙏 ! I'll give this a try and submit a PR if I can get it working 👍 !

iMacTia commented 2 years ago

Hey @dorner, I've opened a Draft PR (#134) with the progress I made so far. Unfortunately, I'm struggling with a compatibility issue on the avro_turf side: I've left additional info in the PR description and would love your input on it!