Could JulieOps operate managed connectors in Confluent Cloud

[purbon]

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[shan916]

@purbon does Julie currently not support managed connectors in Confluent Cloud. This is a feature we would like to use.

[purbon]

currently not, but happily i would get contributions or if you are confluent customer happy to get engaged privately.

On Tue, 15 Feb 2022, 19:43 Zeeshan Khaliq @.***> wrote:

@purbon https://github.com/purbon does Julie currently not support managed connectors in Confluent Cloud. This is a feature we would like to use.

— Reply to this email directly, view it on GitHub https://github.com/kafka-ops/julie/issues/446#issuecomment-1040644884, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQXPDZPKOQHSXM5N7FTNTU3KNEJANCNFSM5NVAZ4OQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[shan916]

@purbon We are a Confluent Cloud customer and we'd like to talk about it.

[purbon]

@shan916 reach out to your account team, and we might find a solution!

[purbon]

fyi, https://docs.confluent.io/cloud/current/api.html#operation/createConnectv1Connector would be the apis to use.

[MichaelBeausoleil]

@purbon is there a list somewhere of known issues/reasons why it currently does not work so if someone wanted to contribute they could make sure they are handling those scenarios?

[purbon]

@MichaelBeausoleil just not implemented, we would need to provide something in this line:

• Build a provider factory (or similar) for selecting which API to use when managing connectors.
• Allow configuration for it, so people can decide if managing connectors on prem or in cloud.

Ideally.

Improve as well config, so confluent cloud provider for ACLs is selected automatically as well as connector API for confluent cloud if confluent cloud mode is given.

Does this makes sense?

[MichaelBeausoleil]

@purbon yes I believe that makes sense.

As a heads up, my particular case was trying to setup a managed RabbitMQ Source Connector and I had opened a ticket with Confluent around clarification on how to provide the Key Store and Trust Store through the Confluent Cloud Create Connector API and they said it wasn't supported. So this will be an issue for anyone trying to implement managing the connectors with Confluent Cloud.

Confluent support request (#105392)

I am trying to figure out how we can utilize the JulieOps tool (https://github.com/kafka-ops/julie) to configure a managed RabbitMQ Source Connector and one of the pieces that I am trying to figure out is what we are supposed to be supplying for the location in the following RabbitMQ Source Connector configuration.

• rabbitmq.https.ssl.truststore.location
• rabbitmq.https.ssl.keystore.location

I am not seeing anything in the RabbitMQ Source Connector configuration properties documentation stating how we should refer to the files. Can you please clarify what type of value/reference is supposed to be provided in these fields when configuring a RabbitMQ Source Connector through the Confluent Cloud Create a Connector API?

---

Response from Confluent:

At present, we do not have a documented way to specify the trust store and key store via the Confluent Cloud Connectors(v1) API. I am working internally with our Connect Engineering colleagues to look further into this. For time being, you may have to use the Confluent UI to update the connector configurations with the keystore and trust store. I will provide further updates before the end of day on Tuesday.

[miller45]

@shan916 @purbon Creating confluent cloud managed connects works with julie-ops. I tested at least the creation of one successfully today (data-gen). You do it basically the same as with on premise connect clusters, but you have to spice up the configuration file like this:

# this is not a full julie-ops file but only 
# add api key config for giving some context
ccloud.cloud.api.key=<YOUR-CLOUD-KEY>
ccloud.cloud.api.secret=<YOUR-CLOUD-SECRET>
#local kafka cluster
platform.servers.connect.0=connect:http://localhost:8083
#cloud kafka cluster
platform.servers.connect.1=cloud:https://api.confluent.cloud/connect/v1/environments/<YOUR-ENV-ID>/clusters/<YOUR-CLUSTER-ID>
platform.servers.basic.auth.0=cloud@<YOUR-CLOUD-KEY>:<YOUR-CLOUD-SECRET>`

.0 entry is for a local kafka connect cluster and the .1 entry for a cloud kafka connect cluster The secret sauce is enhancing you connect url with full path (see platform.servers.connect.1) and adding the authentication information (see platform.servers.basic.auth.0)

The config artefact looks like the same as the configs for local connect clusters, but the "name" field is required by cloud connect, while a local kafka connect cluster does not need it.