search

devshawn / kafka-gitops

🚀Manage Apache Kafka topics and generate ACLs through a desired state file.
https://devshawn.github.io/kafka-gitops
Apache License 2.0
320 stars 71 forks source link

Plans for Move to `confluent` vs. `ccloud` CLI? #85

Open AllenSandersUS opened 2 years ago

AllenSandersUS commented 2 years ago

Given that Confluent is planning to sunset ccloud and has merged its functionality into the latest version of the confluent CLI (https://docs.confluent.io/ccloud-cli/current/migrate.html), are there plans to migrate this tool to leverage the confluent CLI behind-the-scenes instead?

tball-dev commented 2 years ago

I've got a commit ready for this, but confluent has changed the way IDs are returned by the new CLI. For confluent iam service-account list -o json they have removed the numeric ID and replaced the ID field with the alpha-numeric ID. Currently, kafka-gitops generates the resource_principal on ACLs with the numeric ID returned by ccloud service-account list -o json. For existing Confluent Cloud clusters, it will think that all ACLs have been changed.

Update: I have a ticket open with Confluent to discuss

Update 2: Got confirmation from Confluent support that, as of updating this comment (2/11/2022), there is no way to retrieve numeric IDs with the new confluent CLI. (their "workaround" was to grab the numeric ID from a URL in confluent.cloud. Obviously not a good solution for automation.) In addition, alpha-numeric IDs are NOT supported in ACL operations. They are aware of this discrepancy, but I was not given any sort of timeline on a fix. Hopefully they get this fixed before May 9th.

FloEc commented 2 years ago

According to the support team, it is still possible to obtain the IDs directly via the API endpoint https://api.confluent.cloud/service_accounts as a json. This will continue to work for a certain time. At first glance, it looks like this contains the correct numeric ID. Are there any opinions or vetoes on this?

ghost commented 2 years ago

I would be interested with some updates on this, I have created a modified version of kafka-gitops that uses confluent instead, and I am willing to redeploy my infrastructure with the new IDs. However I am less willing to do that if upstream (here) will continue to use the old numeric id field (meaning that eventually we will have to migrate back)

Will kafka-gitops migrate to alphanumeric ids or will it try to continue using numeric ids?

tball-dev commented 2 years ago

It looks like creating an ACL with a service-account alpha-numeric id as the principal only works with confluent kafka acl create. Passing the principal id as User:sa-12345a to the AdminClient or kafka-acls.sh creates the ACL with the alpha-numeric id. ACLs with this principal Id do not work. If you create an ACL with confluent kafka acl list --service-account sa-12345a and retrieve it with the AdminClient, the principal has the numeric id. Confluent must be doing a transformation from the alpha-numeric id to the numeric id at some other layer.

FloEc commented 2 years ago

As a workaround you can use the following script as a ccloud alternative: https://gist.github.com/FloEc/c7c38c4ba24ef3c6b9c5c34da3591b77

This script can be placed in the path and called with "ccloud service-account list -o json". The output corresponds exactly to the result that ccloud would return. This should make it possible to create ACLs even after May 9th. Confluent could not tell us an exact date for how long this workaround will work, but probably until alphanumeric ids are supported.

michael-demian commented 2 years ago

did we manage to get a workaround or updates on this one since ccloud CLI has already been deprecated since May 9th and now the pipeline is broken :(

FloEc commented 2 years ago

@michael-demian you can use https://gist.github.com/FloEc/c7c38c4ba24ef3c6b9c5c34da3591b77 as an alternative ccloud cli to get service account ids from confluent cloud.