Open jaykatti opened 3 years ago
I am using the latest version 0.2.15. This seems to be a complete blocker. Can this please be taken-up on priority ?
Hi @jaykatti, can you provide an example state.yaml file to help reproduce?
Do note that ACLs do not update, they only add/remove. (so a change to an ACL should have one create and one delete).
Hello Shawn @devshawn , Thanks for your response !
For the below sample state.yaml, if I have to change the permission for describe_configs_topic to "DENY". I change the permission to "DENY" for that block and then run the kafka-gitops command for validate, plan. When I run the plan, I see that, the plan indicates "Delete" or "Remvove" for all the ACL's present in my system and not just the ones in the yaml file.
customUserAcls: User:kattijay: describe_configs_topic: name: topic1 pattern: LITERAL host: "" permission: ALLOW type: TOPIC operation: DESCRIBE_CONFIGS describe_topic: name: topic1 pattern: LITERAL host: "" permission: ALLOW type: TOPIC operation: DESCRIBE write_topic: name: topic1 pattern: LITERAL host: "" permission: ALLOW type: TOPIC operation: WRITE read_topic: name: topic1 pattern: LITERAL host: "" permission: ALLOW type: TOPIC operation: READ
host above is * for all the records.
Hello @devshawn , Any further update for me on this please ?
Hi @jaykatti, can you provide an example state.yaml file to help reproduce?
Do note that ACLs do not update, they only add/remove. (so a change to an ACL should have one create and one delete).
I have generated a yaml (initial state) for my existing cluster. Now, this includes a large list of users and ACL's. I will use this as my base configuration and start modifying it based on the requirement. Either update an ACL or delete or Add a new one. Say, I want to add a new ACL to the existing list, How do I go about this ? I do not want any of my existing ACL's to be deleted.
Kindly advise at the earliest as I need to get this wrapped up and start using it on our cluster.
Thanks, Jay.
Hi @jaykatti,
You have to have every ACL listed in your state file. So you need to take your existing cluster and make an initial state file with all topics / ACLs until you run a plan and it says there are no changes. Then, you can start adding/removing ACLs and it will only have the changes you want.
Or am I misunderstanding the situation?
Hello Shawn,
Yes, I have already completed the below steps -
I am running the plan on this yaml and it marks all the existing ACL's for deletions. This is where, I was confused. Ideally, I expected the plan to have nil deletions.
Can you pls. guide me on this ?
Thanks, Jay.
On Tue, Sep 21, 2021 at 6:17 PM Shawn Seymour @.***> wrote:
Hi @jaykatti https://github.com/jaykatti,
You have to have every ACL listed in your state file. So you need to take your existing cluster and make an initial state file with all topics / ACLs until you run a plan and it says there are no changes. Then, you can start adding/removing ACLs and it will only have the changes you want.
Or am I misunderstanding the situation?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/devshawn/kafka-gitops/issues/77#issuecomment-924142230, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5B75XOUJZD6Z5NNLEBEWDUDCVX5ANCNFSM5C5NFEPA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
-- Regards,
Hi @jaykatti
You're stating a customUserAcls
in the above text - do you have a matching users
in your state.yaml file as well?
Every customUserAcls
and customServiceAcls
is connected to a matching users
and service
respectively.
If you're only claiming the customUserAcls
in your state file without the matching users
definition, then the plan will disregard the customUserAcls
listed in your state file.
For example, if I were to add a customUserAcls
named Access_Bob
then I'd have in my state file:
users:
Access_Bob:
principal: User:Bob
customUserAcls:
Access_Bob:
read-kstream-topics:
name: kstream-
type: TOPIC
pattern: PREFIXED
host: "*"
operation: READ
permission: ALLOW
Hi @jaykatti
You're stating a
customUserAcls
in the above text - do you have a matchingusers
in your state.yaml file as well?Every
customUserAcls
andcustomServiceAcls
is connected to a matchingusers
andservice
respectively.If you're only claiming the
customUserAcls
in your state file without the matchingusers
definition, then the plan will disregard thecustomUserAcls
listed in your state file.For example, if I were to add a
customUserAcls
namedAccess_Bob
then I'd have in my state file:users: Access_Bob: principal: User:Bob customUserAcls: Access_Bob: read-kstream-topics: name: kstream- type: TOPIC pattern: PREFIXED host: "*" operation: READ permission: ALLOW
Oh !! Okay, Let me try that.
Thanks !, I will keep you posted.
Sorry, closed by mistake
Hi @jaykatti You're stating a
customUserAcls
in the above text - do you have a matchingusers
in your state.yaml file as well? EverycustomUserAcls
andcustomServiceAcls
is connected to a matchingusers
andservice
respectively. If you're only claiming thecustomUserAcls
in your state file without the matchingusers
definition, then the plan will disregard thecustomUserAcls
listed in your state file. For example, if I were to add acustomUserAcls
namedAccess_Bob
then I'd have in my state file:users: Access_Bob: principal: User:Bob customUserAcls: Access_Bob: read-kstream-topics: name: kstream- type: TOPIC pattern: PREFIXED host: "*" operation: READ permission: ALLOW
Oh !! Okay, Let me try that.
Thanks !, I will keep you posted.
Hello @fireydagostino @devshawn
I tried with that and it seems to be working. So many thanks !
Do we also need to list all the topics as part of the state.yaml ? I see, when I run the plan, it is marking the topics for deletion, even though, there are ACL's for those topics. Kindly confirm !
Thanks, Jay.
Yeah @jaykatti
The state file literally claims everything that should be present within the current state of your Kafka cluster.
Any topics/ACLs that are on Kafka, but not defined for in your state will be privy to deletion upon execution - and respectively the opposite for topics/Acls defined in your state but not present on Kafka.
Your state.yaml
file will need
In my production environment, we have some of these separated into specific directories/files pertaining to different teams/technologies and then utilize yq4
to merge all the information into the state.yaml
file.
Thanks a lot for the clarification. Would you please share a sample state.yaml from ur environment ?
Thanks and regards, Jay
On Tue, 28 Sep 2021, 7:13 pm fireydagostino, @.***> wrote:
Yeah @jaykatti https://github.com/jaykatti
The state file literally claims everything that should be present within the current state of your Kafka cluster.
Any topics/ACLs that are on Kafka, but not defined for in your state will be privy to deletion upon execution - and respectively the opposite for topics/Acls defined in your state but not present on Kafka.
Your state.yaml file will need
- default options / blacklisted topic prefixes
- topics
- users + customUserAcl
- services + customServiceAcl
In my production environment, we have some of these separated into specific directories/files pertaining to different teams/technologies and then utilize yq4to merge all the information into thestate.yaml` file.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/devshawn/kafka-gitops/issues/77#issuecomment-929223107, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5B75VRYZ4HD2T3OMI7YQ3UEHBBZANCNFSM5C5NFEPA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
I have generated a current state.yaml for my existing cluster. I changed some couple of users the permission as "DENY". So I was expecting the "plan" command to show me those many "updates", instead, it is including those records in the plan json file and marks everything else for "REMOVE".
Kindly advise at the earliest.
This currently is not allowing me to deploy onto our server, so we can include this into our process permanently.
Thanks, Jay.