Acquiring a deleted IAMCustomRole doesn't enable the custom role

[maqiuyujoyce]

Describe the bug When I use Config Connector to create, delete, and recreate the same IAMCustomRole, I will get an unusable IAMCustomRole with deleted field set to true:

$ gcloud iam roles describe recreationtest --project my-project
deleted: true
etag: ======
includedPermissions:
- compute.backendServices.get
name: projects/my-project/roles/recreationtest
stage: GA
title: KCC Test

I think the problem is when I delete the IAMCustomRole, the GCP resource doesn't disappear, but the status of it is changed from enabled to deleted. Then if I recreate the same IAMCustomRole, Config Connector will acquire the deleted role, but not updating the status of it.

ConfigConnector Version 1.12.1

To Reproduce Steps to reproduce the behavior:

1. Create an IAMCustomRole using the following YAML snippet.
2. Delete the IAMCustomRole after it is created successfully.
3. Recreate the IAMCustomRole using the same YAML snippet.

YAML snippets:

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMCustomRole
metadata:
  name: recreationtest
spec:
  title: KCC Test
  permissions:
    - compute.backendServices.get

[dpetersen]

This has recently bitten us, too. With Helm charts that have IAMCustomRole objects in our ecosystem, we're having to flag them with the cnrm.cloud.google.com/deletion-policy: abandon annotation to ensure that the charts can be uninstalled and reinstalled without requiring manual intervention.

[toumorokoshi]

Thanks for the report. I'm looking to see if there's any prior art around this issue internally since this is from a while back. And yes, agreed this is most likely due to the flag for deletion behavior of IAM, combined with the fact that marked-for-deletion roles are still queryable in the API.

Thanks for also stating that there is a workaround.

[loudej]

Hi @toumorokoshi thought I'd add a note that I've just gotten bit by this as well while writing kpt packages.

I'll use the workaround above. Only info to add is the first symptom of the bug is very delayed: the apply/delete/apply works perfectly well on the kind: IAMCustomRole resource itself, and the Ready condition becomes True immediately on the second apply.

[toumorokoshi]

Thanks for the follow-up! Unfortunately we haven't been able to make headway on this issue.

We're currently using Terraform to help out, and it suffers from the same issue: https://github.com/hashicorp/terraform-provider-google/issues/9066.

There's no ETA yet on resolving it, but I'll talk to the team and update the issue once we do have a timeline.

[coderanger]

Still happening in 2024, as expected since the underlying tf bug is still open.

[maqiuyujoyce]

@yuwenma fyi if we believe acquiring an deleted IAMCustomRole should undelete it, then this can be a good candidate for direct controller.