external-name annotation on client lost

[TomBillietKlarrio]

Hi,

We have quite some openid clients we create trough crossplane. However, from time to time, they seem to loose the external-name annotation in the metadata section, causing crossplane to give errors on those objects as it will try to create new ones in keycloak and can't do that. We're unsure why this happens, we seems to trigger it sometimes when we restart some k8s nodes during a software release, but haven't been able to pinpoint the exact root. I've extracted some logs that are related to a client at the moment it got corrupted. It looks like it looses the object from the cache, and tries to rebuild it? But then why would it remove the external-name annotation? Any help is appreciated.

2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Async create ended. {"trackerUID": "6be476ee-b482-413b-80e4-920959348371", "resourceName": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "error": "async create failed: failed to create the resource: [{0 error sending POST request to /auth/admin/realms/poc-dsh/clients: 409 Conflict. Response body: {\"errorMessage\":\"Client XXXXX already exists\"}  []}]", "tfID": ""}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Calling the inner handler for Update event. {"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 1}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Successfully requested creation of external resource    {"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=client", "request": {"name":"XXXXX"}, "uid": "6be476ee-b482-413b-80e4-920959348371", "version": "561154598", "external-name": "", "external-name": ""}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Creating the external resource  {"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Async create starting...    {"trackerUID": "6be476ee-b482-413b-80e4-920959348371", "resourceName": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "tfID": ""}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Reconciling {"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=clientdefaultscopes", "request": {"name":"XXXXX"}}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Diff detected   {"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"access_token_lifespan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"access_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"CONFIDENTIAL\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"admin_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"backchannel_logout_session_required\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"base_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_authenticator_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"client-secret\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"XXXXX\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_offline_session_idle_timeout\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_offline_session_max_lifespan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_secret\":*terraform.ResourceAttrDiff{Old:\"\", New:\"b613f5bd-8066-4695-b6f2-483ad720df59\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:true, Type:0x0}, \"client_session_idle_timeout\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"client_session_max_lifespan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"consent_required\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"consent_screen_text\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"description\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"direct_access_grants_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"display_on_consent_screen\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"exclude_session_state_from_auth_response\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"frontchannel_logout_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"full_scope_allowed\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"implicit_flow_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"import\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"oauth2_device_authorization_grant_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"realm_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"poc-dsh\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"resource_server_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"root_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"service_account_user_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"service_accounts_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"standard_flow_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"use_refresh_tokens\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"use_refresh_tokens_client_credentials\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"valid_post_logout_redirect_uris.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"valid_redirect_uris.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"web_origins.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Observing the external resource {"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:09Z    DEBUG   provider-keycloak   Instance state not found in cache, reconstructing...    {"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:08Z    DEBUG   provider-keycloak   Calling the inner handler for Update event. {"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 0}
2024-10-02T12:54:08Z    DEBUG   provider-keycloak   Connecting to the service provider  {"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}
2024-10-02T12:54:08Z    DEBUG   provider-keycloak   Reconciling {"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=client", "request": {"name":"XXXXX"}}
2024-10-02T12:54:08Z    DEBUG   provider-keycloak   Calling the inner handler for Update event. {"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 0}
2024-10-02T12:52:06Z    DEBUG   provider-keycloak   Calling the inner handler for Create event. {"gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client", "name": "XXXXX", "queueLength": 5}
2024-10-02T12:43:07Z    DEBUG   provider-keycloak   External resource is up to date {"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=clientdefaultscopes", "request": {"name":"XXXXX"}, "uid": "a4727f32-e53c-4a2f-8aee-329349046778", "version": "561044047", "external-name": "poc-dsh/41653235-ca40-4412-b629-61d1d0d0ef2a", "requeue-after": "2024-10-02T12:53:07Z"}
2024-10-02T12:43:07Z    DEBUG   provider-keycloak   Reconciling {"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=clientdefaultscopes", "request": {"name":"XXXXX"}}
2024-10-02T12:43:04Z    DEBUG   provider-keycloak   External resource is up to date {"controller": "managed/openidclient.keycloak.crossplane.io/v1alpha1, kind=client", "request": {"name":"XXXXX"}, "uid": "6be476ee-b482-413b-80e4-920959348371", "version": "561043923", "external-name": "41653235-ca40-4412-b629-61d1d0d0ef2a", "requeue-after": "2024-10-02T12:53:04Z"}
2024-10-02T12:43:04Z    DEBUG   provider-keycloak   Observing the external resource {"uid": "6be476ee-b482-413b-80e4-920959348371", "name": "XXXXX", "gvk": "openidclient.keycloak.crossplane.io/v1alpha1, Kind=Client"}

[Breee]

Hm, i need to investigate on that in a test cluster.
If restarting nodes triggers this, then i should be able to reproduce it.
But we should also be able to reproduce it then by killing the provider pod right? (You can also start multiple replicas for HA, maybe that changes it) - However, that should not matter because the real state should be stored in ETCD and not in memory.

[TomBillietKlarrio]

I did try quite a lot of different things to reproduce it, but have not been able to find a root cause what exactly triggers it unfortunately. Just killing the keycloak-crossplane (or crossplane) pod does not trigger it. We're not indeed trying to run 2 instances for the keycloak-crossplane provider to see if that helps