group-sync-operator 0.0.27 crashes since upgrading to Keycloak 23

[bo0ts]

group-sync-operator 0.0.27 crashes since upgrading to Keycloak 23:

1.701861847832853e+09 INFO controllers.GroupSync Beginning Sync {"groupsync": "group-sync-operator/keycloak-group-sync", "Provider": "keycloak"}
1.7018618479017475e+09 INFO syncer_keycloak Successfully Authenticated with Keycloak Provider
1.7018618479557958e+09 INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "groupsync", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "GroupSync", "groupSync": {"name":"keycloak-group-sync","namespace":"group-sync-operator"}, "namespace": "group-sync-operator", "name": "keycloak-group-sync", "reconcileID": "c5ba03ce-e19e-4422-bb24-3abb86ec4937"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x2de82a7]

goroutine 531 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x3592bc0, 0x67d3b10})
/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/panic.go:838 +0x207
github.com/redhat-cop/group-sync-operator/pkg/syncer.(*KeycloakSyncer).processGroupsAndMembers(0xc000276cc0, 0xc0005aaac0, 0x0, {0xc000994510, 0x3})
/home/runner/work/group-sync-operator/group-sync-operator/pkg/syncer/keycloak.go:286 +0x327
github.com/redhat-cop/group-sync-operator/pkg/syncer.(*KeycloakSyncer).Sync(0xc000276cc0)
/home/runner/work/group-sync-operator/group-sync-operator/pkg/syncer/keycloak.go:181 +0x1cc
github.com/redhat-cop/group-sync-operator/controllers.(*GroupSyncReconciler).Reconcile(0xc0002ae060, {0x47924a0, 0xc00051b200}, {{{0xc0002a0b40, 0x13}, {0xc0002a0b28, 0x13}}})
/home/runner/work/group-sync-operator/group-sync-operator/controllers/groupsync_controller.go:115 +0x89b
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x47923f8?, {0x47924a0?, 0xc00051b200?}, {{{0xc0002a0b40?, 0x40d41a0?}, {0xc0002a0b28?, 0x404bd4?}}})
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:121 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00031e460, {0x47923f8, 0xc0002bbf80}, {0x3927880?, 0xc000133940?})
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00031e460, {0x47923f8, 0xc0002bbf80})
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2

Here is the groupsync definition:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: GroupSync
metadata:
  name: keycloak-group-sync
spec:
  schedule: "0 * * * *"
  providers:
  - name: keycloak
    keycloak:
      loginRealm: master
      realm: ...
      caSecret:
        kind: Secret
        name: ...
        namespace: group-sync-operator
        key: ca.crt
      credentialsSecret:
        kind: Secret
        name: keycloak-group-sync
        namespace: group-sync-operator
      groups:
      - ...
      scope: sub
      url: https://.../

The error in the groupsync cr is:

could not get token: Post "https://keycloak.apps.central.okd.testfabrik.intern/realms/master/protocol/openid-connect/token": EOF```

[sabre1041]

@bo0ts The looks to be a communication issue from the operator and Keycloak and less of a compatibility issue

[bo0ts]

@sabre1041 I'm confused why you would think that. Everything was working fine until we updated the keycloak instance to version 23 and since then all our instances of group-sync-operator started crashing. All other Keycloak operations are also still working.

[sabre1041]

@bo0ts if you notice the error, it mentions EOF which typically refers to a connectivity issue to the endpoint

[tormig-softronic]

I have the same problem, Keycloak 23.0.1, group-sync-operator 0.0.27. I do not see any EOF in the log above or in our installation. The line: 1.7018618479017475e+09 INFO syncer_keycloak Successfully Authenticated with Keycloak Provider indicates that the connection seems to work. Does group-sync-operator expect something from Keycloak that now is missing?

[sabre1041]

@tormig-softronic @bo0ts The issue has been identified and will be submitting a patch to resolve shortly

[bo0ts]

@sabre1041 Could you be more open about what the issue was? I'm sure others can benefit from that knowledge as well.

[sabre1041]

@bo0ts the issue was a null check that needed to be added.

https://github.com/redhat-cop/group-sync-operator/pull/299

However, if you look at the linked issue within the PR regarding an upstream bug, it may also be a side effect of that as well. But, the issue you are facing should be resolved with this enhancement

[bo0ts]

@sabre1041 Thanks, could you also publish a new release with that fix at is a real blocker to Keycloak upgrades?

[bo0ts]

@sabre1041 Ping about the new release?

[sabre1041]

@bo0ts we are resolving an issue with our release pipeline. expect a release this week