`presetInvalidated: deleted` annotation on cluster set erroneously

[judge-red]

What happened?

Somehow, presets are considered deleted when they're not. It has affected several (but not all) clusters on our shared master/seed, but none on the standalone seed. Even when the same preset has been used.

on shared master/seed

$ k get cluster gnbqt67pbf -o yaml | yq .metadata.annotations
kubermatic.k8c.io/migrated-aws-node-termination-handler-addon: "yes"
kubermatic.k8c.io/openstack-floating-ip-pool-id: 3cc83f7d-9119-475b-ba17-f3510c7902e8
presetInvalidated: deleted
presetName: sandro.mathys-x9whtzzxl2

$ k get preset sandro.mathys-x9whtzzxl2
NAME                       AGE
sandro.mathys-x9whtzzxl2   432d

on standalone seed

$ k get cluster cq24lf9b6n -o yaml | yq .metadata.annotations
kubermatic.k8c.io/migrated-aws-node-termination-handler-addon: "yes"
kubermatic.k8c.io/openstack-floating-ip-pool-id: 71ad36a2-d833-4e00-8e16-ec5d967762e3
presetName: sandro.mathys-x9whtzzxl2

$ k get preset sandro.mathys-x9whtzzxl2
NAME                       AGE
sandro.mathys-x9whtzzxl2   432d

No idea, also no idea when it happened. But it became very apparent with the upgrade to 2.26.0 as the preset "status" is shown on the dashboard now. I suspect the issue itself is older (and maybe fixed but the annotations stuck?)

Expected behavior

Available presets should not be marked deleted in clusters.

How to reproduce the issue?

Unfortunately, no idea.

How is your environment configured?

• KKP version: 2.26.0
• Shared or separate master/seed clusters?: shared

Provide your KKP manifest here (if applicable)

N/A

What cloud provider are you running on?

OpenStack

What operating system are you running in your user cluster?

Ubuntu 22.04

Additional information

[xrstf]

Just to be sure, the sandro.mathys-x9whtzzxl2 Preset is currently not being deleted, right? It's not just kept alive because of some finalizer?

[judge-red]

on shared master:

$ k get preset sandro.mathys-x9whtzzxl2 -o yaml
apiVersion: kubermatic.k8c.io/v1
kind: Preset
metadata:
  creationTimestamp: "2023-08-25T20:51:41Z"
  finalizers:
  - kubermatic.k8c.io/cleanup-seed-preset
  generation: 1
  name: sandro.mathys-x9whtzzxl2
  resourceVersion: "637885817"
  uid: d3d40ccb-155a-4294-aa20-c6fd25a5c835
spec:
  openstack:
    [REDACTED]
  projects:
  - x9whtzzxl2

no deletionTimestamp means it's not being deleted, right?

[xrstf]

Sure, I'm just asking because https://github.com/kubermatic/kubermatic/blob/main/pkg/controller/seed-controller-manager/preset-controller/controller.go#L106 is the only place where I can see us adding the annotation.

[judge-red]

Right.

I just wanted to test something in this regard and I got a bit surprised. I created a new preset but it wasn't created in our standalone-seed cluster, just the shared master/seed cluster. Nevertheless, I was able to create a UC in a DC on the standalone-seed. Are Presets now only created in the master? If so, when this change happened, is it possible all existing presets were "moved" from seeds to master and that for a shared-master/seed this briefly marked the seed as deleted (moving it away from the seed) and then undeleted (moving it to master)? Can't find any changelog on presets not being mirrored to standalone-seeds anymore, though.

[judge-red]

Also, wondering whether the invalidated marker should be removed again if a preset with the same name was created again. The system will treat a re-created preset just like it was never deleted afaict as all references are by name.

[judge-red]

I just wanted to test something in this regard and I got a bit surprised. I created a new preset but it wasn't created in our standalone-seed cluster, just the shared master/seed cluster. Nevertheless, I was able to create a UC in a DC on the standalone-seed. Are Presets now only created in the master?

Maybe this is a separate bug. After deleting the preset, the dashboard shows an error that the preset can't be found but the invalidated marker hasn't been applied. I think the rollout might not have happened as it should due to an unknown issue.

I created the same preset again and this time it was mirrored immediately. Not sure why that didn't happen last time. Apparently there was a major delay last time as I've now seen that the master-ctrl-mgr logged a problem with applying the finalizer because the preset was deleted already - but I only deleted it several minutes (or more) after creating it, evident by the fact that I created a UC through the dashboard. Well, I guess that's a separate issue.

So ignore this theory, which sounded hard to believe anyway:

If so, when this change happened, is it possible all existing presets were "moved" from seeds to master and that for a shared-master/seed this briefly marked the seed as deleted (moving it away from the seed) and then undeleted (moving it to master)? Can't find any changelog on presets not being mirrored to standalone-seeds anymore, though.