[Rancher v2.7.0][Fleet v0.5.0] Redeploying Fleet-Controller Causes Local Cluster to Lose Annotations and Labels

atsai1220 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Restarting fleet-controller causes fleet-local/local cluster.fleet object to have its labels and annotations removed temporarily. This is problematic if you have gitrepo obejects that depend on labels such as provider.cattle.io: rke2 to install the appropriate bundles.

Besides removing the labels and annotations, it also sets a different objectset.io.cattle.io/hash. Are there multiple controllers touching this object?

The diff includes the following

original

annotations:
    objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4xTTY/jNgz9KwXPVppvOwZ6WMyhhwJ72O19wEi0o0YmDUn2bGaQ/15ITjJusTuYk2XxkXzke3qDjiIajAj1GyCzRIxWOKRfOf5DOgaKC29loTFGRwsrv1sDNTSOKCrthhDJQ/FLsLwwedWOZ6ih9zLaYIUttzPIuCp++8uy+ePpk9UYO4I
anGh0nwKHHjU9SE951wK0pzzt37ajELHroebBuQIcHsnlHWgng1GZuCEPNXx5HTw9pVso4D5+DR5Zn8gH8mMe4BZRxttxApxpne6Fg7jE5XZaGNHnC3qzwNfFaDFgXGjpoADi0XrhjjhCDSycaKSu9KOXMPhUxHIkz3kLTtrnIIPPg97oOGnDFGottyrKmRhqyN9FudRY7ogUHc1KbZfrpUKsNkpv9/t1U9G
+2jRQQIeMLSUWs9U+xrOhd3i5K4KvijDEIaj7VOou0odlPinoCcMpqbjGCg+HwwGrHa1WTVNt9rst7Vea9AbLarUtD82xKaFIjjODThqrEDEOIW8yKmdHmsJZ1/ce70oF0oO38aK0cKQfSQNDDQ4uQgFj+j57cfQ8vQaiPe2OpMxmvVSrUpdqXW5WarvTx+NmUzamqpLj/jfoB94MPen8Jlvi+HWGm6iqGVy
FS4iULHMejvQk3Nj2O2lP8d7q8U5T4dsebqXTwWGI3ylZY7L/nFZiKfyN0Fy+iqHM5IGbBQLUywL8T3F+DroWYChYTybn/mnjN+plSr/ZCeq36y3rP2FPk8GfZOCYR5hXypDOhmC5nc5ibGPJ5B+W+I4S35+QbxH/uB74zPLC+fyCNn7pezflp70NXYf+8tOu9xLX6/XfAAAA//+MPld4UgUAAA
 objectset.rio.cattle.io/id: fleet-cluster
labels:
  name: local
  objectset.rio.cattle.io/hash: f2a8a9999a85e11ff83654e61cec3a781479fbf7

What happens when fleet-controller starts. All other fields are set to null.

  annotations:
    objectset.rio.cattle.io/applied: H4sIAAAAAAAA/3RQzW7bMAx+F56dLMmSePFt6GG3Htq9ACPRiVaZFEQaRRD43QfLjecN3Y3U9yveoSNDj4bQ3AGZxdCCsI6rnH+RMyVb5yBrh2aR1kG+BA8NtJHIVmcRU8uYYKjAZSrin6EjNewSNNzHWEHEM8ViydgRNBDFYYTqvwlX1Cs0cKzbfX1oj7s
jHfb117retcfNtz3uTtttvcGTp8Npv/PbMfwf53HVhI7mqhMyVKCJXPnthdieF7wpf7Wgr/SmRh1U8Naf6Um4DZdXcpnsEbVysVejXIwNrdfZehwiqr0S8eMSy1pjS+EXQn97Fk+lycxbAArNpoL8KS8vSUMFnjRk8kX7I9gLJZnkPmiKeIPmPnyo/oIzqfTZ0ZP0bOULS6dC6YJq4Ms0iw9tIF8WFvvDkpy
uyB9Inp97fmN55zK/Y7DvKcVJP96t7zrMt09THxbDMPwOAAD//2h0km6sAgAA
    objectset.rio.cattle.io/id: fleet-bootstrap
  labels:
    name: local
    objectset.rio.cattle.io/hash: 67f475f626e5473772f6084a291170a9de5942d1

Expected Behavior

Labels and annotations should be not set to null when fleet-controller starts.

Steps To Reproduce

Create GitRepo object that selects the local cluster based on label

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: app
namespace: fleet-local
spec:
branch: master
clientSecretName: k8s-gitops
forceSyncGeneration: 4
insecureSkipTLSVerify: false
paths:
- /fleet/app
paused: false
pollingInterval: 1m
repo: git@git.viasat.com:ContainerServices/k8s-gitops.git
targets:
- clusterSelector:
  matchLabels:
    provider.cattle.io: rke2

Restart fleet-controller and observe the application to be temporarily uninstalled then installed again as labels are propagated back onto the cluster.fleet object.

Environment

- Architecture: amd
- Fleet Version: v0.5.0
- Cluster:
  - Provider: rke
  - Options: Rancher v2.7.0
  - Kubernetes Version: v1.21

Logs

time="2022-12-07T20:37:33Z" level=info msg="uninstall: Deleting viasat-gitm-logging"
Wed, Dec 7 2022 12:37:33 pm | time="2022-12-07T20:37:33Z" level=info msg="preparing upgrade for rancher-logging-crd"
Wed, Dec 7 2022 12:37:33 pm | time="2022-12-07T20:37:33Z" level=info msg="uninstall: Deleting rancher-logging"

Anything else?

Here is a video demonstrating the effect: https://user-images.githubusercontent.com/14171643/206297554-973d07bb-675c-4b28-918e-f9cb53ae2811.mov

atsai1220 commented 1 year ago

Seems to be duplicate of https://github.com/rancher/fleet/pull/1155

atsai1220 commented 1 year ago

Duplicate of https://github.com/rancher/fleet/issues/1079

kkaempf commented 1 year ago

closing as duplicate

manno commented 1 year ago

@atsai1220

Besides removing the labels and annotations, it also sets a different objectset.io.cattle.io/hash. Are there multiple controllers touching this object?

The hash is literally a hash over the annotations: https://github.com/rancher/wrangler/blob/master/pkg/apply/desiredset_apply.go#L223-L225

rancher / fleet