search

splunk / qbec

configure kubernetes objects on multiple clusters using jsonnet
https://qbec.io
Apache License 2.0
172 stars 37 forks source link

Unstable qbec diff results (different versions for same Kind leads to object deletions) #312

Open evgkrsk opened 2 months ago

evgkrsk commented 2 months ago

We have some cluster-wide objects with type=kyverno.io/v1, Kind=ClusterPolicy (CRD-based) applied with qbec and observing such unpredictable "qbec diff" results:

good path:

> qbec diff test -c kyverno-policy --verbose 18 2>&1 |egrep -i 'ClusterPolicy|clusterpolicies|stats|deletions|same'
    kyverno.io/v1:ClusterPolicy                                            => kyverno.io/v1:ClusterPolicy (cluster scoped)
    kyverno.io/v2beta1:ClusterPolicy                                       => kyverno.io/v2beta1:ClusterPolicy (cluster scoped)
clusterpolicies pdbs (source kyverno-policy) unchanged
list objects: type=kyverno.io/v1, Kind=ClusterPolicy,namespace="" took 96ms
stats:
  same: 1

bad path:

> qbec diff test -c kyverno-policy --verbose 18 2>&1 |egrep -i 'ClusterPolicy|clusterpolicies|stats|deletions|same'
    kyverno.io/v1:ClusterPolicy                                            => kyverno.io/v1:ClusterPolicy (cluster scoped)
    kyverno.io/v2beta1:ClusterPolicy                                       => kyverno.io/v2beta1:ClusterPolicy (cluster scoped)
clusterpolicies pdbs (source kyverno-policy) unchanged
list objects: type=kyverno.io/v2beta1, Kind=ClusterPolicy,namespace="" took 94ms
--- live clusterpolicies pdbs (source: qbec annotation)
+++ config clusterpolicies pdbs
-kind: ClusterPolicy
stats:
  deletions:
  - clusterpolicies pdbs
  same: 1

i.e., most of the time qbec wrongly propose to drop already applied object, but sometimes compute right empty diff.

I can confirm that we render/apply only kyverno.io/v1 manifests.

qbec version                                                                                                                                                                                             4s
qbec version: 0.15.2
jsonnet version: v0.18.0
client-go version: kubernetes-1.23.1
go version: 1.17.7
commit: 9f26fb9d14300b3aefd87b89f8d346c3dce48092

(same behavior on 0.15.1) Kubernetes control-plane version: v1.27.14