search

open-policy-agent / gatekeeper

🐊 Gatekeeper - Policy Controller for Kubernetes
https://open-policy-agent.github.io/gatekeeper/
Apache License 2.0
3.7k stars 759 forks source link

Audit takes an unreasonable amount of time. Limited to 1 CPU core #2502

Closed bhattchaitanya closed 1 year ago

bhattchaitanya commented 1 year ago

What steps did you take and what happened: Audit manager locks onto 1 CPU core and slows down the audit. It does not spin up multiple goroutines to make use of all the CPU cores. GOMAXPROCS and --max-serving-threads had no effect.

What did you expect to happen: The audit controller should spin up multiple goroutines in the audit cycle based on constraint kinds or other factors to reduce the audit time. In large clusters, audits take hours to complete which is a serious limitation.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

maxsmythe commented 1 year ago

Are you able to determine whether this is caused by requests to the API server getting throttled?

Unfortunately this would be hard to tease out, since there are no logs that signify scraping the API server has finished, but maybe the K8s client logs throttling?

@ritazh should we add a "resource listing complete" log line?

bhattchaitanya commented 1 year ago

I can confirm that there are NO no instances of client-side throttling or API server-side throttling.

maxsmythe commented 1 year ago

How were you able to determine this?

maxsmythe commented 1 year ago

Another way to experiment with whether this is due to single-threaded Rego execution vs. something about the execution process would be to dump the contents of the cluster to disk and run gator test against it.

Another thought, what are your constraint templates like? Do they use external data? Any Rego calls to http.Send()?

bhattchaitanya commented 1 year ago

When there is throttling on OPA GK, we typically see error message like Waited for 1.047497363s due to client-side throttling, not priority and fairness. We occasionally see it, but haven't seen it lately.

bhattchaitanya commented 1 year ago

thanks for the speedy response. @maxsmythe how to dump the contents of the cluster to disk? Also, can you confirm that audit-controller is single-threaded by design?

maxsmythe commented 1 year ago

how to dump the contents of the cluster to disk?

Not sure if there is a faster way than looping over kubectl get -oyaml for all resource/version combos and piping the results to files (using kubectl api-versions to loop over known resources)

audit-controller is single-threaded by design?

Audit controller is single-threaded, but it doesn't necessarily have to be.

Mostly I want to be sure we're addressing the correct problem and am trying to figure out what we can learn without needing to wait for code that gives better profiling data. Generally, pure Rego shouldn't take ~O(hours) to execute even over large datasets unless there are a lot of referential constraints that scale poorly with the size of the data footprint.

It could also be network latency/throttling, which tends to be more likely for severe slowdowns.

It could also be blocking calls (such as I/O requests), in which case some workqueueing could help.

There are a number of possibilities for the root cause, each with different solutions.

maxsmythe commented 1 year ago

If you are willing to post the logs from the audit pod, that might be interesting to look at as well.

bhattchaitanya commented 1 year ago

Log dump 

{"level":"info","ts":1672966185.8108387,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1672966185.828801,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"mutations.gatekeeper.sh/v1beta1, Kind=AssignMetadata","expectationCount":0}
{"level":"info","ts":1672966185.8305917,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"mutations.gatekeeper.sh/v1beta1, Kind=ModifySet","expectationCount":0}
{"level":"info","ts":1672966185.83201,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"mutations.gatekeeper.sh/v1beta1, Kind=Assign","expectationCount":0}
{"level":"info","ts":1672966185.8389318,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"templates.gatekeeper.sh/v1beta1, Kind=ConstraintTemplate","expectationCount":8}
{"level":"info","ts":1672966185.8494802,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sBlockingPDB","expectationCount":1}
{"level":"info","ts":1672966185.8496025,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=k8sDeprecatedAPI","expectationCount":0}
{"level":"info","ts":1672966185.850645,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sPodCrashLoopBackOff","expectationCount":1}
{"level":"info","ts":1672966185.8527455,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sMisconfiguredPDB","expectationCount":1}
{"level":"info","ts":1672966185.8616266,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sRequiredResources","expectationCount":1}
{"level":"info","ts":1672966185.863489,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sMisconfiguredHPA","expectationCount":1}
{"level":"info","ts":1672966185.865183,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sDisallowedLivenessProbe","expectationCount":1}
{"level":"info","ts":1672966185.8694093,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sRequiredObject","expectationCount":2}
{"level":"info","ts":1672966185.882589,"logger":"setup","msg":"setting up controllers"}
{"level":"info","ts":1672966185.8829014,"logger":"controller-runtime.manager.controller.config-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8829217,"logger":"controller-runtime.manager.controller.config-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966185.8829556,"logger":"controller-runtime.manager.controller.config-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8831317,"logger":"setup","msg":"setting up audit"}
{"level":"info","ts":1672966185.8831599,"logger":"controller-runtime.manager.controller.sync-controller","msg":"Starting EventSource","source":"channel source: 0xc0001009b0"}
{"level":"info","ts":1672966185.8850079,"logger":"controller-runtime.manager.controller.sync-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8850226,"logger":"controller-runtime.manager.controller.sync-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966185.8832982,"logger":"controller-runtime.manager.controller.constraint-template-status-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8850412,"logger":"controller-runtime.manager.controller.constraint-template-status-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8850534,"logger":"controller-runtime.manager.controller.constraint-template-status-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.883704,"logger":"controller-runtime.manager.controller.modifyset-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8850884,"logger":"controller-runtime.manager.controller.modifyset-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.885096,"logger":"controller-runtime.manager.controller.modifyset-controller","msg":"Starting EventSource","source":"channel source: 0xc000101220"}
{"level":"info","ts":1672966185.8851209,"logger":"controller-runtime.manager.controller.modifyset-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8834121,"logger":"controller-runtime.manager.controller.assign-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8851342,"logger":"controller-runtime.manager.controller.assign-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8851604,"logger":"controller-runtime.manager.controller.assign-controller","msg":"Starting EventSource","source":"channel source: 0xc000101220"}
{"level":"info","ts":1672966185.8851788,"logger":"controller-runtime.manager.controller.assign-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.88348,"logger":"controller-runtime.manager.controller.constrainttemplate-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8851898,"logger":"controller-runtime.manager.controller.constrainttemplate-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.885199,"logger":"controller-runtime.manager.controller.constrainttemplate-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8852053,"logger":"controller-runtime.manager.controller.constrainttemplate-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8835435,"logger":"controller-runtime.manager.controller.constraint-controller","msg":"Starting EventSource","source":"channel source: 0xc000100cd0"}
{"level":"info","ts":1672966185.8854446,"logger":"controller-runtime.manager.controller.constraint-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.885457,"logger":"controller-runtime.manager.controller.constraint-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8836205,"logger":"controller-runtime.manager.controller.constraint-status-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8857493,"logger":"controller-runtime.manager.controller.constraint-status-controller","msg":"Starting EventSource","source":"channel source: 0xc000100f50"}
{"level":"info","ts":1672966185.885775,"logger":"controller-runtime.manager.controller.constraint-status-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8837795,"logger":"controller-runtime.manager.controller.assignmetadata-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8859394,"logger":"controller-runtime.manager.controller.assignmetadata-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8859491,"logger":"controller-runtime.manager.controller.assignmetadata-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8838832,"logger":"controller-runtime.manager.controller.mutator-status-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.88624,"logger":"controller-runtime.manager.controller.mutator-status-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.8862498,"logger":"controller-runtime.manager.controller.mutator-status-controller","msg":"Starting EventSource","source":"kind source: /, Kind="}
{"level":"info","ts":1672966185.886262,"logger":"controller-runtime.manager.controller.mutator-status-controller","msg":"Starting Controller"}
{"level":"info","ts":1672966185.8904161,"logger":"setup","msg":"setting up upgrade"}
{"level":"info","ts":1672966185.8905756,"logger":"setup","msg":"setting up metrics"}
{"level":"info","ts":1672966185.8907573,"logger":"metrics","msg":"Starting metrics runner"}
{"level":"info","ts":1672966185.8909314,"logger":"metrics","msg":"metrics","backend":"prometheus"}
{"level":"info","ts":1672966185.8912196,"logger":"metrics","msg":"Starting server for OpenCensus Prometheus exporter"}
{"level":"info","ts":1672966185.8915641,"logger":"controller","msg":"Starting Audit Manager","process":"audit"}
{"level":"info","ts":1672966185.8917038,"logger":"controller","msg":"Starting Upgrade Manager","metaKind":"upgrade"}
{"level":"info","ts":1672966185.9851623,"logger":"controller","msg":"disabling readiness stats","kind":"Config"}
{"level":"info","ts":1672966186.9184391,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"policy/v1, Kind=PodDisruptionBudget","expectationCount":572}
{"level":"info","ts":1672966187.339345,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"autoscaling/v1, Kind=HorizontalPodAutoscaler","expectationCount":607}
{"level":"info","ts":1672966187.6111987,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"/v1, Kind=Namespace","expectationCount":706}
{"level":"info","ts":1672966187.8728373,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sRequiredResources","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966187.9302034,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":1}
{"level":"info","ts":1672966187.9302592,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sRequiredResources"}
{"level":"info","ts":1672966187.9302714,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sRequiredObject","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.0180855,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":2}
{"level":"info","ts":1672966188.0181177,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sRequiredObject"}
{"level":"info","ts":1672966188.0181272,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sDisallowedLivenessProbe","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.0394852,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":1}
{"level":"info","ts":1672966188.039745,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sDisallowedLivenessProbe"}
{"level":"info","ts":1672966188.0398452,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"k8sDeprecatedAPI","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.0714695,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":0}
{"level":"info","ts":1672966188.0714965,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sMisconfiguredPDB","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.0913813,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":1}
{"level":"info","ts":1672966188.0914063,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sMisconfiguredPDB"}
{"level":"info","ts":1672966188.0914147,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sPodCrashLoopBackOff","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.1428885,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":1}
{"level":"info","ts":1672966188.142919,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sPodCrashLoopBackOff"}
{"level":"info","ts":1672966188.1429288,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sMisconfiguredHPA","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.189277,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":1}
{"level":"info","ts":1672966188.1893082,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sMisconfiguredHPA"}
{"level":"info","ts":1672966188.1893172,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"K8sBlockingPDB","group":"constraints.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966188.208177,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":1}
{"level":"info","ts":1672966188.2086508,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"constraints.gatekeeper.sh","version":"v1alpha1","kind":"K8sBlockingPDB"}
{"level":"info","ts":1672966189.012785,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"apps/v1, Kind=Deployment","expectationCount":548}
{"level":"info","ts":1672966189.2063541,"logger":"controller-runtime.manager.controller.constraint-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966190.1412814,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"ConstraintTemplate","group":"templates.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966190.1775677,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":8}
{"level":"info","ts":1672966190.177782,"logger":"controller","msg":"starting update resources loop","metaKind":"upgrade","group":"templates.gatekeeper.sh","version":"v1alpha1","kind":"ConstraintTemplate"}
{"level":"info","ts":1672966192.5584567,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"Assign","group":"mutations.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966192.5785918,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":0}
{"level":"info","ts":1672966192.5786195,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"AssignMetadata","group":"mutations.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966192.6153836,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":0}
{"level":"info","ts":1672966192.6154156,"logger":"controller","msg":"resource","metaKind":"upgrade","kind":"ModifySet","group":"mutations.gatekeeper.sh","version":"v1alpha1"}
{"level":"info","ts":1672966192.623093,"logger":"controller","msg":"resource count","metaKind":"upgrade","count":0}
{"level":"info","ts":1672966192.6252406,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":1672966195.9147346,"logger":"controller-runtime.manager.controller.constraint-template-status-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966195.9267142,"logger":"controller-runtime.manager.controller.mutator-status-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966195.9267762,"logger":"controller-runtime.manager.controller.modifyset-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966195.9268563,"logger":"controller-runtime.manager.controller.constraint-status-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966195.9271274,"logger":"controller-runtime.manager.controller.assignmetadata-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966195.9275217,"logger":"controller-runtime.manager.controller.constrainttemplate-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966195.9282017,"logger":"controller-runtime.manager.controller.assign-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1672966196.0210526,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8spodcrashloopbackoff"}}
{"level":"info","ts":1672966196.022302,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.0271802,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966196.0317702,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdeprecatedapi"}}
{"level":"info","ts":1672966196.049403,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredhpa"}}
{"level":"info","ts":1672966196.0615234,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredobject"}}
{"level":"info","ts":1672966196.0734873,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966196.0897784,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdisallowedlivenessprobe"}}
{"level":"info","ts":1672966196.0764391,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh","name":"k8sblockingpdb"}
{"level":"info","ts":1672966196.1065087,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.107695,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sBlockingPDB"}
{"level":"info","ts":1672966196.1079776,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredpdb"}}
{"level":"info","ts":1672966196.1101828,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966196.1161711,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sblockingpdb"}
{"level":"info","ts":1672966196.1213639,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"blocking-pdb"}
{"level":"info","ts":1672966196.1213832,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sBlockingPDB","constraint_name":"blocking-pdb","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.1220536,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sblockingpdb"}}
{"level":"info","ts":1672966196.12589,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.1337497,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966196.137961,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdeprecatedapi"}}
{"level":"info","ts":1672966196.148298,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sblockingpdb"}}
{"level":"info","ts":1672966196.1508193,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966196.1607554,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-pdb"}}
{"level":"info","ts":1672966196.1843235,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh","name":"k8sdeprecatedapi"}
{"level":"info","ts":1672966196.188421,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.1915545,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966196.192457,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=k8sDeprecatedAPI"}
{"level":"info","ts":1672966196.2026694,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sdeprecatedapi"}
{"level":"info","ts":1672966196.2031689,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdeprecatedapi"}}
{"level":"info","ts":1672966196.2091193,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966196.2162724,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdisallowedlivenessprobe"}}
{"level":"info","ts":1672966196.2172892,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.229802,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdeprecatedapi"}}
{"level":"info","ts":1672966196.2697935,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh","name":"k8sdisallowedlivenessprobe"}
{"level":"info","ts":1672966196.2714813,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.2734718,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966196.2769034,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966196.2770607,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sDisallowedLivenessProbe"}
{"level":"info","ts":1672966196.2888544,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sdisallowedlivenessprobe"}
{"level":"info","ts":1672966196.2891083,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"must-not-have-liveness-probe-setup"}
{"level":"info","ts":1672966196.289357,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sDisallowedLivenessProbe","constraint_name":"must-not-have-liveness-probe-setup","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.292528,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdisallowedlivenessprobe"}}
{"level":"info","ts":1672966196.2959278,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966196.3034525,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966196.3047974,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.3100142,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredhpa"}}
{"level":"info","ts":1672966196.319658,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966196.3214614,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdisallowedlivenessprobe"}}
{"level":"info","ts":1672966196.350251,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh","name":"k8smisconfiguredhpa"}
{"level":"info","ts":1672966196.3516798,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.3527315,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sMisconfiguredHPA"}
{"level":"info","ts":1672966196.3597162,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966196.3614411,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredhpa"}}
{"level":"info","ts":1672966196.3637404,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8smisconfiguredhpa"}
{"level":"info","ts":1672966196.3724594,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"misconfigured-hpa"}
{"level":"info","ts":1672966196.3725233,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sMisconfiguredHPA","constraint_name":"misconfigured-hpa","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.3773472,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredpdb"}}
{"level":"info","ts":1672966196.3791506,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.3856366,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966196.393965,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredhpa"}}
{"level":"info","ts":1672966196.40883,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966196.4589045,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966196.449136,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh","name":"k8smisconfiguredpdb"}
{"level":"info","ts":1672966196.460678,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.4610476,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sMisconfiguredPDB"}
{"level":"info","ts":1672966196.4718308,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8smisconfiguredpdb"}
{"level":"info","ts":1672966196.4725475,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredpdb"}}
{"level":"info","ts":1672966196.4742823,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966196.4830978,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.4917119,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8spodcrashloopbackoff"}}
{"level":"info","ts":1672966196.4921243,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"misconfigured-pdb"}
{"level":"info","ts":1672966196.492541,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sMisconfiguredPDB","constraint_name":"misconfigured-pdb","constraint_action":"deny","constraint_status":"enforced"}
{"level":"info","ts":1672966196.500415,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredpdb"}}
{"level":"info","ts":1672966196.5016313,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966196.5242112,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966196.5666509,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh","name":"k8spodcrashloopbackoff"}
{"level":"info","ts":1672966196.5674593,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.5682743,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sPodCrashLoopBackOff"}
{"level":"info","ts":1672966196.5777278,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8spodcrashloopbackoff"}}
{"level":"info","ts":1672966196.5778074,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966196.578166,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8spodcrashloopbackoff"}
{"level":"info","ts":1672966196.5867975,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.593626,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredobject"}}
{"level":"info","ts":1672966196.6043553,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8spodcrashloopbackoff"}}
{"level":"info","ts":1672966196.6384165,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"container-crashloopbackoff"}
{"level":"info","ts":1672966196.6385946,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sPodCrashLoopBackOff","constraint_name":"container-crashloopbackoff","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.63911,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966196.6459587,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966196.6491618,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966196.6369004,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh","name":"k8srequiredobject"}
{"level":"info","ts":1672966196.6536968,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.6539156,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sRequiredObject"}
{"level":"info","ts":1672966196.6543922,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"/v1, Kind=Pod","expectationCount":2601}
{"level":"info","ts":1672966196.6544387,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966196.6544168,"logger":"readiness-tracker","msg":"ExpectationsDone","gvk":"config.gatekeeper.sh/v1alpha1, Kind=Config","expectationCount":5}
{"level":"info","ts":1672966196.6640055,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8srequiredobject"}
{"level":"info","ts":1672966196.664382,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredobject"}}
{"level":"info","ts":1672966196.6658046,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966196.6663547,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-pdb"}}
{"level":"info","ts":1672966196.6792471,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.6804347,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966196.6809292,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966196.6845434,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"required-pdb"}
{"level":"info","ts":1672966196.6845663,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sRequiredObject","constraint_name":"required-pdb","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.6904151,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966196.7015212,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966196.7018428,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredobject"}}
{"level":"info","ts":1672966196.7106178,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-pdb"}}
{"level":"info","ts":1672966196.7196996,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966196.7466528,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh","name":"k8srequiredresources"}
{"level":"info","ts":1672966196.748053,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.7483134,"logger":"watch-manager","msg":"replaying events","registrar":"constrainttemplate-controller-status","gvk":"constraints.gatekeeper.sh/v1beta1, Kind=K8sRequiredResources"}
{"level":"info","ts":1672966196.7492998,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"required-hpa"}
{"level":"info","ts":1672966196.749327,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sRequiredObject","constraint_name":"required-hpa","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.757188,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966196.7581818,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966196.7588475,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8srequiredresources"}
{"level":"info","ts":1672966196.7607129,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.7617974,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh","name":"k8sblockingpdb"}
{"level":"info","ts":1672966196.7625165,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.762863,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966196.7696395,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-pdb"}}
{"level":"info","ts":1672966196.7801704,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sblockingpdb"}
{"level":"info","ts":1672966196.7812288,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966196.7819598,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.7831972,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh","name":"k8sdeprecatedapi"}
{"level":"info","ts":1672966196.783816,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966196.7842557,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.7934084,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966196.7938194,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966196.798685,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sdeprecatedapi"}
{"level":"info","ts":1672966196.7994132,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"container-must-have-limits-and-requests"}
{"level":"info","ts":1672966196.7994354,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sRequiredResources","constraint_name":"container-must-have-limits-and-requests","constraint_action":"warn","constraint_status":"enforced"}
{"level":"info","ts":1672966196.8007677,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8021605,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh","name":"k8sdisallowedlivenessprobe"}
{"level":"info","ts":1672966196.8031013,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8072731,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966196.8103647,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966196.810679,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sdisallowedlivenessprobe"}
{"level":"info","ts":1672966196.8128104,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8142538,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh","name":"k8smisconfiguredhpa"}
{"level":"info","ts":1672966196.8153164,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8199897,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966196.8274157,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966196.8279657,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8smisconfiguredhpa"}
{"level":"info","ts":1672966196.8312342,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.832603,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh","name":"k8smisconfiguredpdb"}
{"level":"info","ts":1672966196.8337595,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8485675,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8smisconfiguredpdb"}
{"level":"info","ts":1672966196.851078,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8528566,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh","name":"k8spodcrashloopbackoff"}
{"level":"info","ts":1672966196.85417,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8622496,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8spodcrashloopbackoff"}
{"level":"info","ts":1672966196.8645298,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8663137,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh","name":"k8srequiredobject"}
{"level":"info","ts":1672966196.8675542,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8810523,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8srequiredobject"}
{"level":"info","ts":1672966196.8849936,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8874793,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh","name":"k8srequiredresources"}
{"level":"info","ts":1672966196.8887963,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966196.8974977,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8srequiredresources"}
{"level":"info","ts":1672966204.20349,"logger":"readiness-tracker","msg":"readiness satisfied, no further collection"}
{"level":"info","ts":1672966205.5264502,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-pdb"}}
{"level":"info","ts":1672966205.5326476,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdisallowedlivenessprobe"}}
{"level":"info","ts":1672966205.5427234,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-pdb"}}
{"level":"info","ts":1672966205.543028,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966205.551177,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.551901,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8spodcrashloopbackoff"}}
{"level":"info","ts":1672966205.5538101,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sBlockingPDB","name":"blocking-pdb"}}
{"level":"info","ts":1672966205.5549304,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh","name":"k8sdisallowedlivenessprobe"}
{"level":"info","ts":1672966205.5559537,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966205.5562494,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdisallowedlivenessprobe","crdName":"k8sdisallowedlivenessprobe.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.569289,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdeprecatedapi"}}
{"level":"info","ts":1672966205.5721772,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredHPA","name":"misconfigured-hpa"}}
{"level":"info","ts":1672966205.572193,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sdisallowedlivenessprobe"}
{"level":"info","ts":1672966205.575534,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.5763915,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966205.5797713,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh","name":"k8spodcrashloopbackoff"}
{"level":"info","ts":1672966205.581147,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8spodcrashloopbackoff","crdName":"k8spodcrashloopbackoff.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.58548,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sblockingpdb"}}
{"level":"info","ts":1672966205.5892375,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8spodcrashloopbackoff"}
{"level":"info","ts":1672966205.5911033,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.592925,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh","name":"k8sdeprecatedapi"}
{"level":"info","ts":1672966205.5937963,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sdeprecatedapi","crdName":"k8sdeprecatedapi.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.5964084,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966205.5985525,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sPodCrashLoopBackOff","name":"container-crashloopbackoff"}}
{"level":"info","ts":1672966205.60591,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredpdb"}}
{"level":"info","ts":1672966205.6111994,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966205.6126206,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sdeprecatedapi"}
{"level":"info","ts":1672966205.6158662,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.6135406,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredResources","name":"container-must-have-limits-and-requests"}}
{"level":"info","ts":1672966205.6203694,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966205.6251516,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966205.6283765,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sRequiredObject","name":"required-hpa"}}
{"level":"info","ts":1672966205.6300817,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh","name":"k8sblockingpdb"}
{"level":"info","ts":1672966205.6313157,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sblockingpdb","crdName":"k8sblockingpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.6370637,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredhpa"}}
{"level":"info","ts":1672966205.6410108,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sMisconfiguredPDB","name":"misconfigured-pdb"}}
{"level":"info","ts":1672966205.641959,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sblockingpdb"}
{"level":"info","ts":1672966205.647312,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.65428,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh","name":"k8smisconfiguredpdb"}
{"level":"info","ts":1672966205.6555035,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredpdb","crdName":"k8smisconfiguredpdb.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.651211,"logger":"controller","msg":"handling constraint status update","process":"constraint_status_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966205.6517773,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredobject"}}
{"level":"info","ts":1672966205.6688755,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8smisconfiguredpdb"}
{"level":"info","ts":1672966205.6696033,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdisallowedlivenessprobe"}}
{"level":"info","ts":1672966205.6737437,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.6756992,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh","name":"k8srequiredresources"}
{"level":"info","ts":1672966205.6770031,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sDisallowedLivenessProbe","name":"must-not-have-liveness-probe-setup"}}
{"level":"info","ts":1672966205.6796198,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredresources","crdName":"k8srequiredresources.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.680775,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8spodcrashloopbackoff"}}
{"level":"info","ts":1672966205.6872118,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8srequiredresources"}
{"level":"info","ts":1672966205.689341,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.690627,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh","name":"k8smisconfiguredhpa"}
{"level":"info","ts":1672966205.6935296,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sdeprecatedapi"}}
{"level":"info","ts":1672966205.694725,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8smisconfiguredhpa","crdName":"k8smisconfiguredhpa.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.7023435,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8sblockingpdb"}}
{"level":"info","ts":1672966205.7104998,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8smisconfiguredhpa"}
{"level":"info","ts":1672966205.712222,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.7134092,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh","name":"k8srequiredobject"}
{"level":"info","ts":1672966205.7141945,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8srequiredobject","crdName":"k8srequiredobject.constraints.gatekeeper.sh"}
{"level":"info","ts":1672966205.7251785,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredpdb"}}
{"level":"info","ts":1672966205.726849,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8srequiredobject"}
{"level":"info","ts":1672966205.7398882,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredresources"}}
{"level":"info","ts":1672966205.7549374,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8smisconfiguredhpa"}}
{"level":"info","ts":1672966205.767902,"logger":"controller","msg":"handling constraint template status update","process":"constraint_template_status_controller","instance":{"apiVersion":"templates.gatekeeper.sh/v1beta1","kind":"ConstraintTemplate","name":"k8srequiredobject"}}
{"level":"info","ts":1672969785.892401,"logger":"controller","msg":"auditing constraints and violations","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"audit_started"}
{"level":"info","ts":1672969786.94428,"msg":"Waited for 1.042806341s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/networking.istio.io/v1alpha3?timeout=32s\n"}
{"level":"info","ts":1672969787.776588,"logger":"controller","msg":"Auditing via discovery client","process":"audit","audit_id":"2023-01-06T01:49:45Z"}
{"level":"info","ts":1672969859.1230094,"logger":"KubeAPIWarningLogger","msg":"v1 ComponentStatus is deprecated in v1.19+"}
{"level":"info","ts":1672970000.8920937,"logger":"controller","msg":"service-asset-alias: Intuit.billingcomm.billing.ebpicatlpriasync, service-asset-id: x, aws-account-no: x, region: us-west-2, reason: Namespace: <billingcomm-billing-ebpicatlpriasync-usw2-qal>, HorizontalPodAutoscaler <ebpi-catl-pri-async-rollout-hpa> has minReplicas equals to maxReplicas","process":"audit","audit_id":"2023-01-06T01:49:45Z","details":{},"event_type":"violation_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sMisconfiguredHPA","constraint_name":"misconfigured-hpa","constraint_namespace":"","constraint_action":"warn","resource_group":"autoscaling","resource_api_version":"v1","resource_kind":"HorizontalPodAutoscaler","resource_namespace":"billingcomm-billing-ebpicatlpriasync-usw2-qal","resource_name":"ebpi-catl-pri-async-rollout-hpa"}
...
{"level":"info","ts":1672974654.998608,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sRequiredResources","constraint_name":"container-must-have-limits-and-requests","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"1018"}
{"level":"info","ts":1672974654.9991357,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sRequiredObject","constraint_name":"required-pdb","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"53"}
{"level":"info","ts":1672974654.999154,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sDisallowedLivenessProbe","constraint_name":"must-not-have-liveness-probe-setup","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"81"}
{"level":"info","ts":1672974654.9991663,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sPodCrashLoopBackOff","constraint_name":"container-crashloopbackoff","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"25"}
{"level":"info","ts":1672974654.9991775,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sRequiredObject","constraint_name":"required-hpa","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"21"}
{"level":"info","ts":1672974654.9991877,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sBlockingPDB","constraint_name":"blocking-pdb","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"1"}
{"level":"info","ts":1672974654.9992023,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sMisconfiguredHPA","constraint_name":"misconfigured-hpa","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"22"}
{"level":"info","ts":1672974654.9992297,"logger":"controller","msg":"auditing is complete","process":"audit","audit_id":"2023-01-06T01:49:45Z","event_type":"audit_finished"}
bhattchaitanya commented 1 year ago

@maxsmythe the place in the log where you see ... is where I have truncated the logs coz its almost repetitive. nothing interesting happened the same messages kept repeating.

maxsmythe commented 1 year ago

Thank you for the logs!

It looks like the runtime is ~80 minutes.

TBH I don't think I have enough data to root cause as-is. Adding a log line for when scraping is complete and evaluation begins would help disambiguate slow prep from slow execution.

Also, @acpana, the ability to surface per-template metrics like here:

https://github.com/open-policy-agent/gatekeeper/pull/2469

would help us know if there is a specific constraint template that is slow and that might imply a specific cause/fix.

@bhattchaitanya are you able to run a binary built off a PR rather than a proper release?

bhattchaitanya commented 1 year ago

do the PR versions have a docker image in docker hub? if yes, then we can test it @maxsmythe

maxsmythe commented 1 year ago

IIRC we create an image for every merged PR.

Lemme create a PR that adds a log line...

maxsmythe commented 1 year ago

2503 should at least give us an idea of where the majority of time is spent

ritazh commented 1 year ago

@ritazh should we add a "resource listing complete" log line?

Sorry for the delay and thank you for opening the PR!

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.