search

kyverno / kyverno

Cloud Native Policy Management
https://kyverno.io
Apache License 2.0
5.54k stars 841 forks source link

[Bug] reports-controller should not check deprecated API groups #7934

Open mfilocha opened 1 year ago

mfilocha commented 1 year ago

Kyverno Version

1.10.1

Description

reports-controller creates watchers for every API endpoint it can list with kubectl api-versions. For some groups, like autoscaling, there are both autoscaling/v2beta2 (deprecated) and autoscaling/v2 present on Kubernetes 1.25.

That gives at least two problems:

Slack discussion

No response

Troubleshooting

welcome[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template!

mfilocha commented 1 year ago

I0728 15:24:52.480480 1 controller.go:173] resource-report-controller "msg"="start watcher ..." "gvk"={"Group":"autoscaling","Version":"v2","Kind":"HorizontalPodAutoscaler"} "gvr"={"Group":"autoscaling","Version":"v2","Resource":"horizontalpodautoscalers"} "resourceVersion"="1114718682" I0728 15:24:52.480556 1 controller.go:175] resource-report-controller "msg"="creating watcher..." "gvk"={"Group":"autoscaling","Version":"v2","Kind":"HorizontalPodAutoscaler"} "gvr"={"Group":"autoscaling","Version":"v2","Resource":"horizontalpodautoscalers"} "resourceVersion"="1114718682" I0728 15:24:53.385720 1 controller.go:173] resource-report-controller "msg"="start watcher ..." "gvk"={"Group":"autoscaling","Version":"v1","Kind":"HorizontalPodAutoscaler"} "gvr"={"Group":"autoscaling","Version":"v1","Resource":"horizontalpodautoscalers"} "resourceVersion"="1114718689" I0728 15:24:53.385841 1 controller.go:175] resource-report-controller "msg"="creating watcher..." "gvk"={"Group":"autoscaling","Version":"v1","Kind":"HorizontalPodAutoscaler"} "gvr"={"Group":"autoscaling","Version":"v1","Resource":"horizontalpodautoscalers"} "resourceVersion"="1114718689" W0728 15:24:53.391060 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler I0728 15:24:53.391668 1 controller.go:173] resource-report-controller "msg"="start watcher ..." "gvk"={"Group":"autoscaling","Version":"v2beta2","Kind":"HorizontalPodAutoscaler"} "gvr"={"Group":"autoscaling","Version":"v2beta2","Resource":"horizontalpodautoscalers"} "resourceVersion"="1114718689" I0728 15:24:53.391755 1 controller.go:175] resource-report-controller "msg"="creating watcher..." "gvk"={"Group":"autoscaling","Version":"v2beta2","Kind":"HorizontalPodAutoscaler"} "gvr"={"Group":"autoscaling","Version":"v2beta2","Resource":"horizontalpodautoscalers"} "resourceVersion"="1114718689" W0728 15:24:53.393306 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler W0728 15:25:23.244502 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler

chipzoller commented 7 months ago

@eddycharly is this still relevant after the changes as of 1.11?

omarfakhreddine commented 7 months ago

I'm seeing similar things in our EKS Upgrade Insights where the reports-controller user agent is hitting deprecated APIs for k8s version 1.26 and giving an error to fix before upgrading. The APIs are the autoscaling/v2beta2, and flowcontrol.apiserver.k8s.io/v1beta1, we have these set in our Kyverno policies to watch for. We safely ignored the errors and upgraded anyway, however it would be nice if this was fixed to not give a false positive.

twmartin commented 7 months ago

@eddycharly is this still relevant after the changes as of 1.11?

@chipzoller - I do still see this false positive behavior in our AWS EKS 1.28 environment with flowcontrol.apiserver.k8s.io/v1beta2. We are running Kyverno 1.11.4.

GeorgiIvanovITGix commented 5 months ago

Hello Kyverno Team and Community,

Following up on the ongoing discussion about the reports-controller and its behavior regarding deprecated API group checks, I would like to share a direct impact scenario that has affected our AKS cluster upgrade process due to this issue.

Environment:

Issue Summary:

During our attempt to upgrade the AKS cluster, we encountered ValidationError UpgradeBlockedOnDeprecatedAPIUsage errors, explicitly pointing to the usage of deprecated API versions by Kyverno's reports-controller. Specifically, the APIs in question were related to flowcontrol.apiserver.k8s.io/v1beta2 for FlowSchemas and PriorityLevelConfigurations.

Error Details:

Azure AKS blocked our cluster upgrade attempt due to recent usage of APIs that are deprecated in the version we are upgrading to. The detailed error message from AKS indicated that usage of flowcontrol.apiserver.k8s.io.flowschemas.v1beta2 and flowcontrol.apiserver.k8s.io.prioritylevelconfigurations.v1beta2 was detected, which are slated for removal in Kubernetes 1.29.

Observations: I observed that the reports-controller component of Kyverno is querying deprecated API groups. This behavior is not only flagged by Azure Kubernetes Service (AKS) during our cluster upgrade attempt but is also evident from Kyverno's own logs. Specifically, the usage of flowcontrol.apiserver.k8s.io/v1beta2 for FlowSchemas and PriorityLevelConfigurations has been highlighted as deprecated, even though newer, non-deprecated API versions are available. This behavior of the reports-controller directly leads to the upgrade blockade in AKS, as it's the source of the deprecated API usage detected by Azure.

Logs: Here are snippets from the logs that demonstrate the deprecated API calls being made by Kyverno's reports-controller:

W0408 10:53:32.422036       1 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta2 FlowSchema is deprecated in v1.26+, unavailable in v1.29+; use flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema

These log entries clearly indicate that despite the availability of a newer API version (v1beta3), the deprecated v1beta2 version is still being utilized. This aligns with the error messages received during the AKS cluster upgrade process, confirming the source of the deprecated API usage that led to the upgrade being blocked.

Impact:

This issue has a significant impact on our ability to seamlessly upgrade our Kubernetes clusters within Azure AKS. The AKS service now automatically stops upgrade operations involving a minor version change if deprecated APIs are detected, which is a sensible default to prevent post-upgrade failures. However, it places us in a difficult position where we must either bypass this safety mechanism (which is not advisable) or wait for a resolution to the Kyverno reports-controller behavior.

ishaikh68 commented 3 months ago

I am also observing this false positive behavior in our AWS EKS 1.28 and the reports-controller (Helm chart version 3.2.1).

vishal-chdhry commented 2 months ago

reports-controller creates watchers for every API endpoint it can list with kubectl api-versions

reports-controller only creates watcher for kinds you have policies for. See: https://github.com/kyverno/kyverno/blob/206c275eff572bf76ed13628d12257fc734142aa/pkg/controllers/report/resource/controller.go#L236-L270

Therefore if you have a matchblock like this:

    match:
      any:
      - resources:
          kinds:
          - flowcontrol.apiserver.k8s.io/*/FlowSchema
          - flowcontrol.apiserver.k8s.io/*/PriorityLevelConfiguration

It will create watchers for both v1beta2 and v1beta3:

$ kubectl logs -n kyverno deploy/kyverno-reports-controller | grep watcher                                     1s
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:186      start watcher ...       {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=flowschemas", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    LEVEL(-4)       klog    watch/retrywatcher.go:246       Starting RetryWatcher.
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:188      creating watcher...     {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=flowschemas", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:186      start watcher ...       {"gvr": "flowcontrol.apiserver.k8s.io/v1beta2, Resource=flowschemas", "gvk": "flowcontrol.apiserver.k8s.io/v1beta2, Kind=FlowSchema", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    LEVEL(-4)       klog    watch/retrywatcher.go:246       Starting RetryWatcher.
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:188      creating watcher...     {"gvr": "flowcontrol.apiserver.k8s.io/v1beta2, Resource=flowschemas", "gvk": "flowcontrol.apiserver.k8s.io/v1beta2, Kind=FlowSchema", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:186      start watcher ...       {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=prioritylevelconfigurations", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=PriorityLevelConfiguration", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    LEVEL(-4)       klog    watch/retrywatcher.go:246       Starting RetryWatcher.
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:188      creating watcher...     {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=prioritylevelconfigurations", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=PriorityLevelConfiguration", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:186      start watcher ...       {"gvr": "flowcontrol.apiserver.k8s.io/v1beta2, Resource=prioritylevelconfigurations", "gvk": "flowcontrol.apiserver.k8s.io/v1beta2, Kind=PriorityLevelConfiguration", "resourceVersion": "1867"}
2024-06-17T09:02:50Z    LEVEL(-4)       klog    watch/retrywatcher.go:246       Starting RetryWatcher.
2024-06-17T09:02:50Z    DEBUG   resource-report-controller      resource/controller.go:188      creating watcher...     {"gvr": "flowcontrol.apiserver.k8s.io/v1beta2, Resource=prioritylevelconfigurations", "gvk": "flowcontrol.apiserver.k8s.io/v1beta2, Kind=PriorityLevelConfiguration", "resourceVersion": "1867"}

But, if you specify the exact version:

    match:
      any:
      - resources:
          kinds:
          - flowcontrol.apiserver.k8s.io/v1beta3/FlowSchema
          - flowcontrol.apiserver.k8s.io/v1beta3/PriorityLevelConfiguration

It will only create a watcher for that exact version, v1beta3 in this case

$ kubectl logs -n kyverno deploy/kyverno-reports-controller | grep watcher
2024-06-17T09:06:01Z    DEBUG   resource-report-controller      resource/controller.go:186      start watcher ...       {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=flowschemas", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema", "resourceVersion": "2588"}
2024-06-17T09:06:01Z    LEVEL(-4)       klog    watch/retrywatcher.go:246       Starting RetryWatcher.
2024-06-17T09:06:01Z    DEBUG   resource-report-controller      resource/controller.go:188      creating watcher...     {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=flowschemas", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema", "resourceVersion": "2588"}
2024-06-17T09:06:01Z    DEBUG   resource-report-controller      resource/controller.go:186      start watcher ...       {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=prioritylevelconfigurations", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=PriorityLevelConfiguration", "resourceVersion": "2588"}
2024-06-17T09:06:01Z    LEVEL(-4)       klog    watch/retrywatcher.go:246       Starting RetryWatcher.
2024-06-17T09:06:01Z    DEBUG   resource-report-controller      resource/controller.go:188      creating watcher...     {"gvr": "flowcontrol.apiserver.k8s.io/v1beta3, Resource=prioritylevelconfigurations", "gvk": "flowcontrol.apiserver.k8s.io/v1beta3, Kind=PriorityLevelConfiguration", "resourceVersion": "2588"}