rondy-cg commented 1 year ago

Background

I was trying to install istio 1.16.0 via helmCharts + kustomization, after adding helmCharts parts to kustomization.yaml, I pushed the changes to Github and triggered sanity checks via kubeconform, then got one unexpected error saying "Could not find schema for HorizontalPodAutoscaler". So I tried to add horizontalpodautoscaler-autoscaling-v2beta1.json to our schema folder, but still same error occurred, wondering where exactly kubeconform was looking for the schema, can anyone help to advise?

kubeconform commands in Github actions

set -o pipefail

  paths=$(find k8s/argocd/ -type f -regextype posix-egrep -regex ".*\.(yaml|yml)$" | xargs -I {} yq eval 'select(.kind=="Application" and .spec.source.path != null) | .spec.source.path' {})

  for path in ${paths}; do
    echo "==============================="
    echo "check_path: ${path}"

    if [[ -f ${path}/kustomization.yaml ]]; then
      kustomize build --enable-helm ${path} | kubeconform -kubernetes-version ${kubernetes_version} -summary -schema-location default -schema-location "./schemas/{{ .ResourceKind }}{{ .KindSuffix }}.json" -skip CustomResourceDefinition -strict
    else
      kubeconform -kubernetes-version ${kubernetes_version} -summary -schema-location default -schema-location "./schemas/{{ .ResourceKind }}{{ .KindSuffix }}.json" -skip CustomResourceDefinition -strict ${path}
    fi
  done
  shell: /usr/bin/bash -e {0}
  env:
    kubernetes_version: 1.21.4

error output

===============================
check_path: k8s/istio-system/istio-common/overlays/dev
stdin - HorizontalPodAutoscaler istiod-1-16-0 failed validation: could not find schema for HorizontalPodAutoscaler

directory of the problematic path

k8s/istio-system/istio-common
├── base
│   ├── istio-ingressgateway-certs.yaml
│   ├── kustomization.yaml
│   └── values.yaml
└── overlays
    ├── demo
    │   ├── istio-ingressgateway-certs.yaml
    │   └── kustomization.yaml
    ├── dev
    │   ├── istio-ingressgateway-certs.yaml
    │   └── kustomization.yaml
    ├── prod
    │   ├── istio-ingressgateway-certs.yaml
    │   └── kustomization.yaml
    └── test
        ├── istio-ingressgateway-certs.yaml
        └── kustomization.yaml

suspicious kustomization

k8s/istio-system/istio-common/base/kustomization.yaml


apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: istio-system

resources:

istio-ingressgateway-certs.yaml

helmCharts:

name: istiod repo: https://istio-release.storage.googleapis.com/charts version: 1.16.0 releaseName: istiod namespace: istio-system valuesFile: values.yaml


# json schemas file list
- horizontalpodautoscaler-autoscaling-v2beta1.json was added to resolve the error from the file below
    - https://github.com/yannh/kubernetes-json-schema/blob/master/v1.21.4/horizontalpodautoscaler-autoscaling-v2beta1.json

schemas ├── README.md ├── application-argoproj-v1alpha1.json ├── appproject-argoproj-v1alpha1.json ├── authorizationpolicy-security-v1beta1.json ├── clusterexternalsecret-external-secrets-v1beta1.json ├── clustersecretstore-external-secrets-v1alpha1.json ├── clustersecretstore-external-secrets-v1beta1.json ├── destinationrule-networking-v1alpha3.json ├── destinationrule-networking-v1beta1.json ├── envoyfilter-networking-v1alpha3.json ├── externalsecret-external-secrets-v1alpha1.json ├── externalsecret-external-secrets-v1beta1.json ├── externalsecret-kubernetes-client-v1.json ├── gateway-networking-v1alpha3.json ├── gateway-networking-v1beta1.json ├── horizontalpodautoscaler-autoscaling-v2beta1.json ├── istiooperator-install-v1alpha1.json ├── monitoringdashboard-monitoring-v1alpha1.json ├── peerauthentication-security-v1beta1.json ├── proxyconfig-networking-v1beta1.json ├── requestauthentication-security-v1beta1.json ├── secretstore-external-secrets-v1alpha1.json ├── secretstore-external-secrets-v1beta1.json ├── serviceentry-networking-v1alpha3.json ├── serviceentry-networking-v1beta1.json ├── sidecar-networking-v1alpha3.json ├── sidecar-networking-v1beta1.json ├── telemetry-telemetry-v1alpha1.json ├── virtualservice-networking-v1alpha3.json ├── virtualservice-networking-v1beta1.json ├── wasmplugin-extensions-v1alpha1.json ├── workloadentry-networking-v1alpha3.json ├── workloadentry-networking-v1beta1.json ├── workloadgroup-networking-v1alpha3.json └── workloadgroup-networking-v1beta1.json

rondy-cg commented 1 year ago

FYI the debug info below ...

% kustomize build --enable-helm k8s/istio-system/istio-common/overlays/dev | kubeconform -kubernetes-version 1.21.4 -summary -schema-location default -schema-location "./schemas/{{ .ResourceKind }}{{ .KindSuffix }}.json" -skip CustomResourceDefinition -strict -debug 
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/clusterrole-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/clusterrole-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/serviceaccount-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/role-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/rolebinding-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/clusterrolebinding-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/clusterrolebinding-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/clusterrolebinding-rbac-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/configmap-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/configmap-v1.json
2023/02/02 13:46:55 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/service-v1.json
2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/horizontalpodautoscaler-autoscaling-v2.json
2023/02/02 13:46:56 could not open file ./schemas/horizontalpodautoscaler-autoscaling-v2.json
stdin - HorizontalPodAutoscaler istiod-1-16-0 failed validation: could not find schema for HorizontalPodAutoscaler
2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/externalsecret-external-secrets-v1beta1.json
2023/02/02 13:46:56 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/poddisruptionbudget-policy-v1.json
2023/02/02 13:46:56 using schema found at ./schemas/externalsecret-external-secrets-v1beta1.json
2023/02/02 13:46:56 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/deployment-apps-v1.json
2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 using schema found at ./schemas/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 using schema found at ./schemas/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 using schema found at ./schemas/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 using schema found at ./schemas/envoyfilter-networking-v1alpha3.json
2023/02/02 13:46:56 using schema found at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/mutatingwebhookconfiguration-admissionregistration-v1.json
Summary: 25 resources found parsing stdin - Valid: 24, Invalid: 0, Errors: 1, Skipped: 0

Wondering why it's looking for horizontalpodautoscaler-autoscaling-v2.json, can anyone help to advise?

2023/02/02 13:46:56 could not find schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.21.4-standalone-strict/horizontalpodautoscaler-autoscaling-v2.json
2023/02/02 13:46:56 could not open file ./schemas/horizontalpodautoscaler-autoscaling-v2.json
stdin - HorizontalPodAutoscaler istiod-1-16-0 failed validation: could not find schema for HorizontalPodAutoscaler

eyarz commented 1 year ago

it looks like you pinned it down to the right issue - kubeconform is looking for horizontalpodautoscaler-autoscaling-v2 instead of horizontalpodautoscaler-autoscaling-v2beta1.

I tried to reproduce it on my side with no luck:

are you also using the latest version of kubeconform?

rondy-cg commented 1 year ago

@eyarz thanks a lot for your help!

are you also using the latest version of kubeconform?

Yes, we were using the latest version 0.5.0

eyarz commented 1 year ago

The manifest api you're testing (e.g. apiVersion: autoscaling/<version>) is v2 or v2beta1?

wl21st commented 1 year ago

I have met the similar issue today and for HPA, the version v2beta1 is removed in the master version and the last appearance is v1.24.9. In my case, I am using v2beta2 version of HPA, the last appearance is v1.25.6.

Pass the k8s version in the command line resolves the problem. For ex. kubeconform --verbose --kubernetes-version 1.24.9 hpa.yaml fixed my issue.

eyarz commented 1 year ago

@wl21st autoscaling/v2beta1 was removed in version 1.25. I don't think this is the same issue because @rondy-cg mentioned that he is using version 1.21.4.

wl21st commented 1 year ago

@wl21st autoscaling/v2beta1 was removed in version 1.25. I don't think this is the same issue because @rondy-cg mentioned that he is using version 1.21.4.

I see, didn't notice that the k8s version is specified.

rondy-cg commented 1 year ago

@eyarz Sorry for the late response. It's confirmed that the input manifest was being generated with an unexpected version v2 via "kustomize build --enable-helm", hence kubeconform was looking for horizontalpodautoscaler-autoscaling-v2.json, which was normal behavior I think.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler

Anyway, it should not be a kubeconform issue, so I'd like to close it shortly. Sorry for the confusions caused. 🙇‍♂️

eyarz commented 1 year ago

no worries you're right, if your HPA api is v2 this is the expected behavior :)

yannh / kubeconform

Could not find schema for HorizontalPodAutoscaler #174

Background

kubeconform commands in Github actions

error output

directory of the problematic path

suspicious kustomization