search

selkies-project / selkies-operator

K8s operator for per-user stateful workloads
https://selkies.io
Apache License 2.0
72 stars 13 forks source link

Failed to apply deployment #53

Open videlanicolas opened 2 years ago

videlanicolas commented 2 years ago

Brand new GCP project, after I run:

$ ACCOUNT=$(gcloud config get-value account) && \
  gcloud builds submit \
    --project=${PROJECT_ID?} \
    --substitutions=_USER=${ACCOUNT?},_REGION=${REGION?}

I get the following error:

Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": error: unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error: failed to apply deployment: failed to apply CustomResourceDefinition configuration file with name "istiocontrolplanes.install.istio.io" to cluster: failed to apply config from string: command to apply kubernetes config from string to cluster failed: exit status 1

I think the error is at https://github.com/selkies-project/selkies/blob/master/setup/manifests/deploy.sh:

log_cyan "Installing CRDs"
gke-deploy apply --project ${PROJECT_ID} --cluster ${CLUSTER_NAME} --location ${CLUSTER_LOCATION} --filename /opt/istio-operator/deploy/crds/istio_v1alpha2_istiocontrolplane_crd.yaml
gke-deploy apply --project ${PROJECT_ID} --cluster ${CLUSTER_NAME} --location ${CLUSTER_LOCATION} --filename base/pod-broker/crd.yaml

But I'm no expert. Will continue debugging this.

videlanicolas commented 2 years ago

Seems like apiextensions.k8s.io/v1beta1 is [deprecated](https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#:~:text=The%20beta%20CustomResourceDefinition%20API%20(apiextensions.k8s.io/v1beta1). We should migrate to apiregistration.k8s.io/v1.

videlanicolas commented 2 years ago

This failes at this setp actually: https://github.com/selkies-project/selkies/blob/8fa16a1eb02562f8bf16adc847f122bd68d88726/setup/manifests/deploy.sh#L69

We need to update /opt/istio-operator/deploy/crds/istio_v1alpha2_istiocontrolplane_crd.yaml to use apiregistration.k8s.io/v1. This file is downloaded from https://github.com/istio/operator, so we need to update https://github.com/istio/operator/blob/master/deploy/crds/istio_v1alpha1_istiooperator_crd.yaml.

videlanicolas commented 2 years ago

Well, the repo referenced above has been archived, so we should migrate this project to use Istio's new repo.

videlanicolas commented 2 years ago

We should also update CNRM version, since they are using the deprecated API as well: https://github.com/selkies-project/selkies/blob/385aa326b7ac9bd8866378cf399194db225d39d2/images/installer/Dockerfile#L26

videlanicolas commented 2 years ago

There's another API that needs to be migrated: admissionregistration.k8s.io/v1beta1.

It appears that Kubernetes 1.22 breaks a lot of things in this repo, we should update it so that fresh installs still work.

videlanicolas commented 2 years ago

I think the solution to this issue is to migrate this project to use https://github.com/istio/istio/tree/master/operator newer repo instead of the archived one.

videlanicolas commented 1 year ago

After #64 I ran git pull on my project and re-deployed, it fixed the error in this issue.

But it encountered another error:


Step #10 - "deploy-cluster-manifests-region": BUILD FAILURE: Build step failure: build step 1 "gcr.io/<PROJECT ID>/broker-installer" failed: step exited with non-zero status: 1
Step #10 - "deploy-cluster-manifests-region": ERROR: (gcloud.builds.submit) build 1a9cd776-fce1-42d1-b4d0-a3d2a7649c35 completed with status "FAILURE"
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": namespace/autoneg-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": serviceaccount/autoneg-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": role.rbac.authorization.k8s.io/autoneg-leader-election-role created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/autoneg-manager-role created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/autoneg-proxy-role created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": rolebinding.rbac.authorization.k8s.io/autoneg-leader-election-rolebinding created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/autoneg-manager-rolebinding created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/autoneg-proxy-rolebinding created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": service/autoneg-controller-manager-metrics-service created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": deployment.apps/autoneg-controller-manager created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Checking existing istio installation
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": no running Istio pods in "istio-system"
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Installing istio 1.14
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Installing Istio control plane...
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/istiooperators.install.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/peerauthentications.security.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/proxyconfigs.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/requestauthentications.security.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/telemetries.telemetry.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/wasmplugins.extensions.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/workloadentries.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": customresourcedefinition.apiextensions.k8s.io/workloadgroups.networking.istio.io created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/istio-reader-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrole.rbac.authorization.k8s.io/istiod-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/istio-reader-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": clusterrolebinding.rbac.authorization.k8s.io/istiod-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-istio-system created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Step #1 - "deploy-manifests": Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found
Step #10 - "deploy-cluster-manifests-region": Finished Step #1 - "deploy-manifests"
Step #10 - "deploy-cluster-manifests-region": ERROR
Step #10 - "deploy-cluster-manifests-region": ERROR: build step 1 "gcr.io/<PROJECT ID>/broker-installer" failed: step exited with non-zero status: 1
Step #10 - "deploy-cluster-manifests-region": --------------------------------------------------------------------------------
videlanicolas commented 1 year ago

Two notable things pop in here:

Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget

This might relate to kustomization.yaml, so maybe increasing the version to v1 would be enough?

And:

Error from server (NotFound): error when creating "STDIN": namespaces "istio-system" not found

I guess this means the previous step failed: Step #10 - "deploy-cluster-manifests-region": ERROR: (gcloud.builds.submit) build 1a9cd776-fce1-42d1-b4d0-a3d2a7649c35 completed with status "FAILURE", not sure what causes this.