Open therealdwright opened 3 years ago
I can't reproduce on a brand new cluster:
> cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/elastic/cloud-on-k8s/config/crds/v1/patches?ref=a186ea7028eda2402c90cf84c1cc649afe9bfd3e
> kustomize build > crds.yaml
> ls -la crds.yaml && md5sum crds.yaml
-rw-r--r-- 1 michael wheel 502734 Aug 20 12:20 crds.yaml
aec466e6263448a193a10fc94b1c21b9 crds.yaml
> k version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
> k get crds
NAME CREATED AT
eniconfigs.crd.k8s.amazonaws.com 2021-08-20T10:24:04Z
securitygrouppolicies.vpcresources.k8s.aws 2021-08-20T10:24:09Z
> k apply -f crds.yaml
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
> k get crds
NAME CREATED AT
agents.agent.k8s.elastic.co 2021-08-20T11:16:18Z
apmservers.apm.k8s.elastic.co 2021-08-20T11:16:19Z
beats.beat.k8s.elastic.co 2021-08-20T11:16:20Z
elasticmapsservers.maps.k8s.elastic.co 2021-08-20T11:16:21Z
elasticsearches.elasticsearch.k8s.elastic.co 2021-08-20T11:16:22Z
eniconfigs.crd.k8s.amazonaws.com 2021-08-20T10:24:04Z
enterprisesearches.enterprisesearch.k8s.elastic.co 2021-08-20T11:16:23Z
kibanas.kibana.k8s.elastic.co 2021-08-20T11:16:25Z
securitygrouppolicies.vpcresources.k8s.aws 2021-08-20T10:24:09Z
I am experiencing the same issue going from 1.5.0 to 1.7.1.
As mentioned in the documentation could you try to use kubectl replace
instead of kubectl apply
when upgrading the CRDs ?
Thanks
That did the trick for me. I am using ArgoCD and it complained about a shared resource warning but I just synced the app after running the kubectl replace.
I actually had to sync first to add the crd's then do the replace otherwise it would complain some crd's were not there.
Replace worked for me, I'm in the same position as @coolbry95
I actually had to sync first to add the crd's then do the replace otherwise it would complain some crd's were not there.
Sorry, I'm not familiar with ArgoCD. It seems that ArgoCD has the option to use kubectl replace
instead of kubectl apply
. Do you think it could have been an option in your situation ?
Since I am using helm I could of used the force option described in the link you provided or the replace option as you say. If I did the replace it would be best to select only the CRD's to sync and replace.
Seeing the same issue using flux when attempting to upgrade from 1.6.0 to 1.7.1. We tried setting upgrade.force: true
and upgrade.crds: CreateReplace
but had no success
Seeing the same issue using flux when attempting to upgrade from 1.6.0 to 1.7.1. We tried setting
upgrade.force: true
andupgrade.crds: CreateReplace
but had no success
I am no expert in flux
but regarding upgrade.crds: CreateReplace
I am assuming this refers to the CRD feature in Helm which we re not using in ECK. We are using a separate chart for the CRDs (option 2 on the page I linked). I am assuming you have not created a separate HelmRelease
custom resource for our ECK CRD chart (which you usually won't need). But that dependent CRD chart upgrade is the one you have to force for the install to be successful. This would be a one time operation that might be worth doing manually (once you are on the v1 CRDs subsequent upgrades should work automatically )
Thanks @pebrc, we initially tried something really janky (moving ./deploy/eck-operator/charts/eck-operator-crds/templates/all-crds.yaml
to ./deploy/eck-operator/crds/all-crds.yaml
so that the flux options would be applied to our ECK operator HR but that didn't seem to work.
The one time operation seemed to work once a ./deploy/eck-operator/charts/eck-operator-crds/values.yaml
was created:
$ helm upgrade eck-operator-eck-operator chart/eck-operator-crds -n elastic-system --force
Error: UPGRADE FAILED: template: eck-operator-crds/templates/all-crds.yaml:1:21: executing "eck-operator-crds/templates/all-crds.yaml" at <include "eck-operator-crds.effectiveKubeVersion" .>: error calling include: template: eck-operator-crds/templates/_helpers.tpl:57:14: executing "eck-operator-crds.effectiveKubeVersion" at <.Values.global.manifestGen>: nil pointer evaluating interface {}.manifestGen
# added ./deploy/eck-operator/charts/eck-operator-crds/values.yaml
nameOverride: "elastic-operator-crds"
fullnameOverride: "elastic-operator-crds"
global:
manifestGen: false
kubeVersion: 1.16.0
# crds get updated to 1.7.1
$ helm upgrade eck-operator-eck-operator chart/eck-operator-crds -n elastic-system --force
Release "eck-operator-eck-operator" has been upgraded. Happy Helming!
# sts gets updated to 1.7.1
$ helm upgrade eck-operator-eck-operator chart -n elastic-system
Release "eck-operator-eck-operator" has been upgraded. Happy Helming!
Overall, this manual process may work for us temporarily but if there was a way to resolve the spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
, that may be more "flux friendly" for us
Same issue when upgrading on Rancher via UI AppsV2 from 1.6.0 -> 2.0.0 Docker v20.10 K8s: v1.20
helm upgrade --history-max=5 --install=true --namespace=elastic-system --reset-values=true --timeout=10m0s --values=/home/shell/helm/values-eck-operator-2.0.0.yaml --version=2.0.0 --wait=true eck-operator /home/shell/helm/eck-operator-2.0.0.tgz
checking 17 resources for changes
Looks like there are no changes for ServiceAccount "elastic-operator"
Looks like there are no changes for Secret "elastic-operator-webhook-cert"
Looks like there are no changes for ConfigMap "elastic-operator"
error updating the resource "agents.agent.k8s.elastic.co":
cannot patch "agents.agent.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "agents.agent.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "apmservers.apm.k8s.elastic.co":
cannot patch "apmservers.apm.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "apmservers.apm.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "elasticmapsservers.maps.k8s.elastic.co":
cannot patch "elasticmapsservers.maps.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticmapsservers.maps.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "elasticsearches.elasticsearch.k8s.elastic.co":
cannot patch "elasticsearches.elasticsearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticsearches.elasticsearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "enterprisesearches.enterprisesearch.k8s.elastic.co":
cannot patch "enterprisesearches.enterprisesearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "enterprisesearches.enterprisesearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
error updating the resource "kibanas.kibana.k8s.elastic.co":
cannot patch "kibanas.kibana.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "kibanas.kibana.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
Looks like there are no changes for ClusterRole "elastic-operator"
Looks like there are no changes for ClusterRole "elastic-operator-view"
Looks like there are no changes for ClusterRole "elastic-operator-edit"
Looks like there are no changes for ClusterRoleBinding "elastic-operator"
Looks like there are no changes for Service "elastic-operator-webhook"
Error: UPGRADE FAILED: cannot patch "agents.agent.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "agents.agent.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "apmservers.apm.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "apmservers.apm.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "elasticmapsservers.maps.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticmapsservers.maps.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "elasticsearches.elasticsearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticsearches.elasticsearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "enterprisesearches.enterprisesearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "enterprisesearches.enterprisesearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "kibanas.kibana.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "kibanas.kibana.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
Bug Report
What did you do? Unable to use Kustomize base for CRD's. Attempted to upgrade from version 1.6.0 to 1.7.0. I was able to use the CRD's at
github.com/elastic/cloud-on-k8s/config/crds/v1beta1/patches
but notgithub.com/elastic/cloud-on-k8s/config/crds/v1/patches
What did you expect to see?
A clean apply of CRD's
What did you see instead? Under which circumstances?
The below errors.
Environment
ECK version: 1.7.0 ref=a186ea7028eda2402c90cf84c1cc649afe9bfd3e
Kubernetes information: EKS AWS Version 1.21.2