Updating CRD's not working failed to upgrade with helm

[ujwala02]

$ helm list --failed -n velero NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION velero velero 2 2022-03-03 23:30:10.7192487 +0530 IST failed velero-2.28.0 1.8.0

$ helm upgrade velero vmware-tanzu/velero --namespace velero --reuse-values Error: UPGRADE FAILED: post-upgrade hooks failed: job failed: BackoffLimitExceeded

we are using helm version 3

[jenting]

It seems like pulling the kubectl image can succeed, probably hit the docket hub rate limit.

[mshivanna]

In our case the upgrade job fails with this error The CustomResourceDefinition "restores.velero.io" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema customresourcedefinition.apiextensions.k8s.io/volumesnapshotlocations.velero.io unchanged

[jenting]

In our case the upgrade job fails with this error The CustomResourceDefinition "restores.velero.io" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema customresourcedefinition.apiextensions.k8s.io/volumesnapshotlocations.velero.io unchanged

Which Kubernetes version you are running?

[mshivanna]

1.18

[jenting]

1.18

And which chart version?

[mshivanna]

2.29.3

[robertchen]

I got same issue, the upgrade crds pod has this error:

$ k-n util logs velero-upgrade-crds-9b654 Unable to connect to the server: dial tcp 10.96.0.1:443: i/o timeout

[mshivanna]

1.18

On Tue, Mar 29, 2022 at 7:19 PM JenTing Hsiao @.***> wrote:

In our case the upgrade job fails with this error The CustomResourceDefinition "restores.velero.io" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema customresourcedefinition.apiextensions.k8s.io/volumesnapshotlocations.velero.io unchanged

Which Kubernetes version you are running?

— Reply to this email directly, view it on GitHub https://github.com/vmware-tanzu/helm-charts/issues/352#issuecomment-1082460361, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWMRRRD2E3QTZV4OHEVLN3VCOFWTANCNFSM5P3IHELA . You are receiving this because you commented.Message ID: @.***>

[tobiasehlert]

I encounter the same issue on my ~1.19~ 1.23 EKS cluster. :(

Noticed also following error in the initcontainer:

exec /bin/sh: exec format error

Followed following installation process: https://aws.amazon.com/blogs/containers/backup-and-restore-your-amazon-eks-cluster-resources-using-velero/

Here is my values.yaml:

configuration:
  backupStorageLocation:
    - bucket: my-velero-backup-on-s3
  volumeSnapshotLocation:
    - config:
        region: eu-north-1
credentials:
  useSecret: false
initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.7.0
    volumeMounts:
      - mountPath: /target
        name: plugins
serviceAccount:
  server:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::awsaccountid:role/velero-backup"

[jenting]

Noticed also following error in the initcontainer:

exec /bin/sh: exec format error

Could you please check that your host machine is x86/arm64/arm or elsewhere? The default kubectl container image supports x86 only

[tobiasehlert]

Noticed also following error in the initcontainer:

exec /bin/sh: exec format error

Could you please check that your host machine is x86/arm64/arm or elsewhere? The default kubectl container image supports x86 only

The EKS cluster is running with t4g.large instances, so that should be arm64 due to the AWS Graviton2 Processor. Also tried to run it on my Macbook M1 Pro which is linux/arm64/v8.

[jenting]

Please customize to use another kubectl image that supports arm64.

[tobiasehlert]

Please customize to use another kubectl image that supports arm64.

Based on the AWS Console it says OS (Architecture): linux (arm64) on the worker nodes. And on the Docker Hub velero/velero-plugin-for-aws it says it supports linux/arm64.

Also to be more correct about versioning for me it's K8s release 1.23.17 on EKS release eks.8.

[jenting]

Yes, but the problem is the kubectl container image. You could refer to issue #339 for reference.