backupstoragelocation.velero.io default not found

ahmadhajali commented 4 years ago

After upgrading Velero from v1.1.0 to v1.3.1 we are encountering with the following issue: We use the following setting:

configuration:
  | # Cloud provider being used (e.g. aws, azure, gcp).
  | provider: aws
  |  
  | # Parameters for the `default` BackupStorageLocation. See
  | # https://velero.io/docs/v1.0.0/api-types/backupstoragelocation/
  | backupStorageLocation:
      | # Cloud provider where backups should be stored. Usually should
      | # match `configuration.provider`. Required.
      | name: aws

As mentioned in the comment above configuration.backupStorageLocation.name usually should match configuration.provider.

If the name is set to "aws", we getting backupstoragelocation.velero.io default not found errors, and no backups are done. In addition, metrics are all zero except velero_backup_total.

While if name is set to "default", backups occur and metrics are showing all values.

Is it a bug?

Best regards, Ahmad

CarpathianUA commented 4 years ago

Experiencing the same

yadachi commented 4 years ago

it was same for me. but it turns out that I need to specify storage location in my backup by default a backup job look for a storage location name default not aws

ahmadhajali commented 4 years ago

I changed the storage location to default, but in the previous velero version, it accepts different values. Is it a new behavior?

yadachi commented 4 years ago

looks like it changed from default to backupStorageLocation.name recently here if you leave name empty then, it will fall back to default but the comment does mislead you bit there. I wonder if we can pass --default-backup-storage-location option that cli installation has in helm chart.

ahmadhajali commented 4 years ago

will give it a try

optimus-kart commented 4 years ago

Facing the same issue and solutions?

optimus-kart commented 4 years ago

And seriously the documentation needs an update :/

adamrushuk commented 4 years ago

Facing the same issue and solutions?

I've just had this error, and fixed it.

If you've specified a value for configuration.backupStorageLocation.name, ensure you specify the same name in schedules.<YOUR_SCHEDULE_NAME>.template.storageLocation

dguyhasnoname commented 4 years ago

Here is what worked for me:


  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.1.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

metrics:
  enabled: true
  scrapeInterval: 30s

  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8085"
    prometheus.io/path: "/metrics"

  serviceMonitor:
    enabled: true
    additionalLabels: {}    

configuration:
  provider: aws
  backupStorageLocation:
    name: aws
    bucket: ${bucket-name}
    prefix: backup
    config:
      kmsKeyId: $KeyId}
      region: ${region}
  volumeSnapshotLocation:
    name: aws
    config:
      region: ${region}
  logLevel: debug

credentials:
  useSecret: false

deployRestic: false   

schedules:
  hourly-backup:
    schedule: "* * * * *"
    template:
      includeClusterResources: true
      includedNamespaces:
      - '*'
      includedResources:
      - '*'
      storageLocation: aws
      snapshotVolumes: true      
      ttl: 720h0m0s```

bclouser commented 4 years ago

Not sure what the state of this was. I have been hitting a similar (if not identical) issue in the latest 2.12.0 helm chart installed fresh when trying to create a backup. Worth noting i am using helm3

I am able to "launch" the backup process, but it fails with validation error:

Phase:  FailedValidation

Validation errors:  a BackupStorageLocation CRD with the name specified in the backup spec needs to be created before this backup can be executed. Error: backupstoragelocation.velero.io "default" not found

I have tried quite a number of things, including adding --exclude-resource crd and this issue remains.

There is quite some discussion around crds and how to handle them in the issues here and in the main velero repo, so perhaps things are just churning right now.... it does seem that 2.12.0 isn't the most stable of releases :stuck_out_tongue:

davidgibbons commented 4 years ago

@bclouser I believe I got it working with that version of the helm chart. The main thing I did was specify a non-default name for the backupstoragelocation and then remove any of the resources that pointed to the 'default' location.

mshivanna commented 4 years ago

Release 2.12.0 had BackupStorageLocation crd's name hardcoded to default.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default

Fixed in latest version


{{- if .Values.backupsEnabled }}
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: {{ include "velero.backupStorageLocation.name" . }}

bclouser commented 4 years ago

Hmmm

I have blown away all the crds created from velero's helm chart and then installed fresh with v2.12.0 (a number of times)

my config file looks like this:

configuration:
  provider: aws
  backupStorageLocation:
    name: aws
    bucket: tx-eks-torizon-k8s-backup
    config:
      region: us-east-1
  volumeSnapshotLocation:
    name: aws
    config:
      region: us-east-1
snapshotsEnabled: true
credentials:
  existingSecret: cloud-credentials
# Backup schedules to create.
schedules:
  aws-default:
    schedule: "0 0 * * *"
    template:
      ttl: "240h"
      includedNamespaces:
       - default
  aws-others:
      schedule: "0 0 * * *"
      template:
        ttl: "240h"
        excludedNamespaces:
         - default
        snapshotVolumes: false
initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.1.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

and when i create a backup i see the same error, and I even went in and described the BackupResource using kubectl

$ k describe backup bens-totally-random-backup3 -n velero
Name:         bens-totally-random-backup3
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>
API Version:  velero.io/v1
Kind:         Backup
Metadata:
  Creation Timestamp:  2020-07-23T21:05:00Z
  Generation:          2
  Resource Version:    53345180
  Self Link:           /apis/velero.io/v1/namespaces/velero/backups/bens-totally-random-backup3
  UID:                 2bc8ad74-cd28-11ea-aa0a-0e379dd2f475
Spec:
  Hooks:
  Included Namespaces:
    *
  Storage Location:  default
  Ttl:               720h0m0s
  Volume Snapshot Locations:
    aws
Status:
  Expiration:  2020-08-22T21:05:00Z
  Phase:       FailedValidation
  Validation Errors:
    a BackupStorageLocation CRD with the name specified in the backup spec needs to be created before this backup can be executed. Error: backupstoragelocation.velero.io "default" not found
  Version:  1
Events:     <none>

notice the: Storage Location: default

Additionally, on the cli i found this command

$ velero backup-location get
NAME   PROVIDER   BUCKET/PREFIX               ACCESS MODE
aws    aws        tx-eks-torizon-k8s-backup   ReadWrite

so according to cli, velero has the correct location set, however it uses "default" during resource creation :/

I am starting to dig through the code to find out where exactly velero is getting this BackupLocation... Kind of surprised I am the only one with this issue. Our AWS cluster is older than most, currently using 1.13.12 eks so perhaps an api request is failing silently in the velero golang app or is handling unexpected data in a response and therefore uses "default" :shrug: