Helm Chart within terraform / terragrunt not working

[PassionateDeveloper86]

Hi,

I have installed Velero yesterday manuel - which worked fine.

Now I try to automate it with terraform for each of our stages (using terragrunt here) and your helm chart:

provider "kubernetes" {
  host                   = "${var.kube_host}"
  client_certificate     = "${var.kube_client_certificate}"
  client_key             = "${var.kube_client_key}"
  cluster_ca_certificate = "${var.kube_cluster_ca_certificate}"
}

provider "helm" {
  kubernetes {
    host                   = "${var.kube_host}"
    client_certificate     = "${var.kube_client_certificate}"
    client_key             = "${var.kube_client_key}"
    cluster_ca_certificate = "${var.kube_cluster_ca_certificate}"
  }
}

locals {
  namespace = "velero"
}

resource "kubernetes_namespace" "ns" {
  metadata {
    name = local.namespace
  }
}

resource "helm_release" "velero" {
  name       = "velero"
  repository = "https://vmware-tanzu.github.io/helm-charts/"
  chart      = "velero"
  namespace  = local.namespace

  # Provider Azure
  set {
    name  = "configuration.provider"
    value = "azure"
  }

  # Do I need this? 
#  set {
#    name  = "image.repository"
#    value = "velero/velero"
#  }
#
#  set {
#    name  = "initContainers[0].image"
#    value = "velero/velero-plugin-for-microsoft-azure:v1.5.0"
#  }
#
#  set {
#    name  = "initContainers[0].name"
#    value = "velero-plugin-for-microsoft-azure"
#  }

  # Backup Config
  set {
    name  = "configuration.backupStorageLocation.name"
    value = "azure"
  }

  set {
    name  = "configuration.backupStorageLocation.bucket"
    value = "backupcontainer" # Azure Storage Container Name
  }

  set {
    name  = "configuration.backupStorageLocation.config.subscriptionId"
    value = "02787d15-7b73-431c-b99b-435bf2bcfa24"
  }

  set {
    name  = "configuration.backupStorageLocation.config.storageAccount"
    value = "onebackup" # Azure Storage Name
  }

  set {
    name  = "configuration.backupStorageLocation.config.resourceGroup"
    value = "rg-backup-westeurope" # Region only for backups 
  }

  # snapshot config
  set {
    name  = "configuration.volumeSnapshotLocation.config.resourceGroup"
    value = "rg-backup-westeurope" # Region only for backups 
  }

  set {
    name  = "configuration.volumeSnapshotLocation.config.subscriptionId"
    value = "02787d15-7b73-431c-b99b-435bf2bcfa24"
  }
}

I am very unsure what to set here-.

1) Whenever I set the image, it downloads the image within an init container but it never starts:

I tried it iwth 1.5.0 and 1.6.0 but no success.

2) When I start it without an image given, the service and deployment runs but gives following error:

time="2022-12-07T09:22:13Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:181"
time="2022-12-07T09:22:13Z" level=info msg="Starting Velero server v1.9.4 (ddfc962282783cf2f0bf364c9d721f88fa4cc058-dirty)" logSource="pkg/cmd/server/server.go:183"
time="2022-12-07T09:22:13Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:187"
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/admission-webhook-configuration
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/apiservice
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/init-restore-hook
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2022-12-07T09:22:13Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2022-12-07T09:22:14Z" level=info msg="Metrics server is starting to listen" addr=":8080" logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" logger=controller-runtime.metrics
time="2022-12-07T09:22:14Z" level=info msg="Checking existence of namespace." logSource="pkg/cmd/server/server.go:397" namespace=velero
time="2022-12-07T09:22:14Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:403" namespace=velero
I1207 09:22:16.278092       1 request.go:601] Waited for 1.047340164s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/spv.no/v1alpha1?timeout=32s
time="2022-12-07T09:22:17Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:432"
time="2022-12-07T09:22:17Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:466"
time="2022-12-07T09:22:17Z" level=warning msg="Velero restic daemonset not found; restic backups/restores will not work until it's created" logSource="pkg/cmd/server/server.go:529"
time="2022-12-07T09:22:18Z" level=info msg="Starting controllers" logSource="pkg/cmd/server/server.go:590"
time="2022-12-07T09:22:18Z" level=info msg="Backup sync period is 1m0s" logSource="pkg/controller/backup_sync_controller.go:83"
time="2022-12-07T09:22:18Z" level=info msg="Garbage collection frequency: 1h0m0s" logSource="pkg/controller/gc_controller.go:85"
time="2022-12-07T09:22:18Z" level=info msg="Waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:778"
time="2022-12-07T09:22:18Z" level=info msg="Starting metric server at address [:8085]" logSource="pkg/cmd/server/server.go:597"
time="2022-12-07T09:22:19Z" level=info msg="Done waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:781"
time="2022-12-07T09:22:19Z" level=info msg="Informer cache synced" informer="*v1.ResticRepository" logSource="pkg/cmd/server/server.go:792"
time="2022-12-07T09:22:19Z" level=info msg="Informer cache synced" informer="*v1.Backup" logSource="pkg/cmd/server/server.go:792"
time="2022-12-07T09:22:19Z" level=info msg="Informer cache synced" informer="*v1.VolumeSnapshotLocation" logSource="pkg/cmd/server/server.go:792"
time="2022-12-07T09:22:19Z" level=info msg="Informer cache synced" informer="*v1.DeleteBackupRequest" logSource="pkg/cmd/server/server.go:792"
time="2022-12-07T09:22:19Z" level=info msg="Informer cache synced" informer="*v1.Restore" logSource="pkg/cmd/server/server.go:792"
time="2022-12-07T09:22:19Z" level=info msg="Server starting..." logSource="pkg/cmd/server/server.go:871"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocation"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocationList"
time="2022-12-07T09:22:19Z" level=info msg="Starting Controller" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=resticrepository controllerGroup=velero.io controllerKind=ResticRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.ResticRepository"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=resticrepository controllerGroup=velero.io controllerKind=ResticRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.ResticRepositoryList"
time="2022-12-07T09:22:19Z" level=info msg="Starting Controller" controller=resticrepository controllerGroup=velero.io controllerKind=ResticRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=serverstatusrequest controllerGroup=velero.io controllerKind=ServerStatusRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.ServerStatusRequest"
time="2022-12-07T09:22:19Z" level=info msg="Starting Controller" controller=serverstatusrequest controllerGroup=velero.io controllerKind=ServerStatusRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.DeleteBackupRequest"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.DeleteBackupRequestList"
time="2022-12-07T09:22:19Z" level=info msg="Starting Controller" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2022-12-07T09:22:19Z" level=info msg="Starting server" addr="{\"IP\":\"::\",\"Port\":8080,\"Zone\":\"\"}" kind=metrics logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" path=/metrics
time="2022-12-07T09:22:19Z" level=info msg="Starting controller" controller=backup-sync logSource="pkg/controller/generic_controller.go:76"
time="2022-12-07T09:22:19Z" level=info msg="Starting controller" controller=gc logSource="pkg/controller/generic_controller.go:76"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=downloadrequest controllerGroup=velero.io controllerKind=DownloadRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.DownloadRequest"
time="2022-12-07T09:22:19Z" level=info msg="Starting controller" controller=restore logSource="pkg/controller/generic_controller.go:76"
time="2022-12-07T09:22:19Z" level=info msg="Starting Controller" controller=downloadrequest controllerGroup=velero.io controllerKind=DownloadRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2022-12-07T09:22:19Z" level=info msg="Starting controller" controller=backup logSource="pkg/controller/generic_controller.go:76"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Schedule"
time="2022-12-07T09:22:19Z" level=info msg="Starting EventSource" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.ScheduleList"
time="2022-12-07T09:22:19Z" level=info msg="Starting Controller" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2022-12-07T09:22:19Z" level=info msg="Starting workers" controller=serverstatusrequest controllerGroup=velero.io controllerKind=ServerStatusRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=10
time="2022-12-07T09:22:19Z" level=info msg="Starting workers" controller=downloadrequest controllerGroup=velero.io controllerKind=DownloadRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2022-12-07T09:22:19Z" level=info msg="Starting workers" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2022-12-07T09:22:19Z" level=error msg="Error getting backup store for this location" backupLocation=azure controller=backup-sync error="unable to locate ObjectStore plugin named velero.io/azure" logSource="pkg/controller/backup_sync_controller.go:182"
time="2022-12-07T09:22:19Z" level=info msg="Starting workers" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2022-12-07T09:22:19Z" level=info msg="Starting workers" controller=resticrepository controllerGroup=velero.io controllerKind=ResticRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2022-12-07T09:22:19Z" level=info msg="Starting workers" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2022-12-07T09:22:19Z" level=error msg="Error getting a backup store" backup-storage-location=velero/azure controller=backup-storage-location error="unable to locate ObjectStore plugin named velero.io/azure" logSource="pkg/controller/backup_storage_location_controller.go:127"
time="2022-12-07T09:22:19Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/azure controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:110"
time="2022-12-07T09:22:19Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1, BackupStorageLocation \"azure\" is unavailable: unable to locate ObjectStore plugin named velero.io/azure)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:171"
time="2022-12-07T09:22:19Z" level=warning msg="There is no existing BackupStorageLocation set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:180"
time="2022-12-07T09:22:49Z" level=error msg="Error getting backup store for this location" backupLocation=azure controller=backup-sync error="unable to locate ObjectStore plugin named velero.io/azure" logSource="pkg/controller/backup_sync_controller.go:182"
time="2022-12-07T09:23:19Z" level=error msg="Error getting backup store for this location" backupLocation=azure controller=backup-sync error="unable to locate ObjectStore plugin named velero.io/azure" logSource="pkg/controller/backup_sync_controller.go:182"
time="2022-12-07T09:23:19Z" level=error msg="Error getting a backup store" backup-storage-location=velero/azure controller=backup-storage-location error="unable to locate ObjectStore plugin named velero.io/azure" logSource="pkg/controller/backup_storage_location_controller.go:127"
time="2022-12-07T09:23:19Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/azure controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:110"
time="2022-12-07T09:23:19Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"azure\" is unavailable: unable to locate ObjectStore plugin named velero.io/azure)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:171"
time="2022-12-07T09:23:19Z" level=warning msg="There is no existing BackupStorageLocation set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:180"
time="2022-12-07T09:23:49Z" level=error msg="Error getting backup store for this location" backupLocation=azure controller=backup-sync error="unable to locate ObjectStore plugin named velero.io/azure" logSource="pkg/controller/backup_sync_controller.go:182"

[ywk253100]

Could you check the logs of the init container? Why it stay in CrashBackoff status?

[PassionateDeveloper86]

I dont know how to access the data of the init container. I tried 'kubectl logs PODNAME --previous' but didnt get any details. How to access details of the init container?

[ywk253100]

Specify the init container name with -c option

[PassionateDeveloper86]

Hi,

here are all outputs I get:

rene [ ~ ]$ kubectl logs deployment/velero -n velero Defaulted container "velero" out of: velero, velero-plugin-for-microsoft-azure (init) Error from server (BadRequest): container "velero" in pod "velero-6f4449bdcd-z8hc5" is waiting to start: PodInitializing

rene [ ~ ]$ kubectl logs deployment/velero -n velero -c velero-plugin-for-microsoft-azure cp: can't create '/target/.': No such file or directory

[PassionateDeveloper86]

Any news?

[ywk253100]

@PassionateDeveloper86 Seems you use a dev version of the helm chart rather than a released one and hit one known issue that has already been fixed.

Please use the release version of Velero helm chart.

[PassionateDeveloper86]

@ywk253100 Sorry but im still very lost.

1) Which version should I use? 2) How to use not a dev version? As far as I can see, using the latest is / should not the dev. But if the latest is the dev, what is a version I should use and how to select it in terraform files (please provide a valid file that will work with azure) 3) Should I use the image settings or not? (if I use not, then there is the reported error, if I use, then there there is the second reported error)

Please provide your answer fast then 2 weeks this time!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[PassionateDeveloper86]

@ywk253100 Sorry but im still very lost.

Which version should I use?
How to use not a dev version?
As far as I can see, using the latest is / should not the dev. But if the latest is the dev, what is a version I should use and how to select it in terraform files (please provide a valid file that will work with azure)
Should I use the image settings or not?
(if I use not, then there is the reported error, if I use, then there there is the second reported error)

Please provide your answer fast then 2 weeks this time!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closing the stale issue.