Issue running helm on VM Scaleset created from Ubuntu Image

[DevOpsAzurance]

Description

Pipeline runs as AzDevOps and fails with the following even when running:

DevOps@ServerInstance:~$ helm
WARNING: stat /home/<VMSSAdminUserwhencreatingSCaleset>/.config/helm/registry/config.json: permission denied
/home/<VMSSAdminUserwhencreatingSCaleset>/.config/helm/registry/config.json
oras.land/oras-go/pkg/auth/docker.NewClientWithDockerFallback
        oras.land/oras-go@v1.2.4/pkg/auth/docker/client.go:80
helm.sh/helm/v3/pkg/registry.NewClient
        helm.sh/helm/v3/pkg/registry/client.go:84
main.newDefaultRegistryClient
        helm.sh/helm/v3/cmd/helm/root.go:288
main.newRootCmd
        helm.sh/helm/v3/cmd/helm/root.go:156
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:66
runtime.main
        runtime/proc.go:267
runtime.goexit
        runtime/asm_amd64.s:1650

Even though I am logged in as 'DevOps' (not AzDevOps, but same thing), it defaults to the $HOME path of the account used when creating the VMSS resource:

Since this account is not part of the Packer build, it was created only during the spin up of the VMSS instance.

If I run:

DevOps@ServerInstance:~$ sudo helm
The Kubernetes package manager

Common actions for Helm:

- helm search:    search for charts
- helm pull:      download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts

Environment variables:

| Name                               | Description                                                                                                |
|------------------------------------|------------------------------------------------------------------------------------------------------------|
| $HELM_CACHE_HOME                   | set an alternative location for storing cached files.                                                      |
| $HELM_CONFIG_HOME                  | set an alternative location for storing Helm configuration.                                                |
| $HELM_DATA_HOME                    | set an alternative location for storing Helm data.                                                         |
| $HELM_DEBUG                        | indicate whether or not Helm is running in Debug mode                                                      |
| $HELM_DRIVER                       | set the backend storage driver. Values are: configmap, secret, memory, sql.                                |
| $HELM_DRIVER_SQL_CONNECTION_STRING | set the connection string the SQL storage driver should use.                                               |
| $HELM_MAX_HISTORY                  | set the maximum number of helm release history.                                                            |
| $HELM_NAMESPACE                    | set the namespace used for the helm operations.                                                            |
| $HELM_NO_PLUGINS                   | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.                                                 |
| $HELM_PLUGINS                      | set the path to the plugins directory                                                                      |
| $HELM_REGISTRY_CONFIG              | set the path to the registry config file.                                                                  |
| $HELM_REPOSITORY_CACHE             | set the path to the repository cache directory                                                             |
| $HELM_REPOSITORY_CONFIG            | set the path to the repositories file.                                                                     |
| $KUBECONFIG                        | set an alternative Kubernetes configuration file (default "~/.kube/config")                                |
| $HELM_KUBEAPISERVER                | set the Kubernetes API Server Endpoint for authentication                                                  |
| $HELM_KUBECAFILE                   | set the Kubernetes certificate authority file.                                                             |
| $HELM_KUBEASGROUPS                 | set the Groups to use for impersonation using a comma-separated list.                                      |
| $HELM_KUBEASUSER                   | set the Username to impersonate for the operation.                                                         |
| $HELM_KUBECONTEXT                  | set the name of the kubeconfig context.                                                                    |
| $HELM_KUBETOKEN                    | set the Bearer KubeToken used for authentication.                                                          |
| $HELM_KUBEINSECURE_SKIP_TLS_VERIFY | indicate if the Kubernetes API server's certificate validation should be skipped (insecure)                |
| $HELM_KUBETLS_SERVER_NAME          | set the server name used to validate the Kubernetes API server certificate                                 |
| $HELM_BURST_LIMIT                  | set the default burst limit in the case the server contains many CRDs (default 100, -1 to disable)         |
| $HELM_QPS                          | set the Queries Per Second in cases where a high number of calls exceed the option for higher burst values |

Helm stores cache, configuration, and data based on the following configuration order:

- If a HELM_*_HOME environment variable is set, it will be used
- Otherwise, on systems supporting the XDG base directory specification, the XDG variables will be used
- When no other location is set a default location will be used based on the operating system

By default, the default directories depend on the Operating System. The defaults are listed below:

| Operating System | Cache Path                | Configuration Path             | Data Path               |
|------------------|---------------------------|--------------------------------|-------------------------|
| Linux            | $HOME/.cache/helm         | $HOME/.config/helm             | $HOME/.local/share/helm |
| macOS            | $HOME/Library/Caches/helm | $HOME/Library/Preferences/helm | $HOME/Library/helm      |
| Windows          | %TEMP%\helm               | %APPDATA%\helm                 | %APPDATA%\helm          |

Usage:
  helm [command]

Available Commands:
  completion  generate autocompletion scripts for the specified shell
  create      create a new chart with the given name
  dependency  manage a chart's dependencies
  env         helm client environment information
  get         download extended information of a named release
  help        Help about any command
  history     fetch release history
  install     install a chart
  lint        examine a chart for possible issues
  list        list releases
  package     package a chart directory into a chart archive
  plugin      install, list, or uninstall Helm plugins
  pull        download a chart from a repository and (optionally) unpack it in local directory
  push        push a chart to remote
  registry    login to or logout from a registry
  repo        add, list, remove, update, and index chart repositories
  rollback    roll back a release to a previous revision
  search      search for a keyword in charts
  show        show information of a chart
  status      display the status of the named release
  template    locally render templates
  test        run tests for a release
  uninstall   uninstall a release
  upgrade     upgrade a release
  verify      verify that a chart at the given path has been signed and is valid
  version     print the client version information

Flags:
      --burst-limit int                 client-side default throttling limit (default 100)
      --debug                           enable verbose output
  -h, --help                            help for helm
      --kube-apiserver string           the address and the port for the Kubernetes API server
      --kube-as-group stringArray       group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --kube-as-user string             username to impersonate for the operation
      --kube-ca-file string             the certificate authority file for the Kubernetes API server connection
      --kube-context string             name of the kubeconfig context to use
      --kube-insecure-skip-tls-verify   if true, the Kubernetes API server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kube-tls-server-name string     server name to use for Kubernetes API server certificate validation. If it is not provided, the hostname used to contact the server is used
      --kube-token string               bearer token used for authentication
      --kubeconfig string               path to the kubeconfig file
  -n, --namespace string                namespace scope for this request
      --qps float32                     queries per second used when communicating with the Kubernetes API, not including bursting
      --registry-config string          path to the registry config file (default "/home/<VMSSAdminUserwhencreatingSCaleset>/.config/helm/registry/config.json")
      --repository-cache string         path to the file containing cached repository indexes (default "/root/.cache/helm/repository")
      --repository-config string        path to the file containing repository names and URLs (default "/home/<VMSSAdminUserwhencreatingSCaleset>/.config/helm/repositories.yaml")

Use "helm [command] --help" for more information about a command.

That works, but it shouldn't be required. also you can see in the Flags section under registry-config, the wrong path is ALSO listed. How is it defaulting to that?

I have reviewed the ENV and nothing helm is set.

Platforms affected

• [X] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [X] Ubuntu 22.04
• [ ] macOS 11
• [ ] macOS 12
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [ ] macOS 14
• [ ] macOS 14 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

dev

Is it regression?

No

Expected behavior

Running helm should return the usage output.

Pipeline is running as AzDevOps.

Actual behavior

Permission is denied and requires additional elevation to root admin home directory.

Repro steps

Packer build is run and deployed to VMSS and managed via ADO integration,

Helm is run but does not succeed unless elevation due to wrong path.

[erik-bershel]

Hey @DevOpsAzurance!

We can recommend only to use sudo when invoking helm. It's not an image or build issue itself.