helmv4: Authentication issue when pulling helm charts from private OCI registries.

jonnylangefeld commented 1 week ago

What happened?

helmv4 allows to download helm charts from private OCI registries by specifying username and password:

args := &helmv4.ChartArgs{
    Chart: pulumi.String("oci://myregistry.com/my-chart"),
    RepositoryOpts: helmv4.RepositoryOptsArgs{
        Username: pulumi.String("username"),
        Password: pulumi.String("password"),
    },
}

However, this results in authentication errors despite correct credentials. Different kinds for the different cloud provider registries:

on AWS using ECR

```bash error: an unhandled error occurred: waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials ```

on GCP using GCR

```bash error: an unhandled error occurred: waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: unexpected status from HEAD request to https://gcr.io/v2/gke-shared-dev/upstream/helm.cilium.io/tetragon/mainfests/1.1.2: 401 Unauthorized ```

We did some investigation and the following turns out to be the culprit:

When reproducing the issue with the CLI then this works:

helm registry login myregistry.com -u "<username>" -p "<token>"
helm pull oci://myregistry.com/my-chart

but this doesn’t, even though it should:

helm pull oci://myregistry.com/my-chart --username="<username>" --password="<token>"

The Pulumi helm v4 provider does the equivalent to the latter.

This is recognized as an open upstream helm issue: https://github.com/helm/helm/pull/12769

The main issue is the inconsistency. helm pull with username and password without previous helm registry login works for https:// helm registries, but not for oci://.

If this is an upstream helm issue, why is this posted on the pulumi-kubernetes repo?

The main reason I'm posting this here is that the helmv4 Pulumi provider copy/pastes some private upstream functions. For instance the newRegistryClient() function that the upstream helm bugfix PR is attempting to fix here by adding the username and password to the signature, is copy/pasted into this repo because it's a private function: https://github.com/pulumi/pulumi-kubernetes/blob/f5f368dad365317f265f484b2d61a9180ee352f6/provider/pkg/helm/tool.go#L341

This means that even if the upstream helm PR is merged, the helmv4 Pulumi provider would still suffer from this issue. So it has to be fixed here as well. But we might still need to wait for the upstream fix, for the new upstream NewClient() function.

The upstream helm PR has been open for 7 months with most recent activity 1 month ago. We could help push it along.

Output of `pulumi about`

╰─ pulumi about
CLI
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Host
OS       darwin
Version  14.6.1
Arch     arm64

Backend
Name           pulumi.com
URL            https://app.pulumi.com/jonny-langefeld
User           jonny-langefeld
Organizations  jonny-langefeld, snowflake
Token type     personal

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

EronWright commented 1 week ago

I believe it is a known limitation that Helm OCI support relies on ambient credentials, as set by helm registry login or by docker login. The upstream PR seeks to allow for adhoc credentials via flags, and Pulumi would assumedly incorporate that by wiring up the RepositoryOpts. At this time, the RepositoryOpts is applicable only to HTTP-based chart repositories.

blampe commented 1 week ago

Strictly speaking I don't think we would need to wait for the fix upstream, we could probably plumb the credentials through with what we already have.

stepan-romankov commented 2 days ago

Is it any walk around for this issue @blampe @EronWright? I'm stuck with installation of AWS Gateway AIP Controller chart as it is located only in OCI repository.

pulumi / pulumi-kubernetes