How to configure credentials for S3 repository?

[miguelaferreira]

I'm trying to deploy a helm_release that uses and S3 bucket as the repository. There is no public access to that S3 bucket. I'm able to do helm install/upgrade locally and for that I need to provide AWS credentials to the helm command, for it to download the repo index (index.yaml) and the required chart.

However, when I do that via the terraform helm provider I keep getting Access Denied. I have both an AWS and a Helm provider declared. The AWS provider has the required access to the S3 bucket, but the helm provider does not.

This is the debug log from the terraform execution.

2024-01-16T15:46:27.363+0100 [WARN]  unexpected data: registry.terraform.io/hashicorp/helm:stderr="Error: fetch from s3 url=s3://my-bucket/stable/index.yaml: fetch object from s3: AccessDenied: Access Denied
        status code: 403, request id: TEXGRX3GTAM7X25J, host id: ZKnoOxLAojLjYY3tJ8/8sR4mMHoR2GakO/wawTSJM/lA8xMXyKh4GONazXPD8DNWdLv6IT+RQpU="
2024-01-16T15:46:27.366+0100 [ERROR] provider.terraform-provider-helm_v2.12.1_x5: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="looks like "s3://my-bucket/stable" is not a valid chart repository or cannot be reached: plugin "bin/helm-s3 download" exited with error" tf_proto_version=5.4 tf_provider_addr=provider diagnostic_detail= tf_req_id=abd6ff83-54cc-6719-88c7-298ecc3a5380 tf_resource_type=helm_release tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto timestamp=2024-01-16T15:46:27.365+0100
2024-01-16T15:46:27.375+0100 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-01-16T15:46:27.375+0100 [ERROR] vertex "helm_release.my_release" error: looks like "s3://my-bucket/stable" is not a valid chart repository or cannot be reached: plugin "bin/helm-s3 download" exited with error

Terraform version, Kubernetes provider version and Kubernetes version

Terraform v1.5.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v5.31.0
+ provider registry.terraform.io/hashicorp/helm v2.12.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.24.0

Terraform configuration

provider "aws" {
  region = "some region"
  assume_role {
    role_arn = "some role arn"
  }
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args = [
      "eks",
      "--region", var.aws_region,
      "get-token", "--role-arn", "some role",
      "--cluster-name", module.eks_cluster.cluster_name
    ]
  }
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks_cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority[0].data)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args = [
        "eks",
        "--region", var.aws_region,
        "get-token", "--role-arn", "some role",
        "--cluster-name", module.eks_cluster.cluster_name
      ]
    }
  }
}

resource "helm_release" "my_release" {
  repository = "s3://my-bucket/stable"
  chart      = "my-chart"
  name       = "my-release"
  namespace  = "my-namespace"
  version    = "0.1.0"
}

Question

Is there a way to manipulate the helm provider environment and inject the required AWS credentials?

[arybolovlev]

Hi @miguelaferreira,

Since support for S3 buckets is implemented in a plugin, I think you need to refer to the S3 plugin documentation.

I hope that helps. Thank you!

[miguelaferreira]

Thanks for getting back to me @arybolovlev.

I get the principle of what I need to do. It's just that when my AWS provider takes the credentials I give to the terraform process, and then assumes a role (on another account), that role is the one that has access to the S3 bucket where the helm repo is. However, the helm provider will have those underlying credentials I gave to the terraform process, and those don't have access to the S3 bucket where the helm repo is.

Ideally there would be a way to configure the AWS role to be assumed in the helm provider. As a workaround I had to setup cross-account policies to allow the underlying credentials to access the S3 bucket, and that breaks the permission segregation that I had before.

[Foxite]

Did you manage to solve this issue?

[miguelaferreira]

Yes I did work around it. What I did was to allow the user with the credentials to access the S3 bucket. My AWS provider assumes a role with those credentials, and the helm s3 plugin uses the credentials directly.

Ideally the helm s3 will plugin would also assume the role first, but like I describe in the issue it can't.