Helm release with AWS ECR private repository

ISPHOST commented 1 year ago

Terraform, Provider, Kubernetes and Helm Versions

Terraform version:
Terraform v1.4.2
on windows_amd64
+ provider registry.terraform.io/gavinbunney/kubectl v1.14.0
+ provider registry.terraform.io/hashicorp/aws v4.67.0
+ provider registry.terraform.io/hashicorp/helm v2.9.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.20.0
+ provider registry.terraform.io/hashicorp/tls v4.0.4

Kubernetes version:
1.24

Affected Resource(s)

helm_release

Terraform Configuration Files


provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      # This requires the awscli to be installed locally where Terraform is executed
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
  registry {
    url      = "oci://ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com"
    password = data.aws_ecr_authorization_token.token.password
    username = data.aws_ecr_authorization_token.token.user_name
  }
}

resource "helm_release" "mychart" {
  name                = "mychart"
  chart               = "oci://ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com/my-chart"
  version             = 0.1
  wait                = false
}

Debug Output

╷ │ Error: could not login to OCI registry "ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com": login attempt to https://ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com/v2/ failed with status: 400 Bad Request │ │ with provider["registry.terraform.io/hashicorp/helm"], │ on main.tf line 45, in provider "helm": │ 45: provider "helm" { │ ╵ 2023-05-17T15:19:17.930+0300 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info 2023-05-17T15:19:17.931+0300 [TRACE] statemgr.Filesystem: unlocked by closing terraform.tfstate 2023-05-17T15:19:17.934+0300 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF" 2023-05-17T15:19:17.954+0300 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/helm/2.9.0/windows_amd64/terraform-provider-helm_v2.9.0_x5.exe pid=7092 2023-05-17T15:19:17.969+0300 [DEBUG] provider: plugin exited

Steps to Reproduce

terraform apply

Expected Behavior

Helm chart should be installed from AWS ECR private repo

Actual Behavior

Helm provider not able to get Helm chart from AWS ECR private repo

Important Factoids

References

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

steverukuts commented 1 year ago

I encountered a very similar problem today and was able to solve it. The actual error message is obscured, even at the highest level of TF_LOG, but after attaching a debugger and with a mild bit of hacking, I was able to see the message:

{"errors":[{"code":"DENIED","message":"Your Authorization Token is invalid."}]}

I was able to get the same error by calling docker login <ecr url> -u AWS -p <password>. I compared the password that Terraform wanted to use with the output of aws ecr get-login-password and noticed they looked very different. I then noticed an error in my configuration. I would expect a 401 or 403 error in the case of an invalid password but it would seem that AWS returns a 400 instead.

My mistake was to use data.aws_ecr_authorization_token.token.authorization_token, which is not what the helm provider wants. I hope this is useful to someone.

In your specific case, you have specified data.aws_ecr_authorization_token.token.authorization_token, which is correct, however you might want to take a look at the token you are receiving. As ECR apparently doesn't obey normal status codes, it is possible that your security principal does not have permission to access ECR, or there is some other kind of problem.

My debugging idea for you is to use local_file to write out the username, password and server, and then try to use docker login with those credentials against the ECR server you want to access, and see what the output is. If this works then you have a different problem.

emoreth commented 1 year ago

I battled with the same error for a long time but I did not share @steverukuts issue of "code":"DENIED".

My issue was that my EKS cluster was in one region and my ECR was in another region, so I needed to use

data "aws_ecr_authorization_token" "token" {
  provider = aws.us-west-2
}

yasinzaehringer-paradime commented 5 months ago

I just saw a similar error - for me it came down to this: I only set repository_username and repository_password on helm_release and (then suddenly?!) I got this error. My solution here for me was: I added the username and password (+ url) on the provider via registry. Then it worked again.

joaquin386 commented 2 weeks ago

I battled with the same error for a long time but I did not share @steverukuts issue of "code":"DENIED".

My issue was that my EKS cluster was in one region and my ECR was in another region, so I needed to use
data "aws_ecr_authorization_token" "token" {
  provider = aws.us-west-2
}

This was my issue exactly. After addign the provider the error 400 is gone now.

hashicorp / terraform-provider-helm