aws / karpenter-provider-aws

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
https://karpenter.sh
Apache License 2.0
6.68k stars 934 forks source link

Could not login to OCI registry "public.ecr.aws": login attempt to https://public.ecr.aws/v2/ failed with status: 403 Forbidden #6357

Closed RobinFrcd closed 3 months ago

RobinFrcd commented 3 months ago

Description

Hi, I've been managing Karpenter with the helm chart + terraform for a while now, but I've had login issues since yesterday.

Error: could not login to OCI registry "public.ecr.aws": login attempt to https://public.ecr.aws/v2/ failed with status: 403 Forbidden

The setup is really simple:

provider "aws"{
  alias = "use1"
  region = "us-east-1"
}

data "aws_ecrpublic_authorization_token" "token" {
    # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecrpublic_authorization_token
    provider = aws.use1
}

resource "helm_release" "karpenter" {
  namespace           = "kube-system"

  name                = "karpenter"
  repository          = "oci://public.ecr.aws/karpenter"
  repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart               = "karpenter"
  # https://github.com/aws/karpenter-provider-aws/releases
  version             = "0.37.0"
  wait                = false
}

I have already tried to do docker logout public.ecr.aws or helm registry logout public.ecr.aws.

Is there any other source to pull the chart from ? This ecr repo is a bit annoying. I see I'm not the only one having this issue (https://github.com/aws/karpenter-provider-aws/issues/3441#issuecomment-2106370162).

Thanks !

jmdeal commented 3 months ago

I'm going to guess you can't provide an authorization token to ECR public, could you try removing these lines?

data "aws_ecrpublic_authorization_token" "token" {
    # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecrpublic_authorization_token
    provider = aws.use1
}

The public ECR is currently the only official release source for the helm chart.

RobinFrcd commented 3 months ago

Thanks for your answer, took it from the official examples, seems there's an issue there then ! https://github.com/terraform-aws-modules/terraform-aws-eks/blob/323fb759d718ed1f295c7278ef89a2e2f060d4a1/examples/karpenter/main.tf#L147

Thanks again !

DRBeauchamp commented 2 months ago

Why is this closed? I am hitting the same issue. What is the solution?

RobinFrcd commented 2 months ago

Remove repository_username and repository_password for public repositories.

semusings commented 2 months ago

Just tested today by installing Karpenter. As mentioned by @RobinFrcd, it works fine after removing repository_username and repository_password for public repositories.

Additionally, we can safely remove data "aws_ecrpublic_authorization_token" if we don't use private ECR repositories.

Harshithraj24 commented 1 month ago

docker logout public.ecr.aws

This works

Reference : https://gallery.ecr.aws/karpenter/karpenter

DingGGu commented 1 week ago

Logout via helm cli works fine.

$ helm registry logout public.ecr.aws

Check below commands for works well.


$ helm fetch oci://public.ecr.aws/karpenter/karpenter-crd