Unable to use AWS IAM role for S3 backend storage

[davidg-sainsbury]

Expected behavior and actual behavior:

The registry pod via a Kubernetes ServiceAccount and IAM policy should be able to access an S3 bucket as backend storage.

Steps to reproduce the problem:

# Harbor
apiVersion: goharbor.io/v1beta1
kind: HarborCluster
metadata:
  name: harborcluster
  namespace: my-ns
spec:
  version: 2.3.0
  imageSource:
    repository: ghcr.io/goharbor
  registry:
    serviceAccountName: harbor-sa
storage:
    kind: S3
    spec:
      # Configure S3 as the backend storage of Harbor.
      # An implementation of the storagedriver.StorageDriver interface which uses Amazon S3 or S3 compatible services for object storage.
      # See: https://docs.docker.com/registry/storage-drivers/s3/
      s3:
        region: eu-west-1 # Required
        bucket: my-s3-bucket # Required

ServiceAccount snippet:

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1111111111111:role/harbor

IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/*",
                "arn:aws:s3:::my-bucket"
            ]
        }
    ]
} 

Versions: Please specify the versions of following systems.

• harbor operator version: 1.1.1
• harbor version: 2.3.0
• kubernetes version: AWS EKS 1.19

Additional context: registry operator errors when trying to create the deployment:

Reconciler error    {"reconciler group": "goharbor.io", "reconciler kind": "Registry", "name": "harborcluster-harbor-harbor", "namespace": " │
│ harbor-operator-ns", "error": "cannot set status to error: cannot set conditions to error: apply apps/v1, Kind=Deployment (harbor-operator-ns/harborcluster-harbor-harbor-registry): apply: Deployment.apps \"harborcluster-harbor- │
│ harbor-registry\" is invalid: spec.template.spec.containers[0].env[3].valueFrom.secretKeyRef.name: Invalid value: \"\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end │
│  with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'): apply apps/v1, Kind=Deployment (harbor-operator-ns/harborcluster-harbor-harb │
│ or-registry): apply: Deployment.apps \"harborcluster-harbor-harbor-registry\" is invalid: spec.template.spec.containers[0].env[3].valueFrom.secretKeyRef.name: Invalid value: \"\": a DNS-1123 subdomain must consist of lower case │

If I create a secret (key=secret) modify the harbor-cluster config to include the secretkeyref for that secret the registry pod starts but still fails to access the S3 bucket as it appears it's not using the IAM policy via SA.

s3:
      secretkeyRef: test-123
      region: eu-west-1 # Required
      bucket: my-bucket # Required

Refering to the implementation of the S3 https://docs.docker.com/registry/storage-drivers/s3/ then it should only be included if you are using the AWS secret credentials rather then role/policy. secretkey	no	Your AWS Secret Key. If you use IAM roles, omit to fetch temporary credentials from IAM.

In your CRD for harborcluster only bucket and region are required fields. required:

• bucket
• region

However, in this PR https://github.com/goharbor/harbor-operator/pull/266 the secretkeyref is always expected but if that is the case then we can't use IAM roles as per the implementation of the S3 https://docs.docker.com/registry/storage-drivers/s3/

[davidg-sainsbury]

Any update on this? Thanks.

[davidg-sainsbury]

Created PR - https://github.com/goharbor/harbor-operator/pull/811

[sagikazarmark]

Would be nice to get this released as v1.1.2

[sagikazarmark]

@davidg-sainsbury did you manage to run Harbor with IAM role on EKS?

I'm trying to do the same thing, but even with your fix, I believe I'm hitting this issue: https://github.com/distribution/distribution/issues/3275

Switching to an IAM account with the same policies works just fine, but I get access denied errors when no credentials are specified. Unfortunately, it's not really easy to determine what kind of access the registry is trying to use.

[davidg-sainsbury]

@davidg-sainsbury did you manage to run Harbor with IAM role on EKS?

I'm trying to do the same thing, but even with your fix, I believe I'm hitting this issue: distribution/distribution#3275

Switching to an IAM account with the same policies works just fine, but I get access denied errors when no credentials are specified. Unfortunately, it's not really easy to determine what kind of access the registry is trying to use.

We switched to the helm version of harbor rather than using the operator before I had chance to test this. However, we are are experiencing the same issue as you and are waiting for the fix in the distribution engine - https://github.com/goharbor/harbor/pull/16435. As per your link to the issue, until distribution is using the newer AWS SDK you won't be able to use the AWS IAM permissions to access S3 from the harbor registry pods.

[sagikazarmark]

Thanks for the confirmation @davidg-sainsbury !

[davidg-sainsbury]

Thanks for the confirmation @davidg-sainsbury !

@sagikazarmark - harbor registry still has an issue with access to S3 via IAM roles with the latest pre-release of Harbor - v2.5.0-rc1 https://github.com/goharbor/harbor/issues/16490. ChartMusuem appears to be fixed.

[William-Young-97]

Any further updates on this? Is STS : https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html perhaps a viabale workaround?

[alirezaghey]

Hey folks, I'd like to know if this already fixed or not? I'm having trouble s3 storage backends using IaM roles and IRSA, but I'm not sure if this a problem on my end or generally still not solved. Appreciate any feedback.

[stonezdj]

It is an issue in Harbor operator, not in Harbor AFAIK, the instance should be inside the AWS cloud. If the Harbor is installed in EC2 VM with docker-compose, you should grant the IAM role to the EC2 instance. For pods in EKS, there is another approach you can find in the AWS document. When you configure Harbor using s3 storage with IAM roles, you just remove the accesskey and secretkey in harbor.yml, then it uses the IAM role by default.

storage_service:
  s3:
    region: us-west-2
    regionendpoint:
    bucket: photon-harbor-registry
    encrypt: true
    secure: true
    v4auth: true
    chunksize: 5242880 
    multipartcopychunksize: 33554432
    multipartcopymaxconcurrency: 100
    multipartcopythresholdsize: 33554432
    rootdirectory: /harbor-ci-1

Harbor operator should remove this access key/secret key check

[chrisminton]

@stonezdj it is not resolved for IRSA in EKS, which is still unsupported by Harbor as of writing.

[krab-skunk]

Any chances to have IRSA support ? :( using 2.9.0

[vinaysi9838]

it issue occur when we use karpenter