[Bug]: Insufficient Permissions for ProviderConfig Listing in Crossplane AWS Provider v1.6.0 with EKS

edgarsilva948 commented 4 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Affected Resource(s)

elbv2.aws.upbound.io/v1beta1 - LB elbv2.aws.upbound.io/v1beta1 - LBTargetGroup elbv2.aws.upbound.io/v1beta1 - LBTargetGroupAttachment elbv2.aws.upbound.io/v1beta1 - LBListener

Resource MRs required to reproduce the bug

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: ${name}-runtime-config
spec:
  deploymentTemplate:
    spec:
      replicas: 1
      selector: {}
      template:
        spec:
          containers:
            - name: package-runtime
              args:
                - --debug
              env:
                - name: http_proxy
                  value: "http://proxy-server:80"
                - name: https_proxy
                  value: "http://proxy-server:443"
                - name: HTTP_PROXY
                  value: "http://proxy-server:80"
                - name: HTTPS_PROXY
                  value: "http://proxy-server:443"
                - name: no_proxy
                  value: "10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, .svc, .cluster.local"
                - name: NO_PROXY
                  value: "10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, .svc, .cluster.local"
          securityContext:
            fsGroup: 2000
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: ${provider_role_arn}

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-${name}-${family}
spec:
  package: ${ecr_account_id}.dkr.ecr.${aws_region}.amazonaws.com/${repo_path}/provider-aws-${family}:${version}
  skipDependencyResolution: true
  package: xpkg.upbound.io/upbound/provider-aws-${family}:${version}
  runtimeConfigRef:
    name: ${name}-runtime-config

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-${name}
spec:
  package: ${ecr-account-id}.dkr.ecr.${aws-region}.amazonaws.com/upbound/provider-family-aws:${version}
  package: xpkg.upbound.io/upbound/provider-family-aws:${version}
  runtimeConfigRef:
    name: ${name}-runtime-config

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: provider-${name}
spec:
  credentials:
    source: WebIdentity
    webIdentity:
      roleARN: ${provider_role_arn}
  assumeRoleChain:
    - roleARN: "arn:aws:iam::${account_id}:role/${role_name}"

Steps to Reproduce

Deploy Crossplane v1.16 on EKS.
Install the AWS provider v1.6.0 using the provided manifests.
Create the DeploymentRuntimeConfig with the provided configuration.
Create the family Provider with the provided configuration.
Create the elbv2 Provider with the provided configuration.
Create the elbv2 ProviderConfig with the provided configuration.

What happened?

Actual Behavior

The elbv2 provider deployment succeeds, but attempts to create any resource using the provider result in failure, accompanied by permission errors in the logs. The service account used by the provider does not have sufficient permissions to list ProviderConfig resources, preventing resource creation.

Expected Behavior

The AWS provider's service account should have sufficient permissions to list ProviderConfig resources in the aws.upbound.io API group. This would allow the provider to create resources without encountering errors.

Relevant Error Output Snippet

W0715 14:09:50.615204       1 reflector.go:539] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1beta1.ProviderConfig: providerconfigs.aws.upbound.io is forbidden:User "system:serviceaccount:crossplane-system:provider-upjet-aws-elbv2-99de9eeb558d" cannot list resource "providerconfigs" in API group "aws.upbound.io" at the cluster scope
E0715 14:09:50.615332       1 reflector.go:147] k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1beta1.ProviderConfig: failed to list *v1beta1.ProviderConfig: providerconfigs.aws.upbound.io is forbidden: User "system:serviceaccount:crossplane-system:provider-upjet-aws-elbv2-99de9eeb558d" cannot list resource "providerconfigs" in API group "aws.upbound.io" at the cluster scope

Crossplane Version

v1.16.0

Provider Version

v1.6.0

Kubernetes Version

Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.6-eks-db838b0

Kubernetes Distribution

EKS

Additional Info

Providers such as IAM, EKS, and EC2 are working fine and do not encounter this error.
However, other providers, including WAF, Api Gateway, ELBv2, and CloudFront, are not working and exhibit the same permission error.

If I manually add cluster-admin permission to the service account, the providers work perfectly.

kubectl create clusterrolebinding elbv2-full-access --clusterrole=cluster-admin --serviceaccount=crossplane-system:provider-upjet-aws-elbv2-99de9eeb558d

haarchri commented 4 months ago

Can you show kubectl get pkgrev ?

the family provider needs to be installed from the same registry then the providers - otherwise the RBAC Manager will not add the permissions

haarchri commented 4 months ago

and it looks like you have multiple times spec.package wich override your first

edgarsilva948 commented 4 months ago

Can you please clarify why it is necessary to have the same registry for all the image providers? Is this a bug or an expected behavior?

Provider Revisions


$ kubectl get pkgrev
providerrevision.pkg.crossplane.io/provider-upjet-aws-apigateway-522f26dcf869       True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-apigateway:v1.6.0           Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-cloudfront-6ade770a3698       True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-cloudfront:v1.6.0           Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-cloudwatch-a574cf0b7903       True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-cloudwatch:v1.6.0           Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-cloudwatchlogs-4f0b0596620a   True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-cloudwatchlogs:v1.6.0       Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-dynamodb-30524fc5a5bd         True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-dynamodb:v1.6.0             Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-ec2-9992fffc7b00              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-ec2:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-eks-287ac2697d84              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-eks:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-elasticache-021e92bd0c1b      True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-elasticache:v1.6.0          Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-elbv2-99de9eeb558d            True      1          222222222222.dkr.ecr.REGION.amazonaws.com/upbound/provider-aws-elbv2:v1.6.0                          Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-fc9896a8d96a                  True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-family-aws:v1.6.0               Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-iam-b29203ad115c              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-iam:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-kms-8713a070abe3              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-kms:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-lambda-7f8fe1d557bf           True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-lambda:v1.6.0               Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-mq-9e573a8c1f29               True      1          222222222222.dkr.ecr.REGION.amazonaws.com/upbound/provider-aws-mq:v1.6.0                             Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-rds-7a0b2a843a4b              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-rds:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-s3-6cbd0623c595               True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-s3:v1.6.0                   Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-sns-51f29cdcc163              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-sns:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-sqs-04017876a5e0              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-sqs:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-vpc-b43bc957fed5              True      1          111111111111.dkr.ecr.REGION.amazonaws.com/REPO-PATH/upbound/provider-aws-vpc:v1.6.0                  Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-waf-797914a35bea              True      1          222222222222.dkr.ecr.REGION.amazonaws.com/upbound/provider-aws-waf:v1.6.0                            Active                               6d20h
providerrevision.pkg.crossplane.io/provider-upjet-aws-wafregional-7bf10629d9c1      True      1          222222222222.dkr.ecr.REGION.amazonaws.com/upbound/provider-aws-wafregional:v1.6.0                    Active                               6d20h

haarchri commented 4 months ago

yes the RBAC Manager expecting that the Providers from the Same org / registry to grant permissions

edgarsilva948 commented 4 months ago

is there any documentation or references that explain this behavior of the RBAC Manager expecting providers to be from the same organization/registry to grant permissions? It would be helpful to understand this requirement in more detail.

haarchri commented 4 months ago

https://github.com/crossplane/crossplane/blob/master/internal/controller/rbac/provider/roles/reconciler.go#L300-L307

crossplane-contrib / provider-upjet-aws