search

eksctl-io / eksctl

The official CLI for Amazon EKS
https://eksctl.io
Other
4.93k stars 1.41k forks source link

[Bug] Not authorized to access pricing API errors in Karpenter log #5672

Closed walkley closed 2 years ago

walkley commented 2 years ago

What were you trying to accomplish?

Create EKS cluster with Karpenter support.

What happened?

EKS cluster and Karpenter components were created, but have error messages in Karpenter controller log:

2022-09-05T12:16:18.813Z    ERROR   controller.aws.pricing  updating spot pricing, UnauthorizedOperation: You are not authorized to perform this operation.
    status code: 403, request id: 8f99fa14-175e-48a5-bfc9-747616e81ded, using existing pricing data from 2022-08-17T00:19:52Z   {"commit": "3d87474"}
...
2022-09-05T12:16:19.227Z    ERROR   controller.aws.pricing  updating on-demand pricing, AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
    status code: 400, request id: ca453ae3-e67e-4d3f-9c9a-a1b54958a6e6; AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
    status code: 400, request id: 4976ddb4-3943-4f34-ba76-d647e638d2f6, using existing pricing data from 2022-08-17T00:19:52Z   {"commit": "3d87474"}

How to reproduce it?

Create EKS cluster with following config file:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster-with-karpenter
  region: ap-northeast-1
  version: '1.22'
  tags:
    karpenter.sh/discovery: cluster-with-karpenter

iam:
  withOIDC: true

karpenter:
  version: '0.15.0'
  createServiceAccount: true # default is false

managedNodeGroups:
  - name: managed-ng-1
    minSize: 1
    maxSize: 2
    desiredCapacity: 1

Logs

Log of Karpenter controller:

2022-09-05T12:16:18.399Z    INFO    Successfully created the logger.
2022-09-05T12:16:18.399Z    INFO    Logging level set to: debug
{"level":"info","ts":1662380178.4083061,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
2022-09-05T12:16:18.408Z    INFO    controller  Initializing with version v0.15.0   {"commit": "3d87474"}
2022-09-05T12:16:18.445Z    DEBUG   controller.aws  Using AWS region ap-northeast-1 {"commit": "3d87474"}
2022-09-05T12:16:18.661Z    DEBUG   controller.aws  Discovered caBundle, length 1099    {"commit": "3d87474"}
2022-09-05T12:16:18.661Z    INFO    controller  loading config from karpenter/karpenter-global-settings {"commit": "3d87474"}
2022-09-05T12:16:18.661Z    INFO    controller.aws.pricing  Updating EC2 pricing information    {"commit": "3d87474"}
I0905 12:16:18.771350       1 leaderelection.go:243] attempting to acquire leader lease karpenter/karpenter-leader-election...
2022-09-05T12:16:18.771Z    INFO    controller  starting metrics server {"commit": "3d87474", "path": "/metrics"}
I0905 12:16:18.800252       1 leaderelection.go:253] successfully acquired lease karpenter/karpenter-leader-election
2022-09-05T12:16:18.813Z    ERROR   controller.aws.pricing  updating spot pricing, UnauthorizedOperation: You are not authorized to perform this operation.
    status code: 403, request id: 8f99fa14-175e-48a5-bfc9-747616e81ded, using existing pricing data from 2022-08-17T00:19:52Z   {"commit": "3d87474"}
2022-09-05T12:16:18.872Z    DEBUG   controller.aws.launchtemplate   Hydrating the launch template cache with tags matching "karpenter.k8s.aws/cluster: cluster-with-karpenter"  {"commit": "3d87474"}
2022-09-05T12:16:18.872Z    INFO    controller.controller.provisioning  Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z    INFO    controller.controller.provisioning  Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-09-05T12:16:18.872Z    INFO    controller.controller.provisioning  Starting workers    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 10}
2022-09-05T12:16:18.872Z    INFO    controller.controller.node-state    Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z    INFO    controller.controller.node-state    Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-09-05T12:16:18.872Z    INFO    controller.controller.pod-state Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z    INFO    controller.controller.pod-state Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-09-05T12:16:18.872Z    INFO    controller.controller.node  Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z    INFO    controller.controller.node  Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z    INFO    controller.controller.node  Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z    INFO    controller.controller.node  Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-09-05T12:16:18.873Z    INFO    controller.controller.termination   Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z    INFO    controller.controller.termination   Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-09-05T12:16:18.873Z    INFO    controller.controller.podmetrics    Starting EventSource    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z    INFO    controller.controller.podmetrics    Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-09-05T12:16:18.873Z    INFO    controller.controller.provisionermetrics    Starting EventSource    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z    INFO    controller.controller.provisionermetrics    Starting Controller {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-09-05T12:16:18.873Z    INFO    controller.controller.counter   Starting EventSource    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z    INFO    controller.controller.counter   Starting EventSource    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z    INFO    controller.controller.counter   Starting Controller {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-09-05T12:16:18.965Z    DEBUG   controller.aws.launchtemplate   Finished hydrating the launch template cache with 0 items   {"commit": "3d87474"}
2022-09-05T12:16:18.973Z    INFO    controller.controller.pod-state Starting workers    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 10}
2022-09-05T12:16:18.973Z    INFO    controller.controller.node-state    Starting workers    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-09-05T12:16:18.973Z    INFO    controller.controller.termination   Starting workers    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-09-05T12:16:18.973Z    INFO    controller.controller.podmetrics    Starting workers    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 1}
2022-09-05T12:16:19.073Z    INFO    controller.controller.provisionermetrics    Starting workers    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 1}
2022-09-05T12:16:19.074Z    INFO    controller.controller.node  Starting workers    {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-09-05T12:16:19.075Z    INFO    controller.controller.counter   Starting workers    {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 10}
2022-09-05T12:16:19.227Z    ERROR   controller.aws.pricing  updating on-demand pricing, AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
    status code: 400, request id: ca453ae3-e67e-4d3f-9c9a-a1b54958a6e6; AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
    status code: 400, request id: 4976ddb4-3943-4f34-ba76-d647e638d2f6, using existing pricing data from 2022-08-17T00:19:52Z   {"commit": "3d87474"}

Anything else we need to know?

The IAM policy for Karpenter lacks of some actions when compared to Karpenter CloudFormation: Karpenter IAM policy in eksctl: https://github.com/weaveworks/eksctl/blob/main/pkg/cfn/builder/karpenter.go#L112 Karpenter IAM policy in Karpenter Cloudformation: https://github.com/aws/karpenter/blob/main/website/content/en/v0.15.0/getting-started/getting-started-with-eksctl/cloudformation.yaml#L43

Versions

$ eksctl info
eksctl version: 0.110.0
kubectl version: v1.22.6-eks-7d68063
OS: linux
Himangini commented 2 years ago

@walkley Good spot. We'll add the fix soon 👍🏻

saireddyb commented 1 year ago

Please share show you resolved to solve this issue