Bottlerocket: `credential-providers` not configured through userData

johanvandeweerd commented 9 months ago

Description

Observed Behavior:

Setting ecr-credential-provider in userData for Bottlerocket is not picked up when using the following configuration:

    [settings.host-containers.admin]
    enabled = true

    [settings.kubernetes]
    api-server = "https://***.eks.amazonaws.com"
    cluster-certificate = "***"
    cluster-name = "c2-k8s-development"

    [settings.kubernetes.node-labels]
    "ecr" = "true"

    [settings.aws]
    config = "W3Byb2ZpbGUgZWNyXQpyb2xlX2FybiA9IGFybjphd3M6aWFtOjo8QUNDT1VOVF9JRD46cm9sZS9lY3ItcmVhZC1vbmx5CmNyZWRlbnRpYWxfc291cmNlID0gRWMySW5zdGFuY2VNZXRhZGF0YQo="

    [settings.kubernetes.credential-providers.ecr-credential-provider]
    enabled = true
    image-patterns = ["*.dkr.ecr.*.amazonaws.com"]

    [settings.kubernetes.credential-providers.ecr-credential-provider.environment]
    AWS_PROFILE = "ecr"

New instances are spun up but when going into the EC2 instance and requesting the settings using apiclient, it returns the following (default?) output:

[root@admin]# apiclient get settings.kubernetes.credential-providers
{
  "settings": {
    "kubernetes": {
      "credential-providers": {
        "ecr-credential-provider": {
          "cache-duration": "12h",
          "enabled": true,
          "environment": null,
          "image-patterns": [
            "*.dkr.ecr.*.amazonaws.com",
            "*.dkr.ecr.*.amazonaws.com.cn",
            "*.dkr.ecr-fips.*.amazonaws.com",
            "*.dkr.ecr.us-iso-east-1.c2s.ic.gov",
            "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
          ]
        }
      }
    }
  }
}

Expected Behavior:

[root@admin]# apiclient get settings.kubernetes.credential-providers
{
  "settings": {
    "kubernetes": {
      "credential-providers": {
        "ecr-credential-provider": {
          "cache-duration": "12h",
          "enabled": true,
          "environment": {
            "AWS_PROFILE": "ecr"
          },
          "image-patterns": [
            "*.dkr.ecr.*.amaonaws.com"
          ]
        }
      }
    }
  }
}

This output was generated after running the following command:

[root@admin]# apiclient set --json '{"settings":{"kubernetes":{"credential-providers":{"ecr-credential-provider":{"enabled":true,"image-patterns":["*.dkr.ecr.*.amaonaws.com"],"environment":{"AWS_PROFILE":"ecr"}}}}}}'

Reproduction Steps (Please include YAML):

---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: ecr
spec:
  amiFamily: Bottlerocket
  subnetSelector:
    karpenter.sh/discovery: "c2-k8s-development"
  securityGroupSelector:
    karpenter.sh/discovery/k8s-development: "all,k8s-worker,c2-k8s-worker"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 4Gi
        volumeType: gp3
        encrypted: true
    - deviceName: /dev/xvdb
      ebs:
        volumeSize: 200Gi
        volumeType: gp3
        encrypted: true
  userData: |
    [settings.host-containers.admin]
    enabled = true

    [settings.kubernetes]
    api-server = "https://***.eks.amazonaws.com"
    cluster-certificate = "***"
    cluster-name = "c2-k8s-development"

    [settings.kubernetes.node-labels]
    "ecr" = "true"

    [settings.aws]
    config = "W3Byb2ZpbGUgZWNyXQpyb2xlX2FybiA9IGFybjphd3M6aWFtOjo8QUNDT1VOVF9JRD46cm9sZS9lY3ItcmVhZC1vbmx5CmNyZWRlbnRpYWxfc291cmNlID0gRWMySW5zdGFuY2VNZXRhZGF0YQo="

    [settings.kubernetes.credential-providers.ecr-credential-provider]
    enabled = true
    image-patterns = ["*.dkr.ecr.*.amazonaws.com"]

    [settings.kubernetes.credential-providers.ecr-credential-provider.environment]
    AWS_PROFILE = "ecr"
  tags:
    Name: ecr.c2-k8s-development.us-east-1
    Environment: development
    Role: Kubernetes Node
    Type: ecr
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1
    httpTokens: required
---
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: ecr
spec:
  providerRef:
    name: ecr
  taints:
    - key: dedicated
      value: ecr
      effect: NoSchedule
  requirements:
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values: ["c"]
    - key: karpenter.k8s.aws/instance-generation
      operator: In
      values: ["4", "5"]
    - key: karpenter.sh/capacity-type # If not included, the webhook for the AWS cloud provider will default to on-demand
      operator: In
      values: ["spot", "on-demand"]
    - key: "karpenter.k8s.aws/instance-cpu"
      operator: In
      values: ["8", "16"]
    - key: ecr
      operator: Exists
  limits:
    resources:
      cpu: "4500"
      memory: 9500Gi
  consolidation:
    enabled: true

Versions:

Chart Version: v0.31.2

Kubernetes Version (kubectl version):

Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4-eks-8cb36c9

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

wkd3475 commented 8 months ago

https://github.com/aws/karpenter-provider-aws/blob/main/pkg/providers/amifamily/bootstrap/bottlerocketsettings.go

The "credential-providers" are not defined in this structure, so that option is being ignored.

johanvandeweerd commented 8 months ago

Created PR https://github.com/aws/karpenter-provider-aws/pull/5530 that adds credential-providers to that structure.

aws / karpenter-provider-aws

Bottlerocket: `credential-providers` not configured through userData #5507

Description