Allow to add additional credential-providers

[mcanevet]

What I'd like: It looks like the only supported credential-providers is ecr-credential-provider.

IIUC it would technically be possible to support additional credential-providers if we were able to deploy a credentials provider plugin in the image-credential-provider-bin-dir (/x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/kubernetes/kubelet/plugins?).

Then we would just have to declare it in settings.kubernetes.credential-providers.

Any alternatives you've considered: We currently propagate an image pull secret and patch the pods using Kyverno policies instead.

[vyaghras]

Hi @mcanevet Can you share which credential provider you are looking for?

[mcanevet]

@vyaghras for example I'd like my users to be able to download images from Gitlab container registry or JFrog with having to deploy a Secret and specify ImagePullSecret. I could write the plugin, but the problem is that AFAIK there's no way to deploy it right now as the filesystem is read-only.

[yeazelm]

I spent a bit of time reading about credential providers in k8s and https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/#installing-plugins-on-nodes is the point that makes this problematic on Bottlerocket. We try to make sure every executable on the system is backed by the read only filesystem, so our SELinux policy attempts to restrict executables that are backed by the read only root filesystem.

I think our first choice would be to add the credential provider directly to Bottlerocket and make it available to be configured by settings. If there is a credential provider that is well supported, we might consider adding it.

For a custom credential provider, we don't have a great way to do this right now in Bottlerocket but you could build your own variant with it included. I admit that is probably too much work for the outcome you are looking for though.

From your comment @mcanevet, it sounds like there isn't currently a credential provider and you are thinking about writing one?

[mcanevet]

According to this documentation, it should be quite easy to create a credential provider. The question is how to do it in a non-opinionated way so that it can be embedded in BottleRocket... For example, as we are using static credentials for Gitlab, we could expose it to the credential provider through environment variables that way:

[settings.kubernetes.credential-providers.gitlab-credential-provider]
enabled = true
image-patterns = [
  "*.gitlab.com"
]

[settings.kubernetes.credential-providers.gitlab-credential-provider.environment]
"USERNAME" = "my-user"
"PASSWORD" = "my-password"

And we would just need a credential provider that outputs this to stdout:

{
  "apiVersion": "kubelet.k8s.io/v1",
  "kind": "CredentialProviderResponse",
  "auth": {
    "cacheDuration": "6h",
    "gitlab.com/my-app": {
      "username": "$USERNAME",
      "password": "$PASSWORD"
    }
  }
}

When it receives this kind of payload on stdin:

{
  "apiVersion": "kubelet.k8s.io/v1",
  "kind": "CredentialProviderRequest",
  "image": "gitlab.com/my-app"
}

This binary (or shell script if BottleRocket allows it) would be very easy to write, but this is a very opinionated (an probably insecure) way to pass credentials. One could prefer store credentials in SecretsManager with a rotation function, and allow the credential provider to retrieve it. Hence, the environment variables could be the ARN of the secret. Again, this credential provider would be very easy to write, but again it is an opinionated way to do it...

[mcanevet]

Actually we could have a generic aws-secretsmanager-credential-provider that would take the secret ARN as environment variable, we'd just need to be able to specify the binary location to that we could call it multiple times. Something like this would do the trick:

[settings.kubernetes.credential-providers.gitlab-credential-provider]
enabled = true
binary = aws-secretsmanager-credential-provider
image-patterns = [
  "*.gitlab.com"
]

[settings.kubernetes.credential-providers.gitlab-credential-provider.environment]
"AWS_SECRET_ARN" = "my-gitlab-secret-arn"

[settings.kubernetes.credential-providers.jfrog-credential-provider]
enabled = true
binary = aws-secretsmanager-credential-provider
image-patterns = [
  "*.jfrog.com"
]

[settings.kubernetes.credential-providers.jfrog-credential-provider.environment]
"AWS_SECRET_ARN" = "my-jfrog-secret-arn"

Do you think it would be possible to add this feature to BottleRocket? I think it's not too opinionated and not so hard to maintain. 2 things are missing right now:

• the possibility to override the binary in the credential provider settings
• the aws-secretsmanager-credential-provider binary (which should not be so hard to write)

[yeazelm]

Hey @mcanevet, this is a great idea! I think that if there was an AWS Secrets Manager credential provider that worked as you describe, we would strongly consider adding it into Bottlerocket.

[mcanevet]

@yeazelm we would still need to be able to pass the binary location in order to be able to instantiate it multiple times. I'll try to create an AWS Secrets Manager credentials provider

[mcanevet]

This very simple (AI generated) shell script would do the trick (it would maybe be better to write it in another language though to avoid depending on aws-cli and jq):

#!/bin/bash

# Ensure AWS_SECRET_ARN is set
if [[ -z "$AWS_SECRET_ARN" ]]; then
  echo "Error: AWS_SECRET_ARN environment variable is not set."
  exit 1
fi

# Read the input JSON from stdin
read -r input_json

# Extract the image name using jq
image=$(echo "$input_json" | jq -r '.image')

# Fetch secret from AWS Secrets Manager
secret_json=$(aws secretsmanager get-secret-value --secret-id "$AWS_SECRET_ARN" --query 'SecretString' --output text)

# Extract the username and password from the secret
USERNAME=$(echo "$secret_json" | jq -r '.username')
PASSWORD=$(echo "$secret_json" | jq -r '.password')

# Generate the output JSON
output_json=$(jq -n \
  --arg apiVersion "kubelet.k8s.io/v1" \
  --arg kind "CredentialProviderResponse" \
  --arg cacheDuration "6h" \
  --arg image "$image" \
  --arg username "$USERNAME" \
  --arg password "$PASSWORD" \
  '{
    apiVersion: $apiVersion,
    kind: $kind,
    auth: {
      cacheDuration: $cacheDuration,
      ($image): {
        username: $username,
        password: $password
      }
    }
  }')

# Output the result to stdout
echo "$output_json"

But we'd still need to be able to specify the binary path in settings.kubernetes.credential-providers.* in order to be able to use it multiple times.

[mcanevet]

Here is a version in go:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "os"
    "strings"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/secretsmanager"
)

// Request and Response structures
type CredentialProviderRequest struct {
    APIVersion string `json:"apiVersion"`
    Kind       string `json:"kind"`
    Image      string `json:"image"`
}

type Auth struct {
    Username string `json:"username"`
    Password string `json:"password"`
}

type CredentialProviderResponse struct {
    APIVersion string            `json:"apiVersion"`
    Kind       string            `json:"kind"`
    Auth       map[string]Auth   `json:"auth"`
    CacheDuration string          `json:"cacheDuration"`
}

func main() {
    // Read the secret ARN from the environment
    secretARN := os.Getenv("AWS_SECRET_ARN")
    if secretARN == "" {
        log.Fatal("Error: AWS_SECRET_ARN environment variable is not set.")
    }

    // Read input JSON from stdin
    var request CredentialProviderRequest
    err := json.NewDecoder(os.Stdin).Decode(&request)
    if err != nil {
        log.Fatalf("Failed to decode input JSON: %v", err)
    }

    // Fetch the secret from AWS Secrets Manager
    secret, err := getSecret(secretARN)
    if err != nil {
        log.Fatalf("Failed to retrieve secret: %v", err)
    }

    // Unmarshal the secret string into a map
    var secretMap map[string]string
    err = json.Unmarshal([]byte(secret), &secretMap)
    if err != nil {
        log.Fatalf("Failed to parse secret JSON: %v", err)
    }

    username := secretMap["username"]
    password := secretMap["password"]

    // Construct the response
    response := CredentialProviderResponse{
        APIVersion:   "kubelet.k8s.io/v1",
        Kind:         "CredentialProviderResponse",
        CacheDuration: "6h",
        Auth: map[string]Auth{
            request.Image: {
                Username: username,
                Password: password,
            },
        },
    }

    // Output the response as JSON to stdout
    err = json.NewEncoder(os.Stdout).Encode(response)
    if err != nil {
        log.Fatalf("Failed to encode output JSON: %v", err)
    }
}

// getSecret fetches the secret from AWS Secrets Manager
func getSecret(secretARN string) (string, error) {
    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("us-west-2"),
    })
    if err != nil {
        return "", fmt.Errorf("failed to create session: %w", err)
    }

    svc := secretsmanager.New(sess)
    input := &secretsmanager.GetSecretValueInput{
        SecretId: aws.String(secretARN),
    }

    result, err := svc.GetSecretValue(input)
    if err != nil {
        return "", fmt.Errorf("failed to get secret: %w", err)
    }

    // Return the SecretString
    if result.SecretString != nil {
        return *result.SecretString, nil
    }

    return "", fmt.Errorf("secret string is nil")
}

I could provide a separated project on Github with that code, but maybe it would be better to integrate it in BottleRocket?

[laserpedro]

Hello @mcanevet thank you for opening this since currently that would be a big plus for us when using a different registry than ECR. Could you provide an example of how the configuration of the node would look like for a custom registry mirroring docker.io using a sm to store the username / pwd ? It is not clear to me how this could be generic with binaries that may need to be specific to registries ? Tks !

[mcanevet]

Here is what I'm thinking about:

If we have this configuration in the userdata:

[settings.kubernetes.credential-providers.aws-secretsmanager-credential-provider]
enabled = true
image-patterns = [
  "*.docker.io"
]

[settings.kubernetes.credential-providers.aws-secretsmanager-credential-provider.environment]
"AWS_SECRET_ARN" = "my-dockerhub-secret-arn"

and a binary /x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/kubernetes/kubelet/plugins/aws-secretsmanager-credential-provider that returns this to stdout:

{
  "apiVersion": "kubelet.k8s.io/v1",
  "kind": "CredentialProviderResponse",
  "auth": {
    "cacheDuration": "6h",
    "gitlab.com/my-app": {
      "username": "$USERNAME",
      "password": "$PASSWORD"
    }
  }
}

when receiving this payload as stdin (which the Kubelet Credentials Provider will do according to this doc):

{
  "apiVersion": "kubelet.k8s.io/v1",
  "kind": "CredentialProviderRequest",
  "image": "docker.io/my-app"
}

It should work.

But the thing is that we'd have to deploy one binary per registry while this process is completely generic. It just take the ARN of the secret in environment variable. So we could easily generalize it to any registry that uses username/password as credentials. The only thing missing from my POV is a way to override the binary to use in the settings.kubernetes.credential-providers section in order to be able to use the aws-secretsmanager-credential-provider multiple time with different secrets.

[bcressey]

@mcanevet this is a clever piece of design (and implementation!) work.

For the remaining difficulty, I don't have an equally clever idea.

One very easy approach would be to add a fixed number of symlinks to the binary in the packaging, like:

aws-secretsmanager-credential-provider-slot001

And then each config could reference a slot number and end up invoking the same binary.

The downside is the use of opaque slot numbers vs. something more descriptive. It'd be functional but ugly.

If that's too objectionable then we could add an API specifically to set up cred provider aliases, but then there's more implementation work to make it work and we'd give up the immutability of the cred provider directory.