[Bug]: The security token included in the request is invalid

[czduanqi2009]

Describe the expected outcome

Expected outcome should be like in the log of the aws-pca-issuer pod:

{"level":"info","ts":"2024-10-30T14:34:53Z","logger":"controllers.GenericIssuer","msg":"sts.GetCallerIdentity","genericissuer":{"name":"cismt-cluster-issuer"},"arn":"arn:aws:sts::677467515554:assumed-role/cismt-cchmt110hammer-apca-issuer-us-east-1/1730298893809131180","account":"677467515554","user_id":"AROAZ3PBV5KRAUHBILOR6:1730298893809131180"} {"level":"info","ts":"2024-10-30T14:34:53Z","logger":"controllers.GenericIssuer","msg":"Calling StoreProvisioner","genericissuer":{"name":"cismt-cluster-issuer"}} {"level":"info","ts":"2024-10-30T14:34:54Z","logger":"controllers.GenericIssuer","msg":"sts.GetCallerIdentity","genericissuer":{"name":"cismt-cluster-issuer"},"arn":"arn:aws:sts::677467515554:assumed-role/cismt-cchmt110hammer-apca-issuer-us-east-1/1730298893934620953","account":"677467515554","user_id":"AROAZ3PBV5KRAUHBILOR6:1730298893934620953"} {"level":"info","ts":"2024-10-30T14:34:54Z","logger":"controllers.GenericIssuer","msg":"Calling StoreProvisioner","genericissuer":{"name":"cismt-cluster-issuer"}}

Describe the actual outcome

{"level":"error","ts":"2024-10-30T19:10:46Z","msg":"Reconciler error","controller":"awspcaissuer","controllerGroup":"awspca.cert-manager.io","controllerKind":"AWSPCAIssuer","AWSPCAIssuer":{"name":"pca-issuer","namespace":"cismt-app-ext-devhammer"},"namespace":"cismt-app-ext-devhammer","name":"pca-issuer","reconcileID":"06bb206d-3bd8-48a4-87c9-b16caa3d67cc","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: b9650ce5-f22c-4ccd-878b-83933013c6d9, api error InvalidClientTokenId: The security token included in the request is invalid.","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:222"}

Steps to reproduce

1. Install the service aws-privateca-issuer

2. Create one secret as below: apiVersion: v1 kind: Secret metadata: name: aws-creds-for-ca-role namespace: cismt-app-ext-devhammer stringData: AWS_ACCESS_KEY_ID: awsAccessKeyId(not based64) AWS_SECRET_ACCESS_KEY: awsSecretAccessKey(not based64)

3. AWSPCAIssuer cluster-issuer.yaml apiVersion: awspca.cert-manager.io/v1beta1 kind: AWSPCAIssuer metadata: name: pca-issuer namespace: cismt-app-ext-devhammer spec: arn: @apacaArn@ region: @apcaRegion@ secretRef: namespace: cismt-app-ext-devhammer name: aws-creds-for-ca-role

4. Use the below command to create the pcaissuer: kubectl apply -f cluster-issuer.yaml

5. Check the log of the aws-privateca-issuer and found some errors: {"level":"error","ts":"2024-10-30T19:10:46Z","msg":"Reconciler error","controller":"awspcaissuer","controllerGroup":"awspca.cert-manager.io","controllerKind":"AWSPCAIssuer","AWSPCAIssuer":{"name":"pca-issuer","namespace":"cismt-app-ext-devhammer"},"namespace":"cismt-app-ext-devhammer","name":"pca-issuer","reconcileID":"06bb206d-3bd8-48a4-87c9-b16caa3d67cc","error":"operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: b9650ce5-f22c-4ccd-878b-83933013c6d9, api error InvalidClientTokenId: The security token included in the request is invalid.","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.2/pkg/internal/controller/controller.go:222"}

6. If I remove the below part, and no any errors. secretRef: namespace: cismt-app-ext-devhammer name: aws-creds-for-ca-role

Relevant log output

No response

Version

cert-manager: 1.8.2

Have you tried the following?

• [X] Check the Troubleshooting section
• [X] Search open issues

Category

Authentication Issue

Severity

Severity 1

[HanleyYin]

Hi, thanks for opening a bug report!

It looks like the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that you are passing into the Secret is not base64 encoded.

From the Troubleshooting section of the README,

Check the secret with the AWS credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values have to be base64 encoded.

For an example, you can take a look at https://github.com/cert-manager/aws-privateca-issuer/blob/main/config/samples/secret.yaml.

[czduanqi2009]

Hi @HanleyYin, thank you for your quick response. But I use the date type stringData instead of data. For the stringData, no need to encode.

[HanleyYin]

I believe that's true for Secrets in general; however, the aws-privateca-issuer requires that the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY be base64 encoded.

Do you have a special use case where you do not wish to encode your credentials to base64?

[czduanqi2009]

No special case. OK, let me have a try to use the encoding one. Will let you know the results soon.

[czduanqi2009]

I used the based64 in the secret but still hit the same error.

credentials=aws sts assume-role --role-arn ${CROSS_ACCOUNT_ROLE_ARN} --role-session-name assumeResourceRole log_info "credentials: ${credentials}" accessKeyId=echo $credentials | jq '.Credentials.AccessKeyId' | sed 's/\"//g' secretAccessKey=echo $credentials | jq '.Credentials.SecretAccessKey' | sed 's/\"//g' sessionToken=echo $credentials | jq '.Credentials.SessionToken' | sed 's/\"//g' log_info "accessKeyId: ${accessKeyId}" log_info "secretAccessKey: ${secretAccessKey}" log_info "sessionToken: ${sessionToken}"

In default namespace

# If the secret is existed, remove and recreate it
kubectl get secret --namespace default aws-creds-for-ca-role > /dev/null 2>&1
if [ $? -eq 0 ];then
    log_info "The K8S secret aws-creds-for-ca-role is existed and delete to recreate it."
    kubectl delete secret aws-creds-for-ca-role --namespace default
    if [ $? -eq 0 ];then
        log_info "The K8S secret aws-creds-for-ca-role is deleted successful."
    else
        log_error "Failed to delete the K8S secret aws-creds-for-ca-role."
        exit 1
    fi
fi

    accessKeyId1=$(echo -n "$accessKeyId" | base64)
secretAccessKey1=$(echo -n "$secretAccessKey" | base64)
#sessionToken1=$(echo -n "$sessionToken" | base64)

kubectl create secret generic aws-creds-for-ca-role \
    --from-literal=AWS_ACCESS_KEY_ID="${accessKeyId1}" \
    --from-literal=AWS_SECRET_ACCESS_KEY="${secretAccessKey1}"

[czduanqi2009]

cchmt110hammer@default.infor-sandbox-healthcare.us-east-1 /dma # kubectl describe secret aws-creds-for-ca-role Name: aws-creds-for-ca-role Namespace: default Labels: Annotations:

Type: Opaque

Data

AWS_ACCESS_KEY_ID: 28 bytes AWS_SECRET_ACCESS_KEY: 56 bytes