Unable to get s3 credentials from aws role using credential_process with web identity in gce VM instance

[bvs-amagi]

Describe the bug

Role was created with web identity of google

1. Credentials obtained with aws role through aws cli -- validated role
2. Credentials obtained with aws access key and secret in cpp unit test app -- validated test app
3. Unable to get credentials from aws role in cpp unit test app

Please check why credentials are not being extracted from role for aws cpp sdk

Expected Behavior

Credentials should be extracted using output from file pointed by credential_process

Current Behavior

Credentials are not extracted using output from file pointed by credential_process. Resulting in empty credentials and 403 error

Reproduction Steps

export AWS_ROLE=arn:aws:iam::xxxxxxxxxx:role/vm_s3_readonly credential_process = _credentials script using AWSROLE Below is the credentials format obtained by running file pointed by credential_process { "Version": 1, "AccessKeyId": "xxxxxxxxx", "SecretAccessKey": "yyyyyyyyyyy", "SessionToken": "zzzzzzzzzzz" "Expiration": "2024-07-11T17:00:32+00:00" } Verified aws s3 cp which worked properly with the same aws role But running following cpp did not provide any credentials

Aws::Auth::ProfileConfigFileAWSCredentialsProvider provider("default");
auto credentials = provider.GetAWSCredentials();
std::cout << "Access Key: " << credentials.GetAWSAccessKeyId() << std::endl;
std::cout << "Secret Key: " << credentials.GetAWSSecretKey() << std::endl;

Error 403 is being observed on running the following

    s3_crt_client_ = Aws::New<Aws::S3Crt::S3CrtClient>("test",config);
    sscanf(s3_uri, "s3://%[^/]/%s", bucket, key);
    Aws::S3Crt::Model::HeadObjectRequest head_object_request;
    Aws::S3Crt::Model::HeadObjectOutcome outcome;

    head_object_request.SetBucket(bucket);
    head_object_request.SetKey(key);
    outcome = s3_crt_client_->HeadObject(head_object_request);

The above cpp code is working fine with access key and secret mentioned in credentials file (not working with credential process only)

Possible Solution

Extract credentials properly from the output of credential process of the format { "Version": 1, "AccessKeyId": "xxxxxxxxx", "SecretAccessKey": "yyyyyyyyyyy", "SessionToken": "zzzzzzzzzzz" "Expiration": "2024-07-11T17:00:32+00:00" }

Additional Information/Context

No response

AWS CPP SDK version used

1.11.269

Compiler and Version used

gcc version 4.8.5 20150623 (Red Hat 4.8.5-4)

Operating System and version

CentOS 7

[sbiscigl]

Hey thanks for opening the issue. Your reproduction steps are not a complete reproduction, going to need to have a Minimal, Reproducible Example. I tried to replicate your issues but could not using the following reproduction steps.

project tree:

~/sdk-usage-workspace
├── CMakeLists.txt
├── Dockerfile
├── config
├── main.cpp
├── provide-creds.sh
└── replicate.sh

Dockerfile:

# Using offical Amazon Linux 2023 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2023

# Install compiler et al.
RUN yum groupinstall "Development Tools" -y

# Install required dependencies
RUN yum install -y curl-devel openssl-devel ninja-build cmake3

# install sdk
RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake -G Ninja -DBUILD_ONLY="s3" -DAUTORUN_UNIT_TESTS=OFF .. && \
    cmake --build . && \
    cmake --install .

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
COPY provide-creds.sh /provide-creds.sh
COPY config /config
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -G Ninja .. &&\
    cmake --build .

CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS s3)
add_executable(${PROJECT_NAME} "main.cpp")
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

main.cpp

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/HeadObjectRequest.h>

using namespace Aws;
using namespace Aws::Auth;
using namespace Aws::S3;
using namespace Aws::S3::Model;

const char* ALLOCATION_TAG = "process-creds-test";
const static char* BUCKET_NAME = replace_with_your_bucket_name ;
const static char* KEY = replace_with_your_key;

int main()
{
    SDKOptions options;
    InitAPI(options);
    {
        const auto credentialsProvider = Aws::MakeShared<ProcessCredentialsProvider>(ALLOCATION_TAG, "test-process-creds");
        S3ClientConfiguration s3ClientConfig{"test-process-creds"};
        S3Client client{credentialsProvider, nullptr, s3ClientConfig};
        const auto response = client.HeadObject(HeadObjectRequest()
                                                .WithBucket(BUCKET_NAME)
                                                .WithKey(KEY));
        if (response.IsSuccess())
        {
            std::cout << "Call succeeded!\n";
        }
        else
        {
            std::cout << "Call failed: " << response.GetError().GetMessage() << "\n";
        }
    }
    ShutdownAPI(options);
    return 0;
}

provide-creds.sh:

#!/bin/sh

echo "{ \"Version\": 1, \"AccessKeyId\": \"${AWS_ACCESS_KEY_ID}\", \"SecretAccessKey\": \"${AWS_SECRET_ACCESS_KEY}\", \"SessionToken\": \"${AWS_SESSION_TOKEN}\" }"

config:

[profile test-process-creds]
credential_process = "/provide-creds.sh"

replicate.sh

#!/bin/zsh

set -u

# build image
docker build -t test-image .

# run example
docker run \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    -e AWS_CONFIG_FILE=/config \
    --name test-image test-image /sdk-example/build/sdk_usage_workspace

when i run with ./replicate.sh i see:

Call succeeded!

If you can update the preceding reproduction to fail, would be more than happy to help out. also you need to make sure your process provider script is marked as executable by the running user, if it is not it will fail and log that it cannot run it. you might need to add execution permissions on it.

[bvs-amagi]

Thank you for a quick response.

Below is the Minimal, Reproducible Example changed to get aws credetials from credential process only. get_s3_creds.tar.gz (Sergey's EDIT: link removed due a unknown binary inside) Getting empty credentials on running s3_reader_ut1 binary But able to get credentials running provide-creds.sh.

Please check above scripts and provide fix to obtain credetials using credential_process in cpp sdk

[SergeyRyabinin]

Hi @bvs-amagi We are not going to execute your binary. Please provide the source code for the s3_reader_ut1 executable instead.

Best regards, Sergey

[sbiscigl]

adding on to sergey, i provided you with a minimal example that works, update that to not work in the way you expect it to work.

[bvs-amagi]

Hi @sbiscigl @SergeyRyabinin,

Below is the Minimal, Reproducible Example changed to get aws credetials from credential process only. get_s3_creds_source_only.tar.gz

The binary executable is removed from the tar. Please check above scripts and provide fix to obtain credetials using credential_process in cpp sdk

Regards, Viswanath

[sbiscigl]

I extracted your code and ran it locally and i see credentials returned. my build tree is:

~/sdk-usage-workspace ❯❯❯ tree -L 1
.
├── CMakeLists.txt
├── Dockerfile
├── config
├── main.cpp
├── provide-creds.sh
└── replicate.sh

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS core)
add_executable(${PROJECT_NAME} "main.cpp")
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

Dockerfile

# Using offical Amazon Linux 2023 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2023

# Install compiler et al.
RUN yum groupinstall "Development Tools" -y

# Install required dependencies
RUN yum install -y curl-devel openssl-devel ninja-build cmake3

# install sdk
RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake -G Ninja -DBUILD_ONLY="s3" -DAUTORUN_UNIT_TESTS=OFF .. && \
    cmake --build . && \
    cmake --install .

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
COPY provide-creds.sh /provide-creds.sh
COPY config /config
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -G Ninja .. &&\
    cmake --build .

config

[default]
credential_process = /provide-creds.sh

main.cpp

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>

using namespace Aws;
using namespace Aws::Auth;

const char* ALLOCATION_TAG = "process-creds-test";

int main()
{
    SDKOptions options;
    InitAPI(options);
    {
        const auto credentialsProvider = Aws::MakeShared<ProcessCredentialsProvider>(ALLOCATION_TAG, "default");
        auto credentials= credentialsProvider->GetAWSCredentials();
        std::cout << "Access Key: " << credentials.GetAWSAccessKeyId() << std::endl;
        std::cout << "Secret Key: " << credentials.GetAWSSecretKey() << std::endl;
        std::cout << "Session Token: " << credentials.GetSessionToken() << std::endl;
    }
    ShutdownAPI(options);
    return 0;
}

provide-creds.sh

#!/bin/sh

echo "{ \"Version\": 1, \"AccessKeyId\": \"${AWS_ACCESS_KEY_ID}\", \"SecretAccessKey\": \"${AWS_SECRET_ACCESS_KEY}\", \"SessionToken\": \"${AWS_SESSION_TOKEN}\" , \"Expiration\": \"${AWS_EXPIRATION}\" }"

replicate.sh

# build image
docker build -t test-image .

# set runtime credentials
export AWS_ACCESS_KEY_ID=REDACTED
export AWS_SECRET_ACCESS_KEY=REDACTED
export AWS_SESSION_TOKEN=REDACTED

# run example
docker run \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    -e AWS_CONFIG_FILE=/config \
    --name test-image test-image /sdk-example/build/sdk_usage_workspace

when I run I see

Access Key: REDACTED Secret Key: REDACTED Session Token: REDACTED

since i can verify it works in a standalone docker environment, the difference is in your build enviornment. Try running in a docker container to isolate the changes between your build environment and the standalone docker environment, there is something different.

Also please provide logs of the run, the logs will show if there is any error that occured while calling the credentials provider please provide the logs if you still see it failing.

[bvs-amagi]

@sbiscigl From above example it is clear that the sdk is built properly in public.ecr.aws/amazonlinux/amazonlinux:2023. Could you please provide the same for centos:7.9.2009 ?

[sbiscigl]

Also please provide logs of the run, the logs will show if there is any error that occured while calling the credentials provider please provide the logs if you still see it failing.

please provide logs

Could you please provide the same for centos:7.9.2009?

I would prefer if you create a docker image to replicate your environment. I do not know your requirements or environment so it would be more meaningful if you try to replicate it as I would be guessing what your environment is.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.