AWS SDK crashed in Aws::Client::AWSClient::AttemptExhaustively

srinivasab3 commented 2 months ago

Describe the bug

In our application, when it makes the repeated calls to LambdaClient::Invoke. After some time it crashes and following is the backtrace.

#0  0x0000fffff63e1f14 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const ()
   from /opt/ros/jellyfish/tools/lib/libaws-cpp-sdk-core.so
(gdb) bt
#0  0x0000fffff63e1f14 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const ()
   from /opt/ros/jellyfish/tools/lib/libaws-cpp-sdk-core.so
#1  0x0000fffff63e2518 in Aws::Client::AWSClient::MakeRequestWithUnparsedResponse(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const ()
   from /opt/ros/jellyfish/tools/lib/libaws-cpp-sdk-core.so
#2  0x0000fffff6fd44e8 in Aws::Lambda::LambdaClient::Invoke(Aws::Lambda::Model::InvokeRequest const&) const () from /opt/ros/jellyfish/tools/lib/libaws-cpp-sdk-lambda.so
#3  0x0000aaaaaac90c18 in LambdaInterface::invokeSensorBatteryLambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, Json::Value&) ()

Previously we were using 1.8.130 and now we have updated to 1.11.367 assuming that SDK update would solve this issue. But still the issue persists.

Expected Behavior

No crash

Current Behavior

Segmentation fault.

Reproduction Steps

Multiple repeated calls to LambdaClient::Invoke.

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.367

Compiler and Version used

G++ 14

Operating System and version

Linux 5.10.4

jmklix commented 2 months ago

Can you provide minimal reproduction code for this crash?

sbiscigl commented 2 months ago

Reproduction Steps Multiple repeated calls to LambdaClient::Invoke.

I tried reproducing for you, and I cannot

project structure

~/sdk-usage-workspace ❯❯❯ tree
.
├── CMakeLists.txt
├── Dockerfile
├── main.cpp
└── replicate.sh

replicate.sh

#!/bin/zsh

set -u

# build image
docker build -t test-image .

# run example
docker run \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    --name test-image test-image /sdk-example/build/sdk_usage_workspace

Dockerfile

# Using offical Amazon Linux 2023 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2023

# Install compiler et al.
RUN yum groupinstall "Development Tools" -y

# Install required dependencies
RUN yum install -y curl-devel openssl-devel ninja-build cmake3

# install sdk
RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake -G Ninja -DBUILD_ONLY="lambda" -DAUTORUN_UNIT_TESTS=OFF .. && \
    cmake --build . && \
    cmake --install .

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -G Ninja .. &&\
    cmake --build .

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS lambda)
add_executable(${PROJECT_NAME} "main.cpp")
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

main.cpp

#include <aws/core/Aws.h>
#include <aws/lambda/LambdaClient.h>
#include <aws/lambda/model/InvokeRequest.h>

using namespace Aws::Lambda;
using namespace Aws::Lambda::Model;

const char* FUNCTION_NAME = "YourFunctionName";
const int FUNCTION_ITERATIONS = 100;

class AwsSDKState
{
public:
    explicit AwsSDKState(Aws::SDKOptions options)
        : options_(std::move(options))
    {
        Aws::InitAPI(options_);
    }

    AwsSDKState(const AwsSDKState& other) = delete;
    AwsSDKState(AwsSDKState&& other) noexcept = delete;
    AwsSDKState& operator=(const AwsSDKState& other) = delete;
    AwsSDKState& operator=(AwsSDKState&& other) noexcept = delete;

    ~AwsSDKState()
    {
        Aws::ShutdownAPI(options_);
    };

private:
    Aws::SDKOptions options_;
};

int main()
{
    AwsSDKState object{{}};
    LambdaClient client;
    for (size_t iterations = 0; iterations < FUNCTION_ITERATIONS; ++iterations)
    {
        const auto result = client.Invoke(InvokeRequest().WithFunctionName(FUNCTION_NAME));
        assert(result.IsSuccess());
        if (result.IsSuccess())
        {
            std::cout << "Ran function: " << FUNCTION_NAME << "\n";
        }
        else
        {
            std::cout << "Failed to invoke function: " << FUNCTION_NAME << "\n";
        }
    }
    return 0;
}

If you can update this example to replicate your issue would be happy to look at it, but as far as we can tell this is working as expected.

github-actions[bot] commented 1 month ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

aws / aws-sdk-cpp