search

aws / aws-sdk-cpp

AWS SDK for C++
Apache License 2.0
1.92k stars 1.04k forks source link

Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-common/src/v0.9.10-492b35b91d.clean/source/allocator.c:172: allocator != ((void *)0) #3030

Closed westfly closed 1 day ago

westfly commented 2 weeks ago

Describe the bug

As https://github.com/aws/aws-sdk-cpp/issues/2609 issue describe , We newly Use vcpkg As cpp-library manager while aws-sdk-cpp updated version from 1.7.108 to 1.11.225, the problem still exsit.

We Use the project manged by vcpkg to build basic example and run as expected successful，so We think Our build-env not the core problem.

Expected Behavior

works same as aws-sdk-cpp 1.7.108 building native without vcpkg manager

Current Behavior

https://github.com/aws/aws-sdk-cpp/issues/2609 Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-common/src/v0.9.10-492b35b91d.clean/source/allocator.c:172: allocator != ((void *)0) Exiting Application ################################################################################ Stack trace: ################################################################################ /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_backtrace_print+0x61) [0x7f1ade9b9e81] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_fatal_assert+0x4d) [0x7f1ade9aec0d] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_mem_acquire+0x3e) [0x7f1ade9add0e] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_string_new_from_array+0x25) [0x7f1ade9bafb5] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_json_value_get_from_object+0x2a) [0x7f1ade9b676a] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(aws_endpoints_ruleset_new_from_string+0x76) [0x7f1ade9a7916] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN3Aws3Crt9Endpoints10RuleEngineC1ERK15aws_byte_cursorS5_P13aws_allocator+0x30) [0x7f1ade8887a0] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha11S3SDKStreamC1Ev+0xfc) [0x7f1ade13bd1c] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha12S3FileSystem4OpenERKNS_3URIEPKcb+0x23) [0x7f1ade1304e3] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha6Stream6CreateEPKcS2_b+0x12e) [0x7f1ade12b6fe] /data/aresyang/gitlab/ps/build_vcpkg/_mindalpha.so(_ZN9mindalpha13StreamReadAllERKSs+0x41) [0x7f1ade12c601]

Reproduction Steps

As describe above, the offical sdk example work right. We use awk-sdk-cpp to access s3 bucket implement at s3_sdk_filesys.cpp which call Aws::InitAPI, We build static library mindalpha_filesystem via cmake

add_library(mindalpha_filesystem STATIC
    cpp/mindalpha/io.cpp
    cpp/mindalpha/filesys.cpp
    cpp/mindalpha/local_filesys.cpp
    cpp/mindalpha/s3_sdk_filesys.cpp
    cpp/mindalpha/stack_trace_utils.h
    cpp/mindalpha/stack_trace_utils.cpp
)
target_link_libraries(mindalpha_filesystem PRIVATE
    aws-cpp-sdk-s3
    aws-cpp-sdk-core
    spdlog::spdlog
)

and add a little test case on mindalpha_filesystem 

#include <mindalpha/io.h>
int main(int argc, char const *argv[]) {
  if (argc < 1) {
    printf("usage: binary s3_path\n");
  }
  {
      std::string path(argv[1]);
      auto content = mindalpha::StreamReadAll(path);
      printf("%s\t%ld\n", content.data(), content.size());
  }
  return 0;
}

below is cmake

macro(unittest_helper_lite target cpp_file)
    add_executable(${target}
        ${cpp_file}
    )
    target_compile_features(${target} INTERFACE cxx_std_17)
    target_link_libraries(${target} PRIVATE
        mindalpha_filesystem
        spdlog::spdlog
        dl
        util
        pthread
        ${Boost_LIBRARIES}    )
endmacro(unittest_helper_lite)
unittest_helper_lite(s3_filesystem tests/cpp/s3_filesystem_test.cpp)
unittest_helper_lite(hello_s3 tests/cpp/hello_s3.cpp)

the coredump as below

 ./s3_filesystem s3://mob-emr-test/ml-platform/mindalpha/server.conf
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-common/src/v0.9.10-492b35b91d.clean/source/allocator.c:172: allocator != ((void *)0)
Exiting Application
################################################################################
Stack trace:
################################################################################
./s3_filesystem(+0x92efd1) [0x55f3c99bcfd1]
./s3_filesystem(+0x923d5d) [0x55f3c99b1d5d]
./s3_filesystem(+0x922e5e) [0x55f3c99b0e5e]
./s3_filesystem(+0x930105) [0x55f3c99be105]
./s3_filesystem(+0x92b8ba) [0x55f3c99b98ba]
./s3_filesystem(+0x91ca66) [0x55f3c99aaa66]
./s3_filesystem(+0x7fd7a0) [0x55f3c988b7a0]
./s3_filesystem(+0xf519c) [0x55f3c918319c]
./s3_filesystem(+0xe9493) [0x55f3c9177493]
./s3_filesystem(+0xe3cbe) [0x55f3c9171cbe]
./s3_filesystem(+0xe4bc1) [0x55f3c9172bc1]
./s3_filesystem(+0xe086b) [0x55f3c916e86b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7a5c86a083]
./s3_filesystem(+0xe2a8e) [0x55f3c9170a8e]
Aborted (core dumped)

but when We make Aws::InitAPI call on main and remove s3_sdk_filesys.cpp call Aws::InitAPI and Aws::ShutdownAPI which only calling once, the program works right

#include <mindalpha/io.h>
#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/core/auth/AWSCredentialsProviderChain.h>
int main(int argc, char const *argv[]) {
  if (argc < 1) {
    printf("usage: binary s3_path\n");
  }
  Aws::SDKOptions options;
  Aws::InitAPI(options);
  {
      std::string path(argv[1]);
      // mindalpha::InputStream stream(path);
      auto content = mindalpha::StreamReadAll(path);
      printf("%s\t%ld\n", content.data(), content.size());
  }
  Aws::ShutdownAPI(options);
  return 0;
}

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.225

Compiler and Version used

gcc 9.4

Operating System and version

ubuntu 20.04

SergeyRyabinin commented 2 weeks ago

Hello, Thank you for submitting this issue. Please refer to our basic use: https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/basic-use.html

SDK was initially designed around global context pointers (such as memory allocator, loggers, etc). In your caae, some of SDK resources (such as endpoint provider within S3 client) leak outside InitAPI { ... } ShutdownAPI scope and fails to de-allocate memory due to memory system being already terminated.

Before it was working thanks to a undefined behavior, but also we had to perform some SDK components refactoring, so the issue related to global context started to pop out.

Best regards, Sergey

westfly commented 2 weeks ago

Hello, Thank you for submitting this issue. Please refer to our basic use: https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/basic-use.html

SDK was initially designed around global context pointers (such as memory allocator, loggers, etc). In your caae, some of SDK resources (such as endpoint provider within S3 client) leak outside InitAPI { ... } ShutdownAPI scope and fails to de-allocate memory due to memory system being already terminated.

Before it was working thanks to a undefined behavior, but also we had to perform some SDK components refactoring, so the issue related to global context started to pop out.

Best regards, Sergey

When removing the vcpkg environment, I build aws-sdk-cpp on native ubuntu20.04 via the shell script below 

function make_aws_cpp_sdk_deps() {
 version=1.7.108
 version=1.11.225
 version=1.11.33
 version=1.8.126
 version=1.9.0
 git clone --depth 1 -b $version https://github.com/aws/aws-sdk-cpp.git /tmp/aws-sdk-cpp && \
    cd /tmp/aws-sdk-cpp && git submodule update --init --recursive &&           \
    cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr        \
          -DBUILD_SHARED_LIBS=OFF -DENABLE_TESTING=OFF -DBUILD_ONLY=s3           \
          -D_GLIBCXX_USE_CXX11_ABI=1                               \
          -H/tmp/aws-sdk-cpp -B/tmp/aws-sdk-cpp/build &&                         \
    cmake --build /tmp/aws-sdk-cpp/build --target install &&                     \
    rm -rf /tmp/aws-sdk-cpp
}

I found 1.8.126 works right, while 1.9.0 which adds aw-crt-cpp as a submodule coredump at aws::InitAPI

image

DmitriyMusatkin commented 2 weeks ago

Re: first issue Are you sure mindalpha::StreamReadAll is actually calling InitAPI internally? Taking a quick look through the mindalpha code im not sure it actually will. In that case it makes sense why adding Init and Shutdown outside of mindalpha fixes the issue. As a side note: mindalpha should really avoid reinitializing client on every api call, that is very expensive.

Re: core dump issue core dumps in init often occur because build had some stale artifacts and things like std lib implementations are no longer matching. make sure to clean your build directory before rebuilding it

github-actions[bot] commented 5 days ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.