buffer_pool->mem_limit error with CRT

[jeanbez]

Describe the bug

While trying to read an object to a buffer with the CRT, the following internal buffer pool error occurs. The same code does not cause this error without using CRT.

Fatal error condition occurred in aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-s3/source/s3_buffer_pool.c:243: size <= buffer_pool->mem_limit
Exiting Application
################################################################################
Stack trace:
################################################################################
1   libaws-c-common.1.0.0.dylib         0x0000000100c32091 aws_backtrace_print + 177
2   libaws-c-common.1.0.0.dylib         0x0000000100c06fe3 aws_fatal_assert + 83
3   libaws-c-s3.1.0.0.dylib             0x000000010081407f aws_s3_buffer_pool_reserve + 175
4   libaws-c-s3.1.0.0.dylib             0x000000010080caed s_s3_auto_ranged_get_update + 1069
5   libaws-c-s3.1.0.0.dylib             0x0000000100827468 aws_s3_meta_request_update + 184
6   libaws-c-s3.1.0.0.dylib             0x000000010081af48 aws_s3_client_update_meta_requests_threaded + 344
7   libaws-c-s3.1.0.0.dylib             0x000000010081dc4a s_s3_client_process_work_default + 1162
8   libaws-c-s3.1.0.0.dylib             0x000000010081e62a s_s3_client_process_work_task + 266
9   libaws-c-common.1.0.0.dylib         0x0000000100c3a15d aws_task_run + 253
10  libaws-c-common.1.0.0.dylib         0x0000000100c3a7f4 s_run_all + 468
11  libaws-c-common.1.0.0.dylib         0x0000000100c3b307 aws_task_scheduler_run_all + 71
12  libaws-c-io.1.0.0.dylib             0x0000000100a3bdb7 aws_event_loop_thread + 1655
13  libaws-c-common.1.0.0.dylib         0x0000000100c330ab thread_fn + 523
14  libsystem_pthread.dylib             0x00007ff80253a202 _pthread_start + 99
15  libsystem_pthread.dylib             0x00007ff802535bab thread_start + 15

Expected Behavior

Read to complete without a buffer error.

Current Behavior

Execution halts due to error.

Reproduction Steps

The error appears while trying to read a ranged request; in this case, it is the whole object.

        Aws::S3Crt::Model::GetObjectRequest request;
        request.SetBucket(fromBucket);
        request.SetKey(objectKey);
        request.SetRange("bytes=" + std::to_string(offset) + "-" + std::to_string(offset + size));
        request.SetResponseStreamFactory(AwsWriteableStreamFactory(buffer, size));

        Aws::S3Crt::Model::GetObjectOutcome outcome = aws_crt_client->GetObject(request);

        if (outcome.IsSuccess()) {
            return true;
        }
        else {
            std::cout << "[AWS-S3] GetObject error:\n" << outcome.GetError() << std::endl << std::endl;

            return false;
        }

class StringViewStream : Aws::Utils::Stream::PreallocatedStreamBuf, public std::iostream {
 public:
  StringViewStream(const void* data, int64_t nbytes)
      : Aws::Utils::Stream::PreallocatedStreamBuf(
            reinterpret_cast<unsigned char*>(const_cast<void*>(data)),
            static_cast<size_t>(nbytes)),
        std::iostream(this) {}
};

Aws::IOStreamFactory AwsWriteableStreamFactory(void* data, int64_t nbytes) {
  return [=]() { return new StringViewStream(data, nbytes); };
}

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.239

Compiler and Version used

Apple clang version 15.0.0 (clang-1500.1.0.2.5)

Operating System and version

MacOs Sonoma 14.0 (23A344)

[jmklix]

Can you provide the full code for a minimal sample that reproduces this?

[DmitriyMusatkin]

Can you provide details on how you are configuring the client and what are the rough sizes of the data you are downloading. The assertion is triggered when client is trying to allocate a single buffer larger than the memory limit, which by default should be around 2 GB

[jeanbez]

The requests were very small, so they should not try to allocate more than the default memory limit in any way. I am unsure why this error got triggered, but multiple object keys I was working on had a ./ at the beginning of their names. After removing that and ensuring / was only used in between, that error no longer occurred. tbh, I do not see how they could be related, but the error seems to have been solved with just this change.

[DmitriyMusatkin]

I've tried a few combinations and so far I was not able to reproduce the error. As far as I can tell, having object keys start with "./" made no difference. Would it be possible to provide a full code sample to repro, including the sizes of objects causing the issue? Can you also provide crt trace logs? (setting SDK logging options to trace would also enable CRT logging)

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.