GetObjectAsync with Aws::FStream as a ResponseStreamFactory creates the file locally and populates it with <Error><Code>NoSuchKey</Code> when the file doesn't exist on S3

[chreniuc]

Describe the bug

Hi,

we are trying to download a file from S3 using GetObjectAsync with Aws::FStream as a ResponseStreamFactory so it downloads it directly in a file locally.

The thing is that when that file doesn't exists on S3, it generates a local file with some xml content that says that the file cannot be found and it also logs this:

Encountered error while trying to download file. Error: HTTP response code: 404
Resolved remote host IP address: 52.218.56.235
Request ID: 
Exception name: 
Error message: 
6 response headers:
content-type : application/xml
date : Tue, 13 Sep 2022 12:19:40 GMT
server : AmazonS3
transfer-encoding : chunked
x-amz-id-2 : NlMtAG1blocEMIDJ6aobXMwEUVGPBKNlojHBgfno37fVgSfTe23ozBIjq1T9jRCtp7YsT2uHzc0=
x-amz-request-id : BJ7HZHSN2TB8F2FA.

Expected Behavior

The local file shouldn't be created locally and populated when it doesn't exist on S3.

Current Behavior

The file is created and populated with the following xml:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>filename</Key><RequestId>DSC3VMT54C5BSWT5</RequestId><HostId>tIOhVB6b689IrIgqn/iuT4o2ciWKa3Of8P8fPLFUlDkX7r//LRgiMnsz1S4N1sKU5bJ5O2yu/o8=</HostId></Error>

Reproduction Steps

Code used to download the file and the handler:

Aws::S3::Model::GetObjectRequest request;
request.WithKey(key.c_str()).WithBucket(m_bucketName.c_str());

request.SetResponseStreamFactory([=]()
{
    return Aws::New<Aws::FStream>("S3DOWNLOAD", localFilePath,
        std::ios_base::out | std::ios_base::binary);
});

std::shared_ptr<Aws::Client::AsyncCallerContext> context =
    Aws::MakeShared<AsyncS3StorageCallerContext>("", shared_from_this());
context->SetUUID(key.c_str());

m_S3Client->GetObjectAsync(request, handler, context);

And our handler is this:

if (!outcome.IsSuccess())
    {
        LOG_ERROR("Encountered error while trying to download file. Error: %s.")
            % outcome.GetError();
        return;
    }

Possible Solution

Do not create the file locally if 404 is received.

Additional Information/Context

No response

AWS CPP SDK version used

1.8.1

Compiler and Version used

gcc 7.4

Operating System and version

Ubuntu 20.04

[jzakrzewski]

We had the same problem. It turns out the SDK has no idea about the file, because it only cares about the stream and the stream must be opened before the transfer. The file gets created by the FStream constructor.

Furthermore, if you wish to actually have an error message, you need to make it possible to rewind the stream. We ended up with std::ios_base::in | std::ios_base::out | std::ios_base::trunc | std::ios_base::binary flags for the stream, and we delete the file ourselves.

[jmklix]

Can you give us a minimal reproduction sample? I'm not able to reproduce the error that you're seeing with the code snippet that you pasted above.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.