Multipart uploads don't work in Riak S2 (CS) v2.1.1 [JIRA: RCS-378]

[manifest]

The problem is that multipart upload identifier cannot be found after it has been successfully initialized. This issue can be easily reproduced:

## This project can be used for a quick roll up of development environment
$ git clone https://github.com/manifest/riak-s2-erlang-client.git
$ cd riak-s2-erlang-client
$ git co feature/multipart-uploads

## The build script will install Riak S2 within Docker container,
## create a user for you and store all information required for
## connection to the .docker.env.config file.
$ ./run-docker.sh

## Running Riak S2 client
$ make app shell

%% Initializing connection to Riak S2
{ok, Conf} = file:consult(".docker.env.config"),
{_, Opts} = lists:keyfind(user, 1, Conf),
{_, #{host := Host, port := Port}} = lists:keyfind(httpc_options, 1, Conf),
{ok, Pid} = gun:open(Host, Port, #{protocols => [http]}).

%% Creating a bucket for the testcase
riaks2c_bucket:put(Pid, <<"test-bucket">>, Opts).

%% Initializing multipart upload
#'InitiateMultipartUploadResult'{'UploadId' = Id} = riaks2c_object_multipart:init(Pid, <<"test-bucket">>, <<"test_file">>, #{}, Opts).
%% #'InitiateMultipartUploadResult'{
%%   anyAttribs = [],
%%   'Bucket' = <<"test-bucket">>,'Key' = <<"test_file">>,
%%   'UploadId' = <<"cj-EXDu-T-q7N6ifBOux9A==">>}

%% Trying to upload a part, we get the error with the code: "NoSuchUpload".
riaks2c_object_multipart:put(Pid, <<"test-bucket">>, <<"test_multipart">>, <<"1">>, Id, 1, #{}, Opts).
%% ** exception error: {bad_upload_id,<<"test-bucket">>,<<"test_multipart">>,<<"cj-EXDu-T-q7N6ifBOux9A==">>}

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchUpload</Code>
  <Message>The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.</Message>
  <UploadId>cj-EXDu-T-q7N6ifBOux9A==</UploadId>
  <HostId>host-id</HostId>
</Error>

We can see that our upload "cj-EXDu-T-q7N6ifBOux9A==" is exist using erlang-s2-erlang-client.

riaks2c_object_multipart:list(Pid, <<"test-bucket">>, #{}, Opts).
%% #'ListMultipartUploadsResult'{
%%   anyAttribs = [],'Bucket' = <<"test-bucket">>,
%%   'KeyMarker' = [],'UploadIdMarker' = undefined,
%%   'NextKeyMarker' = [],'NextUploadIdMarker' = [],
%%   'Delimiter' = [],'Prefix' = [],'MaxUploads' = 1000,
%%   'IsTruncated' = false,
%%   'Upload' = 
%%     [#'Upload'{
%%       anyAttribs = [],'Key' = <<"test_file">>,
%%       'UploadId' = <<"cj-EXDu-T-q7N6ifBOux9A==">>,
%%       'Initiator' = 
%%         #'CanonicalUser'{
%%           anyAttribs = [],'ID' = <<"HTFXXZNN8WD1JG7NQ-G_">>,
%%           'DisplayName' = <<"user">>},
%%           'Owner' = 
%%             #'CanonicalUser'{
%%               anyAttribs = [],'ID' = <<"HTFXXZNN8WD1JG7NQ-G_">>,
%%               'DisplayName' = <<"user">>},
%%           'StorageClass' = <<"STANDARD">>,
%%           'Initiated' = <<"2016-11-06T10:55:13.000Z">>}],
%%   'CommonPrefixes' = undefined}

<?xml version="1.0" encoding="UTF-8"?>
<ListMultipartUploadsResult
  xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Bucket>test-bucket</Bucket>
  <KeyMarker></KeyMarker>
  <NextKeyMarker/>
  <NextUploadIdMarker></NextUploadIdMarker>
  <Delimiter></Delimiter>
  <Prefix></Prefix>
  <MaxUploads>1000</MaxUploads>
  <IsTruncated>false</IsTruncated>
  <Upload>
    <Key>test_file</Key>
    <UploadId>cj-EXDu-T-q7N6ifBOux9A==</UploadId>
    <Initiator>
      <ID>HTFXXZNN8WD1JG7NQ-G_</ID>
      <DisplayName>user</DisplayName>
    </Initiator>
    <Owner>
      <ID>HTFXXZNN8WD1JG7NQ-G_</ID>
      <DisplayName>user</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
    <Initiated>2016-11-06T10:55:13.000Z</Initiated>
  </Upload>
</ListMultipartUploadsResult>

The same using s3cmd client:

$ s3cmd --signature-v2 -c s3cmd.cfg multipart s3://test-bucket
s3://test-bucket/
Initiated   Path    Id
2016-11-06T10:55:13.000Z    s3://test-bucket/test_file  cj-EXDu-T-q7N6ifBOux9A==

The same "NoSuchUpload" error, trying to abort upload:

$ s3cmd --signature-v2 -c s3cmd.cfg abortmp s3://test-bucket/test_multipart cj-EXDu-T-q7N6ifBOux9A==
ERROR: S3 error: 404 (NoSuchUpload): The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

Or trying to list parts of our upload.

$ s3cmd --signature-v2 -c s3cmd.cfg listmp s3://test-bucket/test_multipart cj-EXDu-T-q7N6ifBOux9A==
ERROR: S3 error: 404 (NoSuchUpload): The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

s3cmd.cfg that is used in the testcase

[default]
access_key = HTFXXZNN8WD1JG7NQ-G_
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2010-07-15/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
enable_multipart = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/local/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s
gpg_passphrase = password
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
list_md5 = False
log_target_prefix =
preserve_attrs = True
progress_meter = True
proxy_host = 192.168.99.100
proxy_port = 8080
recursive = False
recv_chunk = 4096
reduced_redundancy = False
secret_key = Fl-QLrrjZVgWrdBnlLA-eW_gGRljWLw9zTOE8g==
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
urlencoding_mode = normal
use_https = False
verbosity = INFO

[manifest]

Are there any comments on this issue?

[ksauzz]

It seems you used the different keys between Initiate Multipart Upload (test_file) and Part Upload (test_multipart). Using the same key over a series of Multipart Upload requests should work.

[manifest]

@ksauzz you're right! My fault. Thanks.