search

minio / minio-cpp

MinIO C++ Client SDK for Amazon S3 Compatible Cloud Storage
https://minio-cpp.min.io/
Apache License 2.0
127 stars 53 forks source link

fix set UploadObjectResponse.version_id when upload big object to bucket with versioning #97

Closed AlexBurnes closed 6 months ago

AlexBurnes commented 11 months ago

minio::s3::Client UploadObject does not return a version-id for big object, but minio returns x-amz-version-id for CompleteMultipartUpload operation. Example code and minio traces are bellow.

The trouble is in PutObject that calls CompliteMulitpartUpload and CompleteMultipartUploadResponse converted into PutObjectResponse via the base object Response that does not have member version_id, so setting this member in CompleteMultipartUploadResponse is lost.

This fix set only version_id field, but other fields defined in CompleteMultipartUploadResponse::ParseXML are lost either and require to be corrected. Perhaps this fix is naive and it is necessary to redesign methods or objects.

upload-big-object.cc 

#include "client.h"
#include "http.h"

int main(int argc, char* argv[]) {

    minio::s3::BaseUrl base_url("http://127.0.0.1:9000", false);
    minio::creds::StaticProvider provider(
      "test", "miniostorage");
    minio::s3::Client client(base_url, &provider);

    // Make bucket 'upload-big-object'
    minio::s3::MakeBucketArgs bucket_args;
    bucket_args.bucket = "test-upload-big-object";
    minio::s3::MakeBucketResponse bucket_resp = client.MakeBucket(bucket_args);
    if (!bucket_resp) {
        std::cout << "unable to create bucket; " << bucket_resp.Error().String() << std::endl;
        return EXIT_FAILURE;
    }

    // Set bucket versioning
    minio::s3::SetBucketVersioningArgs versioning_args;
    versioning_args.bucket = bucket_args.bucket;
    versioning_args.status = true;
    minio::s3::SetBucketVersioningResponse versioning_resp = client.SetBucketVersioning(versioning_args);
    if (!versioning_resp) {
        std::cout << "unable to set bucket versioning; " << versioning_resp.Error().String()
                  << std::endl;
        return EXIT_FAILURE; 
    }

    // Upload object
    minio::s3::UploadObjectArgs upload_args;
    upload_args.bucket = bucket_args.bucket;
    upload_args.object = "vcpkg-master.zip";
    upload_args.filename = "./vcpkg-master.zip";
    minio::s3::UploadObjectResponse upload_resp = client.UploadObject(upload_args);
    if (!upload_resp) {
        std::cout << "unable upload object  '" << upload_args.object << "' , error "
                  << upload_resp.Error().String() << std::endl;
    }
    std::cout << "'" << upload_args.filename << "' is successfully uploaded as "
              << "object '" << upload_args.object << "' "
              << "version '" <<  upload_resp.version_id << "' "
              << "etag '" <<  upload_resp.etag << "'."
              << std::endl;

    std::cout << "x-amz-version-id: " << upload_resp.headers.GetFront("x-amz-version-id") << std::endl;

    // Remove object
    minio::s3::RemoveObjectArgs remove_args;
    remove_args.bucket = bucket_args.bucket;
    remove_args.object = upload_args.object;
    if (upload_resp.version_id != "") {
        remove_args.version_id = upload_resp.version_id;
    }
    minio::s3::RemoveObjectResponse remove_resp = client.RemoveObject(remove_args);
    if (!remove_resp) {
        std::cout << "failed remove object  '" << upload_args.object << "' , error "
                  << remove_resp.Error().String() << std::endl;
    }

    // Delete bucket
    minio::s3::RemoveBucketArgs delete_args;
    delete_args.bucket = bucket_args.bucket;
    minio::s3::RemoveBucketResponse delete_resp = client.RemoveBucket(delete_args);
    if (!delete_resp) {
        std::cout << "failed delete bucket '" << delete_args.bucket << "' , error "
                  << delete_resp.Error().String() << std::endl;
    }

    return upload_resp.version_id == "" 
        ? EXIT_FAILURE
        : EXIT_SUCCESS;
}

Run executable ./upload-big-object 

'./vcpkg-master.zip' is successfully uploaded as object 'vcpkg-master.zip' version '' etag ''.
x-amz-version-id: 
failed delete bucket 'test-upload-big-object' , error BucketNotEmpty: The bucket you tried to delete is not empty

Trace minio shows that x-amz-version-id is present in response: x-amz-version-id: 9749d6f2-b015-4b2f-a575-7b1a3240297c ./minio-client admin trace myminio --verbose

127.0.0.1 [REQUEST s3.NewMultipartUpload] [2023-10-17T11:16:04.208] [Client IP: 127.0.0.1]
127.0.0.1 POST /test-upload-big-object/vcpkg-master.zip?uploads=
127.0.0.1 Proto: HTTP/1.1
127.0.0.1 Host: 127.0.0.1
127.0.0.1 Content-Length: 0
127.0.0.1 Content-Type: application/octet-stream
127.0.0.1 User-Agent: MinIO (Linux; x86_64) minio-cpp/0.1.0
127.0.0.1 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
127.0.0.1 X-Amz-Date: 20231017T031604Z
127.0.0.1 Accept: */*
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=test/20231017/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=f4762f699ae7aaa8eaa56b0285be91d00d7ee558e9d3612466cb86b1b9e82930
127.0.0.1 
127.0.0.1 [RESPONSE] [2023-10-17T11:16:04.208] [ Duration 261µs  ↑ 97 B  ↓ 589 B ]
127.0.0.1 200 OK
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 Strict-Transport-Security: max-age=31536000; includeSubDomains
127.0.0.1 Vary: Origin,Accept-Encoding
127.0.0.1 X-Amz-Request-Id: 178EC6AEEF68D62B
127.0.0.1 X-Content-Type-Options: nosniff
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 Content-Length: 273
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 Content-Type: application/xml
127.0.0.1 Server: MinIO
127.0.0.1 <?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>test-upload-big-object</Bucket><Key>vcpkg-master.zip</Key><UploadId>7ed49475-2a46-4760-8ed1-4b116887a8c8</UploadId></InitiateMultipartUploadResult>
127.0.0.1 
127.0.0.1 [REQUEST s3.PutObjectPart] [2023-10-17T11:16:04.211] [Client IP: 127.0.0.1]
127.0.0.1 PUT /test-upload-big-object/vcpkg-master.zip?partNumber=1&uploadId=7ed49475-2a46-4760-8ed1-4b116887a8c8
127.0.0.1 Proto: HTTP/1.1
127.0.0.1 Host: 127.0.0.1
127.0.0.1 Content-Length: 5242880
127.0.0.1 Content-Type: application/octet-stream
127.0.0.1 User-Agent: MinIO (Linux; x86_64) minio-cpp/0.1.0
127.0.0.1 X-Amz-Content-Sha256: 8d6cef74aa70e408174383ab44581a6dcf4b5bc0d4f10fdf3603a208c66a772e
127.0.0.1 X-Amz-Date: 20231017T031604Z
127.0.0.1 Accept: */*
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=test/20231017/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=71c2a9dbfae9f7b697704d012937a9606c3a8ea92075304338b7dd4214d4d9b2
127.0.0.1 <BODY>
127.0.0.1 [RESPONSE] [2023-10-17T11:16:04.237] [ Duration 25.314ms  ↑ 5.0 MiB  ↓ 325 B ]
127.0.0.1 200 OK
127.0.0.1 ETag: "c69a8b3fa485f2331ef906098c092240"
127.0.0.1 Server: MinIO
127.0.0.1 Strict-Transport-Security: max-age=31536000; includeSubDomains
127.0.0.1 Vary: Origin,Accept-Encoding
127.0.0.1 X-Amz-Request-Id: 178EC6AEEF9CBBFF
127.0.0.1 X-Content-Type-Options: nosniff
127.0.0.1 Content-Length: 0
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 <BODY>
127.0.0.1 
127.0.0.1 [REQUEST s3.PutObjectPart] [2023-10-17T11:16:04.240] [Client IP: 127.0.0.1]
127.0.0.1 PUT /test-upload-big-object/vcpkg-master.zip?partNumber=2&uploadId=7ed49475-2a46-4760-8ed1-4b116887a8c8
127.0.0.1 Proto: HTTP/1.1
127.0.0.1 Host: 127.0.0.1
127.0.0.1 Content-Type: application/octet-stream
127.0.0.1 User-Agent: MinIO (Linux; x86_64) minio-cpp/0.1.0
127.0.0.1 X-Amz-Content-Sha256: 6fab4ed8965a8ad543b37ccfd449c7a35cb36e5ac2eaedffaffe36673f4c93f2
127.0.0.1 X-Amz-Date: 20231017T031604Z
127.0.0.1 Accept: */*
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=test/20231017/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=49d0065697782b07c33a435e12d3610930ecb68c0b9a210e77b82b80eae34398
127.0.0.1 Content-Length: 2480114
127.0.0.1 <BODY>
127.0.0.1 [RESPONSE] [2023-10-17T11:16:04.261] [ Duration 21.561ms  ↑ 2.4 MiB  ↓ 325 B ]
127.0.0.1 200 OK
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 Content-Length: 0
127.0.0.1 ETag: "23b423c1ca78f4ca42aded07790ef91b"
127.0.0.1 Strict-Transport-Security: max-age=31536000; includeSubDomains
127.0.0.1 Vary: Origin,Accept-Encoding
127.0.0.1 X-Amz-Request-Id: 178EC6AEF150AB09
127.0.0.1 X-Content-Type-Options: nosniff
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 Server: MinIO
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 <BODY>
127.0.0.1 
127.0.0.1 [REQUEST s3.CompleteMultipartUpload] [2023-10-17T11:16:04.263] [Client IP: 127.0.0.1]
127.0.0.1 POST /test-upload-big-object/vcpkg-master.zip?uploadId=7ed49475-2a46-4760-8ed1-4b116887a8c8
127.0.0.1 Proto: HTTP/1.1
127.0.0.1 Host: 127.0.0.1
127.0.0.1 Content-Type: application/xml
127.0.0.1 User-Agent: MinIO (Linux; x86_64) minio-cpp/0.1.0
127.0.0.1 X-Amz-Date: 20231017T031604Z
127.0.0.1 Accept: */*
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=test/20231017/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=31b7dae01494e7c7b0536e1019d200f1cf837dbddb553454dfb267b676777a68
127.0.0.1 Content-Length: 223
127.0.0.1 Content-Md5: ljYm4RKls9NAIqMqg9XtEQ==
127.0.0.1 X-Amz-Content-Sha256: 6f7740a334e258e0bbcdda1edba549cb7e9baee738ec0336872026dedd5605cb
127.0.0.1 <CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"c69a8b3fa485f2331ef906098c092240"</ETag></Part><Part><PartNumber>2</PartNumber><ETag>"23b423c1ca78f4ca42aded07790ef91b"</ETag></Part></CompleteMultipartUpload>
127.0.0.1 [RESPONSE] [2023-10-17T11:16:04.278] [ Duration 15.257ms  ↑ 332 B  ↓ 810 B ]
127.0.0.1 200 OK
127.0.0.1 Vary: Origin,Accept-Encoding
127.0.0.1 x-amz-version-id: 9749d6f2-b015-4b2f-a575-7b1a3240297c
127.0.0.1 Content-Type: application/xml
127.0.0.1 Server: MinIO
127.0.0.1 Content-Length: 350
127.0.0.1 X-Amz-Request-Id: 178EC6AEF2B2D4C1
127.0.0.1 X-Content-Type-Options: nosniff
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 Cache-Control: no-cache
127.0.0.1 X-Accel-Buffering: no
127.0.0.1 ETag: "44181358b80e51b26ec8e7d697e283bb-2"
127.0.0.1 Strict-Transport-Security: max-age=31536000; includeSubDomains
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 <?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://127.0.0.1/test-upload-big-object/vcpkg-master.zip</Location><Bucket>test-upload-big-object</Bucket><Key>vcpkg-master.zip</Key><ETag>&#34;44181358b80e51b26ec8e7d697e283bb-2&#34;</ETag></CompleteMultipartUploadResult>
balamurugana commented 6 months ago

Refer https://github.com/minio/minio-cpp/pull/107