storj / team-metainfo

GNU Affero General Public License v3.0
0 stars 0 forks source link

Add support for custom list of parts while committing multipart upload #22

Open mniewrzal opened 2 years ago

mniewrzal commented 2 years ago

Currently, every part that was uploaded with MPU will be part of object after committing object. For better flexibility and S3 compatibility, we need a way to specify list of parts that will be part of objects. Similar to how S3 is doing that: https://docs.aws.amazon.com/cli/latest/reference/s3api/complete-multipart-upload.html under {{--multipart-upload}}

This change requires change in every layer: metainfo protobuf, satellite, uplink.

For metabase we already hve a method that should be used for this purpose: metabase.CommitObjectWithSegments but we still need to wire it up with metainfo endpoint.

One thing that needs to be discussed is if we want to support ETags in parts list. S3 notation:

Parts=[{ETag=string,PartNumber=integer},{ETag=string,PartNumber=integer}]

I’m not even sure if that possible in our case but we need to check that.
Migrated from https://storjlabs.atlassian.net/browse/PG-293

mniewrzal commented 2 years ago

So initial idea would be to send with ObjectCommitRequest array of such struct:

message PartETag {
    int32 part_number = 1;
    bytes encrypted_e_tag = 2;
}

Problem with this solution is:

Fadila82 commented 2 years ago

The encrypted etag is kept in a dedicated column in the segments table.

I think we can support etags in the parts list, but maybe make them optional.

Erikvv commented 2 years ago

Steve asked Jake if we need this for a customer.