nspcc-dev / neofs-s3-gw

NeoFS S3 Protocol Gateway
GNU Affero General Public License v3.0
7 stars 15 forks source link

Remove re-slicing for multipart #931

Closed smallhive closed 5 months ago

smallhive commented 6 months ago

Closes #843

codecov[bot] commented 6 months ago

Codecov Report

Attention: Patch coverage is 1.91388% with 615 lines in your changes are missing coverage. Please review.

Project coverage is 26.92%. Comparing base (aadef93) to head (127dd3e). Report is 1 commits behind head on master.

Files Patch % Lines
api/layer/multipart_upload.go 0.00% 265 Missing :warning:
api/layer/neofs_mock.go 8.54% 102 Missing and 5 partials :warning:
internal/neofs/tree.go 0.00% 86 Missing :warning:
api/layer/tree_mock.go 0.00% 55 Missing :warning:
api/layer/object.go 0.00% 49 Missing and 1 partial :warning:
internal/neofs/neofs.go 0.00% 47 Missing :warning:
api/layer/layer.go 33.33% 3 Missing and 1 partial :warning:
api/handler/acl.go 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #931 +/- ## ========================================== - Coverage 27.97% 26.92% -1.06% ========================================== Files 87 87 Lines 13590 14129 +539 ========================================== + Hits 3802 3804 +2 - Misses 9371 9903 +532 - Partials 417 422 +5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

smallhive commented 6 months ago

Rebased

cthulhu-rider commented 6 months ago

main commit with reslicing removal contains only issue ref which is good for GH but not enough for git history overall. Lets write some text about motivation and used approach

smallhive commented 5 months ago

What happens if we're to reupload a part?

Everything will be broken. We collect full object hash moving from part to part. If you re-upload any part in the middle, we will have to re-read all parts after it, recalculate hashes, and re-upload parts

roman-khimov commented 5 months ago

Everything will be broken. We collect full object hash moving from part to part. If you re-upload any part in the middle, we will have to re-read all parts after it, recalculate hashes, and re-upload parts

Cool. We need to do this. And that's where per-part IDs can become handy because

Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten.

And btw,

Each part must be at least 5 MB in size, except the last part.

smallhive commented 5 months ago

Everything will be broken. We collect full object hash moving from part to part. If you re-upload any part in the middle, we will have to re-read all parts after it, recalculate hashes, and re-upload parts

Cool. We need to do this. And that's where per-part IDs can become handy because

Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten.

And btw,

Each part must be at least 5 MB in size, except the last part.

Added logic for re-upload parts