search

owncloud / ocis

:atom_symbol: ownCloud Infinite Scale Stack
https://doc.owncloud.com/ocis/next/
Apache License 2.0
1.35k stars 174 forks source link

S3ng clean up unfinished multipart-uploads #4968

Open wkloucek opened 1 year ago

wkloucek commented 1 year ago

Describe the bug

When you use S3ng as storage driver and things go bad, you may have a lot of unfinished multipart-uploads, that can cost money

Steps to reproduce

Steps to reproduce the behavior:

  1. upload files
  2. kill the storage-users service while it uploads the file to s3
  3. look at the pending multipart-uploads in your s3 server

Expected behavior

have a command / mechanism that deletes unfinished multipart uploads (that also will never be continued)

Actual behavior

unfinished multipart uploads persist forever and actually occupy space on s3 which may create costs.

Additional context

https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/ / https://docs.aws.amazon.com/de_de/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html

-> Depending on the S3 server / service there are also other ways to abort unfinished multipart uploads

wkloucek commented 1 year ago

Actually I would classify this as documentation only when we decide to leave this up to to S3 storage.

@mmattel we could probably just copy multipart cleanup configuration from the oC10 documentation?

wkloucek commented 11 months ago

Looks like we found one S3 implementation that does not support AbortIncompleteMultipartUpload in a bucket lifecycle policy (https://docs.netapp.com/us-en/storagegrid-116/s3/operations-on-buckets.html)

wkloucek commented 11 months ago

Looks like we found one S3 implementation that does not support AbortIncompleteMultipartUpload in a bucket lifecycle policy (https://docs.netapp.com/us-en/storagegrid-116/s3/operations-on-buckets.html)

It's the exact opposite: If a multipart upload is not completed within 15 days, the operation is marked as inactive and all associated data is deleted from the system. (https://docs.netapp.com/us-en/storagegrid-116/s3/operations-on-buckets.html)

stale[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.