Since Version 1.9.0 File deletion not working properly anymore

[ghost]

Hello folks,

I'm using Storj Gateway x86_64 version 1.9.0. Since this version upgrade, mass deletion of files is not working properly anymore e.g. using S3 Browser. Bulk deletion simply fails with this version using some tools, I have no Idea why this is happening, but as soon as I have a couple of hundred files in a folder the whole process simply breaks, using the 1.8.x or any other previous version does not have this bug despite that I'm using s3.delete-objects-concurrency: 10000 or any higher or lower value ...

Any Idea?

[amwolff]

Can you try the lowest possible setting, i.e., s3.delete-objects-concurrency: 1?

[ghost]

Just for reference, these are my settings:

access: %STORJ_ACCESS%
minio.access-key: %S3_ACCESS_KEY%
minio.secret-key: %S3_SECRET_KEY%
server.address: 127.0.0.1:7777
advanced: false
website: false
s3.disable-copy-object: true
s3.include-custom-metadata-listing: true
s3.max-keys-exhaustive-limit: 10000000
s3.max-keys-limit: 1000000
s3.min-part-size: 8388608
s3.delete-objects-concurrency: 1
log.level: fatal
log.stack: false

The outcome using these settings is that it's working again. To be honest, I never tried one, at least 100 or so. Thanks for the hint.

[ghost]

If I set s3.delete-objects-concurrency to a value of 10, the API responses with "Slow Down". Seems I'm hitting the limit pretty fast. But using "1" works fine for the moment.

[pwilloughby]

Running multi-object delete requests in parallel does hit the satellite api rate limits quickly https://docs.storj.io/dcs/concepts/limits. Until the satellite supports multi-object deletes (I don't see it on the current roadmap) you could request a rate limit increase as a workaround. https://docs.storj.io/dcs/billing-payment-and-accounts-1/pricing/usage-limit-increases

[ghost]

I noticed that an I was always wondering why I have leftovers while deleting many files. Thanks for pointing this out 👍.

What do you think is a good limit to increase? In short time want to move over to production and I will have at least a couple of ten million files...

Thanks in advance

[pwilloughby]

I think the math would look something like this s3.delete-objects-concurrency * 1/your gateway's latency to the satellite * number of concurrent multi-delete object requests + a fudge factor. Another approach might be to keep requesting double your current limits until you don't see "slow down" responses any more.

[ghost]

I now got my limit upgraded. Let's close the task for the moment. Thanks again for your help and your good advice.