search

FredHutch / motuz

Motuz - A web based infrastructure for large scale data movements between on-premise and cloud
MIT License
104 stars 12 forks source link

rclone creates an ugly container for large objects #152

Open dirkpetersen opened 5 years ago

dirkpetersen commented 5 years ago

This is annoying

image

in swift commander we used the convention .segments-[container] so the segment container would be hidden.

I created a pull request that was too simple 4 years ago: https://github.com/ncw/rclone/pull/188

someone else brought this forward https://forum.rclone.org/t/support-segment-container-options-in-rclone-swift/311

I think we need to propose setting segment container as env var or cmdline option

dirkpetersen commented 5 years ago

Looking at this a bit more using a different container just for the segments is now really outdated.

Swiftstack (and also cyberduck) uses a virtual hidden folders called .segments/ at the root of the same container the object is created in.

image

Using the same container has the benefit that if you look at the container size it will always show the true size of all objects in the container. You don't have to add up the sizes from 2 containers. Also i you delete a container you can be sure that all objects are also deleted and not just the SLO manifest

dirkpetersen commented 5 years ago

Swiftstack also set's the default segment size to 500MB and since they have the experience rclone should do this as well. We need to create a pull request to rclone that implements this change as a command line option e.g. --segments-local or something like that.