search

lukasmu / nextcloud-s3-to-disk-migration

Script for migrating Nextcloud primary storage from S3 to local storage
MIT License
20 stars 6 forks source link

Concurrent Objects retrieving #2

Open solracsf opened 3 years ago

solracsf commented 3 years ago

I suggest to implement this class (or similar) in the script to speed-up large instances migration. https://codesamplez.com/programming/amazon-s3-get-multiple-objects-php

lukasmu commented 3 years ago

I think that "just" using the linked class will blow up the memory... There are probably ways to work around this by implementing some chunking logic...

But: PHP does not really have multi-processing/multi-threading capabilities. Instead of trying to implement some multi-threading functionalities from scratch (or using something like https://github.com/krakjoe/pthreads) it would probably make more sense to just port the transfer script (or at least the download part of it) to another programming language with proper multi-threading support (say Python).

Personally I do not really use this transfer script frequently and therefore I am also not planning to make such a modification at the moment. But if you think that this would be useful and if you have time to develop it, please feel free to make a pull request!

solracsf commented 3 years ago

No problem, let this here, if i find the time (or someone else) maybe this will be implemented. (for an instance with 1,3M objects, single thread, it really hurts 🐘 )