tsd-s3cmd user experience and issues for large-scale data sync

[ofrei]

I think we experience multiple issues with tsd-s3cmd during large-scale data sync (100GB+), both for import and export.

Some of the issues are missing functionality (#1), but there is more to it. I have an open ticket [rt.uio.no #3797526] tsd-s3cmd fails uploading large file, about intermittent failures. My workaround was to use --multipart-chunk-size-mb=1000, and keep re-running the same command 10+ times until it succeeded.

@denvdm , what's your experience?

[denvdm]

Yep, failed for me too (also using multipart-chunk), but honestly I thought it was ‘just’ due to crappy connection with MMIL, the only one I tried. I ended up using a very cumbersome workaround, spltting into smaller files and working via NIRD. Would indeed be great to get this fixed, given the huge syncing task.

On 13 Mar 2020, at 09:14, Oleksandr Frei notifications@github.com wrote:

I think we experience multiple issues with tsd-s3cmd during large-scale data sync (100GB+), both for import and export.

Some of the issues are missing functionality (#1 https://github.com/norment/tsd_issues/issues/1), but there is more to it. I have an open ticket [rt.uio.no #3797526] tsd-s3cmd fails uploading large file, about intermittent failures. My workaround was to use --multipart-chunk-size-mb=1000, and keep re-running the same command 10+ times until it succeeded.

@denvdm https://github.com/denvdm , what's your experience?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/norment/tsd_issues/issues/34, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACE3BEFGLL24OCS4N3ARR5DRHHTO3ANCNFSM4LG5ILCA.

[ofrei]

Another relevant issue - s3-api-access-keys seem to be missing on new projects. I'm trying to clarify whether we need to request s3-api for each project separately, including white lists of IP addresses.

[ofrei]

Old ticket, I think this was fixed.