JevinD
commented
3 years ago @dnplkndll FYI
Open JevinD opened 3 years ago
JevinD
commented
3 years ago @dnplkndll FYI
rayrutjes
commented
3 years ago I'm having the same issue on a large bucket/space:
Traceback (most recent call last):
File "/usr/bin/s3cmd", line 3092, in <module>
rc = main()
File "/usr/bin/s3cmd", line 3001, in main
rc = cmd_func(args)
File "/usr/bin/s3cmd", line 1887, in cmd_sync
return cmd_sync_remote2local(args)
File "/usr/bin/s3cmd", line 1475, in cmd_sync_remote2local
ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
File "/usr/bin/s3cmd", line 1321, in _download
response = s3.object_get(uri, dst_stream, dst_file, extra_label = seq_label)
File "/usr/lib/python3/dist-packages/S3/S3.py", line 716, in object_get
response = self.recv_file(request, stream, labels, start_position)
File "/usr/lib/python3/dist-packages/S3/S3.py", line 1655, in recv_file
http_response = conn.c.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3/dist-packages/S3/Custom_httplib3x.py", line 53, in httpresponse_patched_begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without responseDid you find any workaround @JevinD ?
fviard
commented
3 years ago @rayrutjes You are also encountering the problem with Digital Ocean Space service?
rayrutjes
commented
3 years ago Yes, also with DO Space. When I reduce the number of targeted files it works with no problem, but if I pick folders with too much files it crashes sometimes in the middle or then at the very start and doesn't even manage to process a single file.
I've switched to rclone and it seems to work though 🤔
matepaiva
commented
3 years ago I had the problem with DO as well. And now even if I try to get only the remaining part, it crashes anyway.
Environment: Digital Ocean Space Bucket: 20,000+ files Size: 6gb
Steps to reproduce: s3cmd sync s3://$AWS_BUCKETNAME/ /tmp/$PGDATABASE-$TODAY/
Result:
Expected Results: All the files get transferred with out any errors.