search

tsileo / bakthat

Bakthat is a MIT licensed backup framework written in Python, it's both a command line tool and a Python module that helps you manage backups on Amazon S3/Glacier and OpenStack Swift. It automatically compress, encrypt (symmetric encryption) and upload your files.
http://bakthat.readthedocs.org/en/latest/
MIT License
491 stars 67 forks source link

Connection reset by peer #75

Open gianrubio opened 9 years ago

gianrubio commented 9 years ago

im trying to send big file 4GB using bakthat. Every time i make my backup the bakthat died with Connection reset by peer. Its a boto issue or bakthat?

root@server:/etc/udev/rules.d# bakthat backup /srv Backing up /srv Password (blank to disable encryption): Password confirmation: Compressing... Encrypting... Uploading... Upload completion: 0% Upload completion: 0% Upload completion: 0% Upload completion: 0% Upload completion: 0% Upload completion: 0% Upload completion: 0% Traceback (most recent call last): File "/usr/local/bin/bakthat", line 9, in load_entry_point('bakthat==0.6.0', 'console_scripts', 'bakthat')() File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 649, in main app.run() File "/usr/local/lib/python2.7/dist-packages/aaargh/app.py", line 176, in run return func(_kwargs) File "/usr/local/lib/python2.7/dist-packages/bakthat/init.py", line 362, in backup storage_backend.upload(stored_filename, outname, s3_reduced_redundancy=s3_reduced_redundancy) File "/usr/local/lib/python2.7/dist-packages/bakthat/backends.py", line 110, in upload k.set_contents_from_filename(filename, _upload_kwargs) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1362, in set_contents_from_filename encrypt_key=encrypt_key) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file chunked_transfer=chunked_transfer, size=size) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 750, in send_file chunked_transfer=chunked_transfer, size=size) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 951, in _send_file_internal query_args=query_args File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request retry_handler=retry_handler File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request retry_handler=retry_handler) File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1030, in _mexe raise ex socket.error: [Errno 104] Connection reset by peer

shanx commented 7 years ago

Hi @gianrubio see my patch in #84 I find it to be actually a boto and bakthat issue. Boto could have abstracted uploading large files using multipart and just providing a clean interface (maybe boto 3 does that) but boto does provide a way of doing mulitpart transfers which I've implemented in bakthat in my patch. I know you opened this issue a long time ago in a galaxy far far away, but I hope this will help others too.