tsileo / bakthat

Bakthat is a MIT licensed backup framework written in Python, it's both a command line tool and a Python module that helps you manage backups on Amazon S3/Glacier and OpenStack Swift. It automatically compress, encrypt (symmetric encryption) and upload your files.
http://bakthat.readthedocs.org/en/latest/
MIT License
491 stars 67 forks source link

Does not work with the 'eu-west-1' region #21

Closed ghost closed 11 years ago

ghost commented 11 years ago

I run it:

import bakthat
bakthat_conf = {'access_key': '%KEY1%',
                'secret_key': '%KEY2%',
                's3_bucket': '%my_backet_in_europe%',
                'region_name': 'eu-west-1'}
bakthat.backup("%/home/my/dir/to/backup%", destination='s3', conf=bakthat_conf)

and script print that:

Backing up %/home/my/dir/to/backup%
Password (blank to disable encryption):
Compressing...
Uploading...
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Upload completion: 0%
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "bakthat/__init__.py", line 311, in backup
    storage_backend.upload(stored_filename, outname)
  File "bakthat/backends.py", line 109, in upload
    k.set_contents_from_filename(filename, **upload_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1187, in set_contents_from_filename
    encrypt_key=encrypt_key)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1121, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 827, in send_file
    query_args=query_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 490, in make_request
    override_num_retries=override_num_retries)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 932, in make_request
    return self._mexe(http_request, sender, override_num_retries)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 894, in _mexe
    raise e
socket.error: [Errno 104] Connection reset by peer

But! If I run it:

import bakthat
bakthat_conf = {'access_key': '%KEY1%',
                'secret_key': '%KEY2%',
                's3_bucket': '%my_backet_in_us_east%',
                'region_name': 'us-east-1'}
bakthat.backup("%/home/my/dir/to/backup%", destination='s3', conf=bakthat_conf)

and script print that (and create backup at s3):

Backing up %/home/my/dir/to/backup%
Password (blank to disable encryption):
Compressing...
Uploading...
Upload completion: 0%
Upload completion: 11%
Upload completion: 22%
Upload completion: 33%
Upload completion: 44%
Upload completion: 55%
Upload completion: 66%
Upload completion: 77%
Upload completion: 88%
Upload completion: 100%
{%some_data_about_created_backup%}
tsileo commented 11 years ago

Hi,

It seems weird, have you try to run the backup script multiple times ? Does the size of your backup exceed 10GB ?

I backup everything in eu-west-1 too and I never had this issue, but I will take a look today.

Thanks for your feedback, I keep you updated !

ghost commented 11 years ago

It's a very funny moment, because after that bug-report the backup script began to work normally. Copy size ~ 150 MB. I think the issue can be closed, but to the script should add a debug mode and print errors more informative than python's exception.

tsileo commented 11 years ago

Yes but the exception you got was from boto, I catch some of them but I can't catch them all. I close the issue let me know if you have any other issue or suggestion.

Thanks !

jianying commented 11 years ago

I got same error even with us-east-1 with a file a bit less than 10 gigs... any suggestions?