search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
340 stars 95 forks source link

Could not start transporter 's3' #85

Open patrickfournier opened 13 years ago

patrickfournier commented 13 years ago

This happens in version d1c55b8.

I added

.encode('ascii')

to the four self.settings[] passed to S3BotoStorage, in transporter_s3.py line 31:

        self.storage = S3BotoStorage(
            self.settings["bucket_name"].encode('ascii'),
            self.settings["bucket_prefix"].encode('ascii'),
            self.settings["access_key_id"].encode('ascii'),
            self.settings["secret_access_key"].encode('ascii'),
            "public-read",
            self.__class__.headers
        )
wimleers commented 12 years ago

Could you enable DEBUG level logging and then post the exact error you're getting?

I'd prefer to use .encode('utf-8'), unless Boto requires non-unicode encoding, of course.

j0rd commented 12 years ago

Same problem. Did .encode('utf-8') and now it works. You should commit this.

bjunc commented 12 years ago

Definitely should commit this. The documentation is okay, but doesn't go into errors. It took me over an hour to find this post and add the lines above to fix the error.

As it has been mentioned in other posts, it'd be great to see better documentation on this tool. It's great to get a good grade in school, but if it's going to be used for production sites and corporations, it needs to be a bit more solid...

j0rd commented 12 years ago

@bjunc there are many bugs in fileconveyor as it stands. It's not that great for production in it's current state unfortunately. There are many unicode problems aside from this one. There's a bug where if an exception happens in the main loop, fileconveyor hangs and will not sync files any longer. There's a problem with fileconveyor messing up when it is processing a file which disappears (usually happens with vim temp files for me, and it causes a fatal exception and stops the loop). It's also got a fairly high memory and cpu footprint which make it hard to use on a small VPS. Most likely this is due to the processing filters. You'll also encounter this if during the loop you have a bunch of failed files which continuously get retried.

It needs a little help to get up to speed, but it's pretty much working. A decent python coder with a weekend to spare should be able to figure things out. Unfortunately I never code in Python and aside from simple fixes and reporting bugs, I can't really assist.

It would be good if a couple people with a stake in this project and who knew python forked it and resolved some of the major issues while Wim is busy. I guess that's the good thing about open source.

I'm probably going to switch over to origin pull until some of the issues get resolved.

jimmykuo commented 12 years ago

Definitely should commit this! That should be .encode('ascii') or .encode('utf-8') to avoid the error.