search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
341 stars 95 forks source link

very slow copying files #63

Closed hadsie closed 13 years ago

hadsie commented 13 years ago

I've got fileconveyor setup to sync from my rackspace server to rackspace cloudfiles but it seems to be going bizarrely slow. Right now I'm getting less than 15 files transferred per minute on average. Most of these files are below 250KB. I have a total of approx. 150000 files that make up about 18GB. Normally this should be able to easily transfer in several hours... but at the current rate it will probably take several years :).

I've disabled all of the processors with the exception of "unique_filename.Mtime" in hopes that it would be able to move the files over faster, but no luck.

Normally there won't be more than a couple dozen files uploaded per hour (if that), which would make this speed acceptable. Is there another tool in fileconveyor besides the arbitrator that I should be using to initially copy my files over?

Thanks! Scott

wimleers commented 13 years ago

Apparently there's a problem with the Mosse/Rackspace Cloudfiles code currently in File Conveyor. It can apparently be fixed quite easily. See #17. Could you maybe get in touch with davidseth to get his changes in? :) Anyway, let's continue in #17.

davidseth commented 13 years ago

I just found the code from a couple of years ago. Looking for what I did now...

Cheers,

David