search

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.
https://wimleers.com/fileconveyor
The Unlicense
340 stars 95 forks source link

Fileconveyor/CloudFiles transmitting excessive bandwidth per file #89

Closed hrryank closed 12 years ago

hrryank commented 12 years ago

I'm trying to set up File Conveyor to sync files to CloudFiles using the Mosso transporter, but it seems to get stuck transmitting data. It basically pegs the outgoing bandwidth.

I worked down to a test case of syncing 1 small ~20KB image, and even then it was maxing out the bandwidth for an hour. Somtimes it seems to complete, and sometimes not so much. It really seems to peg the bandwidth, and if it is failing and retrying it must be doing so very quickly because monitoring network traffic real-time I see no "dips."

I turned up logging and did not see any sort of trying going on, although if I was trying to sync more files than maxConnections it would keep repeating that there were no open places to send something (which should be normal behavior).

I've tried squeezing some more information out of the logs by putting in some debugging statements, but haven't gotten anywhere with tracking down what's happening.

Any thoughts? I haven't tried this with any other transport, so I can't say whether it's just CloudFiles or not.

wimleers commented 12 years ago

This is a duplicate of #17, which has just been fixed :)