Mysql support - Githubissues

wimleers / fileconveyor

File Conveyor is a daemon written in Python to detect, process and sync files. In particular, it's designed to sync files to CDNs. Amazon S3 and Rackspace Cloud Files, as well as any Origin Pull or (S)FTP Push CDN, are supported. Originally written for my bachelor thesis at Hasselt University in Belgium.

https://wimleers.com/fileconveyor

The Unlicense

341 stars 95 forks source link

Mysql support #146

Open pezholio opened 11 years ago

pezholio commented 11 years ago

I've managed to pick the guts out of @chris-blay's mysql fork, and get it up to date with Master and it seems to work OK. Have also added support for Rackspace UK endpoints, as per @chrisivens' fork too.

Hope this is useful for some peeps. I'm by no means a Python expert, so please feel free to tell me where I might be going wrong!

I'm also hoping to add support for mysql in the Drupal CDN module today, so will shout in the relevant place when that's done.

wimleers commented 11 years ago

Woah! Very nice work! :)

I'll review this PR this weekend :)

Thank you!

pezholio commented 11 years ago

Excellent, cheers! I've also fixed up the Drupal CDN module to support these changes here if you hadn't already seen them.

jarv commented 10 years ago

Any chance this can still be merged into master?

pezholio commented 10 years ago

We've been working with this in our live environment, and we have noticed that the MySQL connections aren't getting closed properly, so we've disabled it for now. Would be great if someone who knew more than me about Python could fix this though!

wimleers commented 10 years ago

I wanted to merge this now, but @pezholio commented a month ago that MySQL connections aren't getting closed… and that's of course very bad. That has the potential to bring down other websites on the same host; SQLite connections don't suffer from that problem.

If somebody with an interest in getting File Conveyor to work with MySQL could fix that, that'd be great.

Sorry guys :(

VeggieMeat commented 10 years ago

I did find that if a file gets added to the synced files table but not dropped from the queue (presumably, not sure that that is exactly what happened, but a DELETE FROM... of affected files stopped the log flood the the files got pushed afterward), the following error occurs:

ERROR - Unhandled exception of type 'NameError' detected, arguments: '("global name 'IntegrityError' is not defined",)'. Traceback (most recent call last): File "fileconveyor/arbitrator.py", line 296, in run self.process_db_queue() File "fileconveyor/arbitrator.py", line 693, in process_db_queue except IntegrityError, e: NameError: global name 'IntegrityError' is not defined

My Python isn't that great, but I know the SQL in the context is an attempt to INSERT when a key for that file name already exists. I'd really love to get this working with MySQL, but I don't have much downtime available at the moment.