Yelp / doloop

Task loop for keeping things updated
Other
10 stars 10 forks source link

use sqlalchemy to support all the databases #2

Closed coyotemarin closed 8 years ago

coyotemarin commented 13 years ago

doloop currently only works with MySQL. It would be great to use sqlalchemy and be able to use any of the databases it supports.

To go to production, this change would need:

coyotemarin commented 8 years ago

Not going to do this. doloop is actually kind of a poor fit for relational databases, and it gets by with explicit table locks. The reason we use MySQL is because it's ubiquitous, not because of specific performance characteristics.

If and when someone manages to use doloop the the point of a performance bottleneck (highly parallelized, short tasks), might make more sense to custom-build a datastore and loosen the semantics about which IDs you get() first (so you can just have a "pool" of ready IDs which you hand out in parallel).