nemein / com_meego_planet

Social news aggregator
MIT License
11 stars 3 forks source link

Parallelize feed fetching and score calculation #1

Open bergie opened 13 years ago

bergie commented 13 years ago

Now we only fetch one feed at a time and import. Feed importing would pass through a lot faster if we had multiple worker processes handling it.

indeyets commented 13 years ago

but proper implementation of this would require some kind of queueing solution. do we have a bug for that?

We need a queue of tasks, with per-task locking and unlocking (if unsuccessful). then we can have multi-process dequeuer, which is started by cron each minute

bergie commented 13 years ago

Queue implementation would indeed make this easier. Would you like to propose how it would work?