search

tether / roach

A very adaptable web crawler framework. Impossible to kill.
Other
7 stars 1 forks source link

queue finished (and failed) #20

Closed bredele closed 10 years ago

bredele commented 10 years ago

Roach has now 2 queue:

Do you think we should have a finished/failed queue ? Once a job is finished we removed it from the active queue and put it into the finished queue. We could also set a each step a status fields (pending, active, finished, failed) just in order to keep some logs.

bredele commented 10 years ago

It's done. Every time a job trigger a stop signal, roach will remove it from the active queue (and pending just to be sure) and add it to a 'finished' queue.