scraperwiki / twitter-follows-tool

Get users who follow or are following a particular user; tool now defunct, though the code is here for reference.
https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/
Other
5 stars 7 forks source link

Make sure only one get.py runs at once #20

Closed frabcus closed 11 years ago

frabcus commented 11 years ago

There are various failure modes:

e.g. user runs it at same time as cron e.g. deleting dataset while longing running existing cron job going

frabcus commented 11 years ago

Maybe use fcntl.flock

frabcus commented 11 years ago

I tried this as per http://stackoverflow.com/a/5255473

However, it never seems to time out (with the alarm). You get the alarm error, but only when the first process (holding the flock) exits.

I'm mildly worried one process might get stuck, then all the others do too, with no errors - just freezing.

signal.alarm(10)
fd = open("box.json", "r")
print "waiting for lock"
fcntl.flock(fd.fileno(), fcntl.LOCK_EX)
signal.alarm(0)
frabcus commented 11 years ago

Let's leave this for now, until we find the problems.