lockfile

[edsu]

It would be useful for diffengine to establish a lock before running in order to prevent a long running cron job from interfering with a newly started one.

[edsu]

If you see something like this in your cron error log you know that jobs have overlapped:

Traceback (most recent call last):
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 3683, in execute_sql
    cursor.execute(sql, params or ())
sqlite3.OperationalError: database is locked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "diffengine.py", line 460, in <module>
    main()
  File "diffengine.py", line 437, in main
    version = entry.get_latest()
  File "diffengine.py", line 171, in get_latest
    self.save()
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 4985, in save
    rows = self.update(**field_dict).where(self._pk_expr()).execute()
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 3316, in execute
    return self.database.rows_affected(self._execute())
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 2833, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 3690, in execute_sql
    self.commit()
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 3514, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 134, in reraise
    raise value.with_traceback(tb)
  File "/home/ed/.pyenv/versions/diffengine/lib/python3.6/site-packages/peewee.p
y", line 3683, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: database is locked

It would be nice to not rely on the db lock to guarantee that multiple processes don't interfere with each other. Perhaps eventually it should run as a daemon rather than from cron. One thing at a time though :-)

[atomotic]

maybe is enough to run the cron script with flock

0,30 * * * * /usr/bin/flock -xn /tmp/diffengine.lock -c "/usr/local/bin/diffengine /home/ed/.diffengine"

[edsu]

That's a beautiful temporary fix that I need right now, thanks! I would still like to have something in the code though? Or perhaps it should just be a daemon... need to sit with it for a bit I guess.

[ryanfb]

Just as a heads up for any other OS X users, you can find a portable version of flock here. It doesn't support -x so you'll need to call it with e.g. flock -n /tmp/diffengine.lock /usr/local/bin/diffengine /path/to/diffengine_config.

[edsu]

Thanks @ryanfb. Working cross-platform is another argument for doing this in Python.