Database file corruption caused by SIGTERM during dump() execution

[dron22]

When a SIGTERM signal (e.g. by hitting Ctrl-C in terminal) is received during dump() execution, the exit is performed immediately which causes corruption of the pickledb database file. This can be very annoying as it might mean the loss of critical data.

Upon execution of load() after a restart, I get this this error:

Traceback (most recent call last):
  File "test.py", line 22, in <module>
    main()
  File "test.py", line 14, in main
    db = get_pickledb()
  File "test.py", line 9, in get_pickledb
    db = pickledb.load(DB_FILEPATH, False)
  File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 34, in load
    return pickledb(location, option)
  File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 41, in __init__
    self.load(location, option)
  File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 49, in load
    self._loaddb()
  File "/home/don/tmp/pickledb_robust/pickledb/pickledb.py", line 186, in _loaddb
    self.db = json.load(open(self.loco, 'rb'))
  File "/usr/lib/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

My pull request will fix this: https://github.com/patx/pickledb/pull/17

[patx]

https://github.com/patx/pickledb/commit/658fdda86abbd1b5a37fb0f9fc678ca145cc25c6

[dron22]

@patx nice, to see this change finally being added to the project :+1:

[patx]

thanks for pointing it out!

[bdeweygit]

@patx and @dron22 the current solution using a signal handler adds a runtime constraint - you can only call pickledb in the main thread because only the main thread is allowed to handle signals. I was trying to use pickledb along with grpc, which spins off side threads to handle rpc requests, and it would not work for that very reason: Exception calling application: signal only works in main thread

In the meantime I made a fork that accepts a new init arg to optionally not set the sig handler.