Powerpool crash: cannot release un-acquired lock

[rango1000]

Any idea how to fix this issue? After 10h of running fine, powerpool crashes showing this error. There is some serious traffic on the stratum server.

Thank you very much, Rango

self.lock.acquire()
  File "/usr/lib/python2.7/threading.py", line 128, in acquire
    rc = self.__block.acquire(blocking)
  File "_semaphore.pyx", line 112, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:3004)
  File "/usr/local/lib/python2.7/dist-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
AssertionError: Impossible to call blocking function in the event loop callback
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gevent_helpers.py", line 127, in alarm_handler
    self.timeout, "".join(traceback.format_stack(frame)))
  File "/usr/lib/python2.7/logging/__init__.py", line 1160, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1267, in _log
    self.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1277, in handle
    self.callHandlers(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1317, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 746, in handle
    self.acquire()
  File "/usr/lib/python2.7/logging/__init__.py", line 697, in acquire
    self.lock.acquire()
  File "/usr/lib/python2.7/threading.py", line 128, in acquire
    rc = self.__block.acquire(blocking)
  File "_semaphore.pyx", line 112, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:3004)
  File "/usr/local/lib/python2.7/dist-packages/gevent/hub.py", line 330, in switch
    switch_out()
  File "/usr/local/lib/python2.7/dist-packages/gevent/hub.py", line 334, in switch_out
    raise AssertionError('Impossible to call blocking function in the event loop callback')
AssertionError: Impossible to call blocking function in the event loop callback
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1658, in shutdown
    h.release()
  File "/usr/lib/python2.7/logging/__init__.py", line 704, in release
    self.lock.release()
  File "/usr/lib/python2.7/threading.py", line 143, in release
    raise RuntimeError("cannot release un-acquired lock")
RuntimeError: cannot release un-acquired lock
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1658, in shutdown
    h.release()
  File "/usr/lib/python2.7/logging/__init__.py", line 704, in release
    self.lock.release()
  File "/usr/lib/python2.7/threading.py", line 143, in release
    raise RuntimeError("cannot release un-acquired lock")
RuntimeError: cannot release un-acquired lock

[icook]

What version are you running? Regular python threads should never be being used in powerpool, and none of these tracebacks include any code from powerpool itself.

[ericecook]

We've had this issue in production (although only once or twice). Restart fixes it - I haven't had time to investigate the cause. Most likely some sort of resource hard limit getting hit

[ericecook]

I'm pretty sure cf989287cdc6623682585fce85fbeb9240dc2e69 will fix this as long as you are running python in optimized mode. Feel free to re-open this if that is not the case