GoogleCloudPlatform / python-compat-runtime

DEPRECATED: gcr.io/google_appengine/python-compat-multicore
Apache License 2.0
22 stars 31 forks source link

gthread workers exit with "socket is not registered" #94

Closed evanj closed 8 years ago

evanj commented 8 years ago

Occasionally I've noticed that the workers exit with the following error message. This might be fixed on this version, since I'm running this using the production python-compat environment. Feel free to close this if this is already known to be fixed or if this is inappropriate way to report issues. Thanks!

[2016-05-10 17:35:06 +0000] [14] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 109, in init_process
    super(ThreadWorker, self).init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.run()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 217, in run
    self.murder_keepalived()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 178, in murder_keepalived
    self.poller.unregister(conn.sock)
  File "/env/local/lib/python2.7/site-packages/gunicorn/selectors.py", line 409, in unregister
    key = super(EpollSelector, self).unregister(fileobj)
  File "/env/local/lib/python2.7/site-packages/gunicorn/selectors.py", line 244, in unregister
    raise KeyError("{0!r} is not registered".format(fileobj))
KeyError: '<socket._socketobject object at 0x7f84e19e7de0> is not registered'
Traceback (most recent call last):
  File "/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 109, in init_process
    super(ThreadWorker, self).init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.run()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 217, in run
    self.murder_keepalived()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 178, in murder_keepalived
    self.poller.unregister(conn.sock)
  File "/env/local/lib/python2.7/site-packages/gunicorn/selectors.py", line 409, in unregister
    key = super(EpollSelector, self).unregister(fileobj)
  File "/env/local/lib/python2.7/site-packages/gunicorn/selectors.py", line 244, in unregister
    raise KeyError("{0!r} is not registered".format(fileobj))
KeyError: '<socket._socketobject object at 0x7f84e19e7de0> is not registered'
[2016-05-10 17:35:06 +0000] [14] [INFO] Worker exiting (pid: 14)
theacodes commented 8 years ago

Thanks for reporting this. Gunicorn will restart any workers that exit, so this is a non-critical issue. I'm not sure why this is happening, but it seems it's not unique to this runtime.

evanj commented 8 years ago

Oops totally true; Sorry, I forgot that gthread is actually part of gunicorn itself, so this is probably a bug for the upstream project? I can report it there if you would prefer.

Agreed: this isn't critical, but I'm betting it kills any in-flight requests, since it causes the worker to restart, so not ideal.

theacodes commented 8 years ago

I'm not 100% on the worker behavior here but I think that in this case it happens when the remote socket is already gone, so the request that gets killed is already cancelled by the client.

Definitely report it upstream and reference this issue.