search

unbit / uwsgi

uWSGI application server container
http://projects.unbit.it/uwsgi
Other
3.45k stars 686 forks source link

ssh2 inside uWsgi crash #1794

Open estevopaz opened 6 years ago

estevopaz commented 6 years ago

Hi,

Recently I added to our project: from pssh.pssh2_client import ParallelSSHClient

And when the lib is used, uwsgi crash:

!!! uWSGI process 11178 got Segmentation Fault !!!
*** backtrace of 11178 ***
uwsgi(uwsgi_backtrace+0x2a) [0x55ac662735da]
uwsgi(uwsgi_segfault+0x23) [0x55ac662739c3]
/lib/x86_64-linux-gnu/libc.so.6(+0x34f00) [0x7f9f8bf69f00]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(EVP_MD_CTX_clear_flags+0x5) [0x7f9f8c812115]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(EVP_DigestInit_ex+0x1d) [0x7f9f8c80338d]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(HMAC_Init_ex+0xdf) [0x7f9f8c81924f]
/home/hoot/.local/lib/python3.6/site-packages/ssh2/.libs/libssh2-155c95ef.so.1.0.1(+0x1e85c) [0x7f9f6c31a85c]
/home/hoot/.local/lib/python3.6/site-packages/ssh2/.libs/libssh2-155c95ef.so.1.0.1(_libssh2_transport_send+0x43e) [0x7f9f6c32d5a5]
/home/hoot/.local/lib/python3.6/site-packages/ssh2/.libs/libssh2-155c95ef.so.1.0.1(+0x27c35) [0x7f9f6c323c35]
/home/hoot/.local/lib/python3.6/site-packages/ssh2/.libs/libssh2-155c95ef.so.1.0.1(libssh2_session_handshake+0x2e) [0x7f9f6c323e0b]
/home/hoot/.local/lib/python3.6/site-packages/ssh2/session.cpython-36m-x86_64-linux-gnu.so(+0x6bdf) [0x7f9f6b7edbdf]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0xf6) [0x7f9f88292756]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x174dff) [0x7f9f88320dff]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x48e1) [0x7f9f88325eb1]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1749fa) [0x7f9f883209fa]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x13a) [0x7f9f8832950a]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x14e) [0x7f9f882283ae]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f9f882284ac]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyObject_Call+0x48) [0x7f9f88228178]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2f02) [0x7f9f883244d2]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x377) [0x7f9f88329747]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x14e) [0x7f9f882283ae]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f9f882284ac]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyObject_Call+0x48) [0x7f9f88228178]
/home/hoot/.local/lib/python3.6/site-packages/greenlet.cpython-36m-x86_64-linux-gnu.so(+0x35cf) [0x7f9f851325cf]
/home/hoot/.local/lib/python3.6/site-packages/greenlet.cpython-36m-x86_64-linux-gnu.so(+0x2f76) [0x7f9f85131f76]
/home/hoot/.local/lib/python3.6/site-packages/greenlet.cpython-36m-x86_64-linux-gnu.so(+0x3e13) [0x7f9f85132e13]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x22a) [0x7f9f8829288a]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x174dff) [0x7f9f88320dff]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x48e1) [0x7f9f88325eb1]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x377) [0x7f9f88329747]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x14e) [0x7f9f882283ae]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f9f882284ac]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyObject_Call+0x48) [0x7f9f88228178]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1cc102) [0x7f9f88378102]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x75aa) [0x7f9f8e0155aa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f9f8c02bcbf]
*** end of backtrace ***
DAMN ! worker 2 (pid: 11178) died, killed by signal 11 :( trying respawn ...

It happens in local with Debian testing and in our servers with stable.

Thanks for your attention, I appreciate a lot any kind of help. Don't hesitate ask me more details. Best, Estevo.

estevopaz commented 6 years ago

With uwsgi 2.0.17 installed through pip3 --user:

  File "/home/hoot/backend/hoot/jobs/__init__.py", line 34, in get_jobs
    output = ssh_client.run_command('ps -o pid,ppid,lstart,pcpu,rss,cmd -u $(whoami)', use_pty=False)
  File "/home/hoot/.local/lib/python3.6/site-packages/pssh/pssh2_client.py", line 182, in run_command
    encoding=encoding, use_pty=use_pty, timeout=timeout)
  File "/home/hoot/.local/lib/python3.6/site-packages/pssh/base_pssh.py", line 91, in run_command
    self.get_output(cmd, output)
  File "/home/hoot/.local/lib/python3.6/site-packages/pssh/base_pssh.py", line 136, in get_output
    (channel, host, stdout, stderr, stdin) = cmd.get()
  File "src/gevent/greenlet.py", line 654, in gevent._greenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 642, in gevent._greenlet.Greenlet.get
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever
    Hub: <Hub '' at 0x7efe0aab6978 epoll pending=0 ref=0 fileno=10 resolver=<gevent.resolver_thread.Resolver at 0x7efdbeac4240 pool=<ThreadPool at 0x7efdbef30f98 0/4/10 hub=<Hub at 0x7efe0aab6978 thread_ident=0x139629343045376>>> threadpool=<ThreadPool at 0x7efdbef30f98 0/4/10 hub=<Hub at 0x7efe0aab6978 thread_ident=0x139629343045376>> thread_ident=0x7efdfd647700>
    Handles:
[]
xrmx commented 6 years ago

Looks like more than an application issue than an uwsgi one. Have you tried googling the exception message?

estevopaz commented 6 years ago

Yes, I'm fighting with this for days. For libcrypto segfault, I didn't find nothing similar in google. For gevent.exceptions.LoopExit: This operation would block forever it's maybe related to gevent and monkey patch. Initially, I solved moving from pssh.pssh_client import ParallelSSHClient to second ans newest implementation from pssh.pssh2_client import ParallelSSHClient.

So, uwsgi and pssh work both very well separately, but together you see.

xrmx commented 6 years ago

If the gevent exception is "solved" why writing it here?

estevopaz commented 6 years ago

Because when no gevent error, libcrypto segfault, and when no segfault, gevent error.

xrmx commented 6 years ago

Have you tried to do whatever you are doing with pssh and gevent without uwsgi? That may be easier to debug for you.

estevopaz commented 6 years ago

Ok, I can try, I never used gevent outside of uwsgi.

abanichev commented 5 years ago

Hello! I have the same problem with my Flask application. It is only fails when running under uwsgi. If i run it with flask run everything is ok.