gevent engine loaded but not seeing "running gevent loop engine"

[richtera]

I am testing on Mac OS and things are working fine... i.e. I see "running gevent loop engine" and my gevent.spawn task (as an example) runs. But I am deploying under uwsgi running under redhat and there it still loads gevent (i.e. uwsgi the module is imported and available) but I am missing the "running gevent loop engine" and my gevent.spawn is obviously not running. I am not quite sure how to figure out how to debug this, since I actually don't see "running gevent loop engine" inside of any of the source code. Where is this message coming from and can I somehow figure out why it's not showing and the event loop not running?

[unbit]

Check the gevent plugin is available in your redhat uwsgi build (uwsgi --plugin-list)

[richtera]

So the plugin is there. Also when I check "gevent" in sys.modules it is correctly loaded when my app initializes (i.e. I don't have to do import gevent) In my code I execute gevent.spawn() during startup to schedule some initialization code. On the Mac this works fine with the same code base. On redhat it ignores the gevent.spawn() call completely although it doesn't throw any exception or anything. I tried to use spawn_later() and spawn_raw() with the same result. On redhat I can execute my app() inline to bootstrap it, but that causes a hang on the Mac because now the gevent loop is somehow associated with the main uwsgi thread and hangs permanently. Unfortunately I still haven't figured out how to debug this other than peppering the code with various print and traceback statements. During one debug session I suddenly got LoopExit exceptions under redhat, but then it went away as well. Something is just preventing the gevent system from initializing properly under redhat, because parts of it are there an loaded.

Here is the output of plugin-list and my startup banner to show versions and so on. uwsgi --plugin-list

* uWSGI loaded generic plugins * gevent nagios rrdtool carbon corerouter fastrouter http ugreen syslog rsyslog logsocket router_uwsgi router_redirect router_basicauth zergpool redislog mongodblog router_rewrite router_http logfile router_cache rawrouter router_static sslrouter cheaper_busyness transformation_tofile transformation_gzip transformation_chunked transformation_offload router_memcached router_redis router_hash router_expires router_metrics transformation_template stats_pusher_socket

* uWSGI loaded request plugins * 0: python 17: spooler 18: symcall 100: ping 110: signal 111: cache 173: rpc --- end of plugins list ---

This is when UWSGI is starting.... * Starting uWSGI 2.0.6 (64bit) on [Thu Dec 4 08:12:43 2014] * compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-4) on 02 December 2014 13:00:44 os: Linux-2.6.32-504.1.3.el6.x86_64 #1 SMP Fri Oct 31 11:37:10 EDT 2014 nodename: ip-10-2-0-217 machine: x86_64 clock source: unix detected number of CPU cores: 8

[richtera]

Some more info. Maybe there is something there: The Mac says the following and then initializes correctly by running the target routine of gevent.spawn() * Starting uWSGI 2.0.8 (64bit) on [Thu Dec 4 08:20:08 2014] * compiled with version: 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54) on 19 November 2014 12:38:38 os: Darwin-14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 nodename: MAC-arichter.home machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 8 current working directory: /Users/andy/prototype detected binary path: /usr/local/bin/uwsgi your processes number limit is 709 your memory page size is 4096 bytes setrlimit(): Invalid argument [core/uwsgi.c line 2622] detected max file descriptor number: 2560

• async cores set to 10 - fd table size: 2560 lock engine: OSX spinlocks thunder lock: disabled (you can enable it with --thunder-lock) uWSGI http bound on 0.0.0.0:5000 fd 8 uwsgi socket 0 bound to TCP address 127.0.0.1:58582 (port auto-assigned) fd 3 Python version: 2.7.8 (default, Oct 31 2014, 15:19:05) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] Python main interpreter initialized at 0x7ff561e00ef0 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 330848 bytes (323 KB) for 10 cores * Operational MODE: threaded *

The redhat box says the following and then just sits there... Could it be because of the pcre support?

* Starting uWSGI 2.0.6 (64bit) on [Thu Dec 4 08:50:50 2014] * compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-4) on 02 December 2014 13:00:44 os: Linux-2.6.32-504.1.3.el6.x86_64 #1 SMP Fri Oct 31 11:37:10 EDT 2014 nodename: ip-10-2-0-217 machine: x86_64 clock source: unix detected number of CPU cores: 8 current working directory: / detected binary path: /usr/local/.../bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! setgid() to 501 setuid() to 501 your processes number limit is 118605 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to UNIX address /usr/local/bridgeway/sites/19998-nosec/uwsgi.sock fd 3 Python version: 2.7.6 (default, Dec 2 2014, 12:59:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] Set PythonHome to /usr/local/bridgeway/sites/19998-nosec/env Python main interpreter initialized at 0x11a2510 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 331008 bytes (323 KB) for 10 cores * Operational MODE: threaded *

[unbit]

Paste your redhat uWSGI config (the --show-config output would be the best thing), that "threaded" operational mode is really wrong there

[richtera]

[uwsgi] socket = /usr/local/bridgeway/sites/19998-nosec/uwsgi.sock chmod-socket = 777 protocol = uwsgi chdir = /usr/local/bridgeway/sites/19998-nosec master = true module = bsiMain.uwsgi callable = app optimize = 1 http-websockets = gevent-monkey-patch = plugins-list = virtualenv = ... uid = bridgeway gid = bridgeway logto = /usr/local/bridgeway/sites/19998-nosec/logs/uwsgi.log logfile-chmod = 644 gevents = 100 process = 2 enable-threads = true smart-attach-daemon = ... attach-daemon = ...

I tried adding threads=10 which is what I use on Mac OS.

[richtera]

I just noticed that the Mac uses 2.7.8 but redhat uses 2.7.6 of python. Could that cause any of these kind of problems. Python that is.

[unbit]

gevents != gevent

process != processes

maybe the problem is only there

(add strict = true to disable automatic placeholdering and to search for other typos)

[richtera]

Genius! gevent = 10 works. The setup files are generated with another script on deployment. I forgot to remove the patches off my eye. Thank you very much.