Deadlock in very specific combination

[nMustaki]

Hi,

first thanks for Green, I love its speed and colors ! I am trying to test a Flask application, but I get a deadlock if I use together :

• Green (of course)
• Flask-Testing (with subclass unittest and save Flask app context)
• Gevent (both stable and beta)
• pkgutil.walk_packages (to load blueprints automatically)

Everything works fine if I do any of the following :

• switch to nose/pytest/unittest
• uninstall flask-testing
• uninstall gevent
• manually load the blueprints

The deadlock happens after my create_app function (required by flask-testing), and before starting to run (or at least print) the first test. Example output :

#  GEVENT_RESOLVER="ares" green --debug --debug --debug -s1 test/test_blueprints/test_api/test_project/test_post_project.py                                            1 ↵
2015-11-27 23:34:27     DEBUG Attempting to load target 'test/test_blueprints/test_api/test_project/test_post_project.py' with file_pattern 'test*.py'
2015-11-27 23:34:27     DEBUG IGNORED exception: Import by filename is not supported.
2015-11-27 23:34:28     DEBUG Load method: FILE - test/test_blueprints/test_api/test_project/test_post_project.py
2015-11-27 23:34:28     DEBUG Found 1 test for target 'test/test_blueprints/test_api/test_project/test_post_project.py'
2015-11-27 23:34:28     DEBUG Attempting to load target 'test.test_blueprints.test_api.test_project.test_post_project' with file_pattern 'test*.py'
2015-11-27 23:34:28     DEBUG Load method: DOTTED OBJECT - test.test_blueprints.test_api.test_project.test_post_project
2015-11-27 23:34:28     DEBUG Found 1 test for target 'test.test_blueprints.test_api.test_project.test_post_project'
2015-11-27 22:34:28     DEBUG Blueprints imported
^C^CTraceback (most recent call last):
  File "/home/user/dev/envs/massive2/bin/green", line 9, in <module>
    load_entry_point('green==2.2.0', 'console_scripts', 'green')()
  File "/home/user/dev/envs/massive2/local/lib/python2.7/site-packages/green/cmdline.py", line 75, in main
    result = run(test_suite, stream, args, testing)
  File "/home/user/dev/envs/massive2/local/lib/python2.7/site-packages/green/runner.py", line 107, in run
    msg = queue.get()
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib/python2.7/unittest/signals.py", line 36, in __call__
    self.default_handler(signum, frame)
KeyboardInterrupt
Process DaemonlessPoolWorker-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/dev/envs/massive2/local/lib/python2.7/site-packages/green/process.py", line 170, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
  File "/usr/lib/python2.7/unittest/signals.py", line 36, in __call__
    self.default_handler(signum, frame)
KeyboardInterrupt

I've just removed Flask-Testing so the issue is "gone" for me, but maybe this issue could help somebody else someday...

[svetlyak40wt]

Trying to test app with gevent, I have similar problem but with such traceback:

Traceback (most recent call last):
  File "/home/art/.envs/proj/bin/green", line 9, in <module>
    load_entry_point('green==2.2.0', 'console_scripts', 'green')()
  File "/home/art/.envs/proj/local/lib/python2.7/site-packages/green/cmdline.py", line 75, in main
    result = run(test_suite, stream, args, testing)
  File "/home/art/.envs/proj/local/lib/python2.7/site-packages/green/runner.py", line 92, in run
    targets = [(target, manager.Queue()) for target in toParallelTargets(suite, args.targets)]
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 667, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 565, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 175, in Client
    answer_challenge(c, authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 428, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
IOError: [Errno 11] Resource temporarily unavailable

[svetlyak40wt]

BTW, I found this thread on stackoverflow. It is about how badly multiprocessing plays with gevent. There is also an interesing answer from ddotsenko who wrote a plugin for nose to run tests in subprocesses but without multiprocessing and issues with gevent.

[CleanCut]

Sorry for the huge delay.

I'm really not interested in diving into crazy world of multprocessing vs. gevent.

If anyone has fixes they want to propose, I'm happy to help make sure they don't break any existing functionality and get them released, but short of that my answer is going to be: green doesn't play nice with gevent.

[nMustaki]

Hi,

I understand your point of view, but can you at least add a disclaimer in the readme about this fact so fewer people loose time on this ? Thanks !

[CleanCut]

Absolutely. That's a great idea.