Grequests use of `monkey.patch_all(...)` breaks my application

[jaytaylor]

The aggressive python stdlib monkey patching originating from https://github.com/kennethreitz/grequests/blob/master/grequests.py#L21 causes a lot of breakage throughout my app. The degree of monkey patching causes the redis client to error out which in turn then breaks all async celery processing.

I really like the grequests lib, but cannot use it if it's going to make things that used to work fine not work anymore.

  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/billiard/pool.py", line 1426, in safe_apply_callback
    fun(*args)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 347, in on_success
    self.acknowledge()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/celery/worker/job.py", line 453, in acknowledge
    self.on_ack(logger, self.connection_errors)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 100, in ack_log_error
    self.ack()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/base.py", line 95, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 511, in basic_ack
    self.qos.ack(delivery_tag)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 127, in ack
    self._remove_from_indices(delivery_tag).execute()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1919, in execute
    return execute(conn, stack, raise_on_error)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1811, in _execute_transaction
    self.parse_response(connection, '_')
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 1882, in parse_response
    self, connection, command_name, **options)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/client.py", line 387, in parse_response
    response = connection.read_response()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 307, in read_response
    response = self._parser.read_response()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 105, in read_response
    response = self.read()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/redis/connection.py", line 90, in read
    return self._fp.readline()[:-2]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 392, in recv
    self._wait(self._read_event)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/socket.py", line 298, in _wait
    self.hub.wait(watcher)
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 341, in wait
    result = waiter.get()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/Users/jay/my-app/venv/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
LoopExit: This operation would block forever

Additional references pertaining to gevent monkey patching issues:

• http://stackoverflow.com/questions/13040458/why-is-gevent-spawn-different-than-a-monkeypatched-threading-thread
• https://github.com/jrief/django-websocket-redis/issues/3
• http://bighow.net/5332403-Why_is__gevent_spawn__different_than_a_monkeypatched__threading_Thread____.html
• http://stackoverflow.com/questions/9192539/using-gevent-monkey-patching-with-threading-makes-thread-work-serially
• http://pastebin.com/TPYMvuVr

[dmk23]

Is there a workaround yet to prevent monkey-patching of redis by grequests?

I am seeing exactly the same error with grequests/ redis combination...

[jaytaylor]

FWIW, my workaround was to stop using grequests and use requests + multiprocessing instead.

On Mon, Sep 1, 2014 at 2:56 PM, dmk23 notifications@github.com wrote:

Is there a workaround yet to prevent monkey-patching of redis by grequests?

I am seeing exactly the same error with grequests/ redis combination...

— Reply to this email directly or view it on GitHub https://github.com/kennethreitz/grequests/issues/55#issuecomment-54093836 .

[piotr-dobrogost]

You could try using https://pypi.python.org/pypi/requests-futures

[jairhenrique]

@jaytaylor :+1: I have the same issue, but with Celery + SQS.

[jmoz]

I also have this prob when using redis AND without.

I recently tried reimplementing grequests as I saw a github post about the LoopExit being fixed in gevent 1.0.1. Well, it's not.

My django app is deployed on AWS elasticbeanstalk. As I said earlier the prob happens with redis, so I eliminated redis from my code and still get:

File "/opt/python/current/app/bitfolio/report/fetcher.py" in get_results_v3
  174.         responses = grequests.map(rs)
File "/opt/python/run/venv/lib/python2.7/site-packages/grequests.py" in map
  111.     gevent.joinall(jobs)
File "/opt/python/run/venv/lib/python2.7/site-packages/gevent/greenlet.py" in joinall
  400.         wait(greenlets, timeout=timeout)
File "/opt/python/run/venv/lib/python2.7/site-packages/gevent/hub.py" in wait
  645.         return list(iwait(objects, timeout))
File "/opt/python/run/venv/lib/python2.7/site-packages/gevent/hub.py" in iwait
  598.             item = waiter.get()
File "/opt/python/run/venv/lib/python2.7/site-packages/gevent/hub.py" in get
  568.                 return self.hub.switch()
File "/opt/python/run/venv/lib/python2.7/site-packages/gevent/hub.py" in switch
  331.         return greenlet.switch(self)

Exception Type: LoopExit at /api/v3/report
Exception Value: This operation would block forever

[dmk23]

With some experimentation I found a possible solution.

First of all, I did a search and found several reports that Redis is able to function just fine with monkey-patched Gevent, without generating any errors like we've seen: http://stackoverflow.com/questions/10928481/redis-py-with-gevent http://stackoverflow.com/questions/10656953/redis-gevent-poor-performance-what-am-i-doing-wrong/10663498#10663498

Then, looking at the difference I noticed that the successful examples use monkey.patch_all(), while grequests does monkey.patch_all(thread=False, select=False) . So apparently Redis has to be blocking because sockets are patched, while threads/selects are not (not sure why).

My workaround was forcing monkey.patch_all()before importing grequests into my application. So far this fix passed the simple tests, but I want to share this solution for community review.

Question: Is there a reason grequests does not monkey patch everything? Could it break because threads / selects are patched? At the minimum, I'd like to validate that there won't be problem in my current setup but ideally grequests should allow to customize monkey patch args.

[humungasaurus]

I tried @dmk23 's above to solve the celery/sqs/django use case, but to no avail:

DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140389787158272 and this is thread id 140389632302128.

Has anyone else solved for this? I'll continue to investigate (or give up and stop using grequests).

[vlsd]

This also seems to be what breaks my fabric app. I suspect the close_fds option in gevent's Popen is somehow busted, since it works fine with the standard Python version of Popen.

Is there a way to disable the aggressive monkey patching or pass an option about how selective it is?

[lingxiaoyang]

I was having the exact same problem with celery(3.1.25)+redis: LoopExit: This operation would block forever.

My workaround inspired by @dmk23:

from gevent import monkey
import socket
import grequests
reload(socket)  # immediately restore the patch
.....
@task(...)
def my_celery_task(...):
    ......   # do some stuff
    # when I need grequests
    monkey.patch_socket()  # repatch
    ...
    # grequests has done its job
    reload(socket)
    .....  # continue to do other stuff

This workaround also solved the problem that Celery task status was not updated while running (no RECEIVED->STARTED), which is perhaps related to redis broker.

Later I had another problem with Google Search API due to the patched ssl module. I did the same: reload(ssl) and monkey.patch_ssl() then the error was fixed.

If you have similar problems, have a look at which 'native' modules it's referencing in the stack trace. Doing a reload and repatch may solve these problems.

But still, I'm not sure if this is a community-verified solution. Perhaps a better solution lies in grequest itself: see #8 .

[michaeljohnbarr]

@lingxiaoyang - This worked for me in a management command in Django. It completely disabled paramiko's SFTP for us. I stumbled upon the above post and it saved us from having to remove grequests from our project. THANK YOU.

[lingxiaoyang]

@michaeljohnbarr You're welcome!

[jmduke]

@lingxiaoyang your solution saved me a number of cluster headaches. thanks for the help :)

[pavan-blackbuck]

Is there any solution for this from the package owner? @kennethreitz

[nicbou]

@kennethreitz, why close the ticket without a solution? As far as I know, this is still an issue.