search

spyoungtech / grequests

Requests + Gevent = <3
https://pypi.python.org/pypi/grequests
BSD 2-Clause "Simplified" License
4.49k stars 331 forks source link

grequests leaks threads #80

Closed f0rk closed 4 years ago

f0rk commented 8 years ago

A long running process (had been running for about 12 days) encountered the following exception:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/grequests.py", line 71, in send
    self.url, **merged_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 330, in send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 562, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 797, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 778, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 570, in create_connection
    for res in getaddrinfo(host, port, 0 if has_ipv6 else AF_INET, SOCK_STREAM):
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 621, in getaddrinfo
    return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
  File "/usr/lib/python2.7/dist-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
    return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
  File "/usr/lib/python2.7/dist-packages/gevent/threadpool.py", line 222, in apply_e
    success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
  File "/usr/lib/python2.7/dist-packages/gevent/threadpool.py", line 159, in spawn
    self.adjust()
  File "/usr/lib/python2.7/dist-packages/gevent/threadpool.py", line 133, in adjust
    self._adjust_step()
  File "/usr/lib/python2.7/dist-packages/gevent/threadpool.py", line 113, in _adjust_step
    self._add_thread()
  File "/usr/lib/python2.7/dist-packages/gevent/threadpool.py", line 142, in _add_thread
    start_new_thread(self._worker, ())
error: can't start new thread
<Greenlet at 0x7f3b72105eb0: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x7f3b6faa83d0>>(stream=False)> failed with error

This process regularly makes requests through grequests and does not do any other work with threads. I'm not sure what to file a bug against, whether it is grequests, urllib3, or gevent.

spyoungtech commented 4 years ago

For the length of time, maybe it's possible you ran out of file descriptors? grequests also has issues with memory buildup over time, so perhaps memory is a concern here, too. If you're not passing in a session object to grequests.get give that a shot.

Since this issue was opened a long time ago, I'm going to go ahead and close it though. Feel free to reopen if you're still available to participate.