search

python-hyper / hyper

HTTP/2 for Python.
http://hyper.rtfd.org/en/latest/
MIT License
1.05k stars 191 forks source link

Unable to handle more than 1024 file descriptors: filedescriptor out of range in select() #383

Closed dmarvp closed 5 years ago

dmarvp commented 6 years ago

I'm running a web service that in turn uses hyper to make http 2 calls. As requests started growing I started to get a "Too many files open" error. After increasing the hosts ulimit for number of open files I have started getting the following errors over time:

ValueError: filedescriptor out of range in select() File "hyper/http20/connection.py", line 281, in request self.endheaders(message_body=body, final=True, stream_id=stream_id) File "hyper/http20/connection.py", line 544, in endheaders self.connect() File "hyper/http20/connection.py", line 377, in connect self._send_preamble() File "hyper/http20/connection.py", line 416, in _send_preamble self._recv_cb() File "hyper/http20/connection.py", line 775, in _recv_cb while count and self._sock is not None and self._sock.can_read: File "hyper/common/bufsocket.py", line 71, in can_read read = select.select([self._sck], [], [], 0)[0]

I understand that select() can't handle more than 1024 file descriptors so I'd like to know if it's possible to adapt the can_read and recv methods inside bufsocket.py to use poll instead or some guidance on how to do it myself.

hashcacher commented 5 years ago

Were you able to resolve this?

dmarvp commented 5 years ago

Yes, but the problem seemed not to be on hyper but on a google library I was using (google cloud speech). The problem went away when changing the google library for direct API calls and managing the connection myself. I will close the issue.

parasyte commented 5 years ago

This is still a legitimate bug in Hyper. The issue is not with 1024 simultaneous sockets, it's a limitation of the select syscall that it cannot work with fd >= 1024.

Or in other words, if your application ever opens ~1020 sockets in its entire lifetime (including reopening dead sockets, opening files in the filesystem, and the 3 std streams), you will hit this error condition. The only fix is restarting the process.

We're planning to either create a PR for Hyper to use select.poll instead, or just monkey-patch the library on our end to workaround the issue in the short term.

liminkai commented 4 years ago

Were you able to resolve this?

hashcacher commented 4 years ago

Monkey patching the library to use select.poll was the way to go. Watch out for OS restrictions.