search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

Batch execusion: filedescriptor out of range in select #434

Open jonfrs opened 2 years ago

jonfrs commented 2 years ago

When I run a loop for collect in different pages, a specific request, after ± 200 loads show this error:

Exception in thread ServerThread (('127.0.0.1', 41075)):
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/jonathan/Projetos/PJ1/venv/lib/python3.7/site-packages/seleniumwire/thirdparty/mitmproxy/master.py", line 36, in run
    self.server.serve_forever()
  File "/home/jonathan/Projetos/PJ1/venv/lib/python3.7/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py", line 650, in serve_forever
    r, w_, e_ = select.select([self.socket], [], [], poll_interval)
ValueError: filedescriptor out of range in select()

I need restart wsl or, in jupyter, reset the session to continue.

wkeeling commented 2 years ago

Thanks for raising this and sorry for the delay in replying.

It looks as though you've hit the maximum open file descriptor limit on your machine. When you start each loop, are you creating a new webdriver instance - i.e. are you calling webdriver.Chrome() at the beginning of each loop? If so, are you closing the browser with webdriver.quit() at the end of each loop? The quit() method will ensure that the Selenium Wire server is properly shutdown behind the scenes.

dabula-s commented 2 years ago

@wkeeling even if I call .quit(), after a day stable working this problem occurs. 

[Selenium Wire Proxy Server] 127.0.0.1:36970: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py", line 493, in do_ssl_handshake
    ssl_connection.do_handshake()
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 1828, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 1541, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/server.py", line 113, in handle
    root_layer()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/modes/http_proxy.py", line 23, in __call__
    layer()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/tls.py", line 285, in __call__
    layer()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/http1.py", line 100, in __call__
    layer()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/http.py", line 206, in __call__
    if not self._process_flow(flow):
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/http.py", line 288, in _process_flow
    return self.handle_upstream_connect(f)
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/http.py", line 254, in handle_upstream_connect
    return layer()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/http.py", line 103, in __call__
    layer()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/tls.py", line 278, in __call__
    self._establish_tls_with_client_and_server()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/tls.py", line 358, in _establish_tls_with_client_and_server
    self._establish_tls_with_server()
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/server/protocol/tls.py", line 445, in _establish_tls_with_server
    self.server_conn.establish_tls(
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/connections.py", line 295, in establish_tls
    self.convert_to_tls(cert=client_cert, sni=sni, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py", line 393, in convert_to_tls
    do_ssl_handshake(sock, self.connection)
  File "/usr/local/lib/python3.9/site-packages/seleniumwire/thirdparty/mitmproxy/net/tcp.py", line 495, in do_ssl_handshake
    rd, _, _ = select.select([sock], [], [], sock.gettimeout())
  File "/usr/local/lib/python3.9/site-packages/gevent/select.py", line 175, in select
    sel_results = _original_select(rlist, wlist, xlist, 0)
ValueError: filedescriptor out of range in select()
dabula-s commented 2 years ago

hm, may be it's not selenium-wire problem.

dabula-s commented 2 years ago

https://github.com/mitmproxy/mitmproxy/issues/4514

wkeeling commented 2 years ago

Selenium Wire still uses the old mitmproxy core, so this will likely continue to affect Selenium Wire (there's no plans to upgrade the proxy core at the present time). I'll see if there's anything we can do.