Open JannisBush opened 1 year ago
What kernel are you running?
I was on Linux 4.15.0-202-generic, but I updated to Linux 5.4.0-136-generic today and the issue persists.
I installed everything in a fresh operation system and get another similar issue:
>>> a = subprocess.check_output(["python", "bin/redbot_cli", "-a", "-o", "har", "http://example.org/"])
Traceback (most recent call last):
File "/home/exp/spec_test/redbot/bin/redbot_cli", line 74, in <module>
main()
File "/home/exp/spec_test/redbot/bin/redbot_cli", line 66, in main
thor.run()
File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/loop.py", line 116, in run
self._run_fd_events()
File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/loop.py", line 357, in _run_fd_events
self._fd_event(event, fileno)
File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/loop.py", line 190, in _fd_event
self._fd_targets[fd].emit(event)
File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/events.py", line 89, in emit
ev(*args)
File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/tcp.py", line 135, in handle_readable
data = self.socket.recv(self.read_bufsize)
OSError: [Errno 101] Network is unreachable
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', 'bin/redbot_cli', '-a', '-o', 'har', 'http://example.org/']' returned non-zero exit status 1.
Using a download of the latest source, try this:
make cli https://www.mnot.net/
I just realized the errors only occur when using http://
(then it fails almost always and only succeeds roughly every fifth run). When using https://
there is no problem and every call works.
The same is true when using make cli
instead of the old command.
Hm. I'm not seeing that, using:
make cli http://www.mnot.net/
... on a 6.1.8 kernel (arch) using Python 3.10.9.
You're using 5.4.0 and 3.9.something.
I did a quick install of Ubuntu 18.04 LTS with kernel 4.15.0-202-generic and Python 3.9.16, and didn't see any issues.
Same on 20.04 LTS, with a 5.4.0-137-generic kernel and Python 3.9.16.
Is there something unusual about your kernel, network setup, etc.?
The new install is using Ubuntu 22.04 LTS with a 4.19 kernel and Python 3.10.6.
The first system is inside a VM, the second one inside an lxc container. Both run on the same server. Not sure, if there is anything unusual about the network setup.
@JannisBush do you have docker or any other container runtime? Probably dockerized version of redbot-cli could help you
Just checking in - are you still seeing this?
I just checked on both machines (now updated to 6.1.38) and both still crash on HTTP with the respective error messages.
OSError: [Errno 101] Network is unreachable
with python3.10 and inside lxd containerOSError: [Errno 9] Bad file descriptor
with python3.9 and inside a VMSure:
$ python -m thor.http.client http://www.example.com/
<frozen runpy>:128: RuntimeWarning: 'thor.http.client' found in sys.modules after import of package 'thor.http', but prior to execution of 'thor.http.client'; this may result in unpredictable behaviour
*** ERROR: DNS Error (Name or service not known)
Update:
python -m thor.http.client http://www.example.org/
<frozen runpy>:128: RuntimeWarning: 'thor.http.client' found in sys.modules after import of package 'thor.http', but prior to execution of 'thor.http.client'; this may result in unpredictable behaviour
exception calling callback for <Future at 0x7f32d78d3da0 state=finished returned list>
Traceback (most recent call last):
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
callback(self)
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/dns/__init__.py", line 25, in done
cb(ff.result())
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/http/client.py", line 202, in _handle_dns
self._initiate_connection()
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/http/client.py", line 221, in _initiate_connection
tcp_client.connect_dns(
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/tcp.py", line 349, in connect_dns
self.handle_socket_error(socket.error(err, os.strerror(err)))
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/tcp.py", line 376, in handle_socket_error
self.handle_conn_error(err_type, err_id, err_str)
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/tcp.py", line 389, in handle_conn_error
self.unregister_fd()
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/loop.py", line 57, in unregister_fd
self._loop.unregister_fd(self._fd)
File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/loop.py", line 339, in unregister_fd
self._epoll.unregister(fd)
FileNotFoundError: [Errno 2] No such file or directory
Other container with python3.11:
python -m thor.http.client http://www.example.org/
<frozen runpy>:128: RuntimeWarning: 'thor.http.client' found in sys.modules after import of package 'thor.http', but prior to execution of 'thor.http.client'; this may result in unpredictable behaviour
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/http/client.py", line 660, in <module>
test_client(sys.argv[1].encode("ascii"), sys.stdout.buffer.write, sys.stderr.write)
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/http/client.py", line 654, in test_client
run()
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/loop.py", line 116, in run
self._run_fd_events()
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/loop.py", line 357, in _run_fd_events
self._fd_event(event, fileno)
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/loop.py", line 190, in _fd_event
self._fd_targets[fd].emit(event)
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/events.py", line 89, in emit
ev(*args)
File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/tcp.py", line 153, in handle_writable
sent = self.socket.send(data)
^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 101] Network is unreachable
I also updated to python3.12 and the original issue persists:
$ make cli http://www.mnot.net/
PYTHONPATH=.:./.venv/bin ./.venv/bin/python redbot/cli.py http://www.mnot.net/
Traceback (most recent call last):
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 153, in handle_writable
sent = self.socket.send(data)
^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/data/spec_test/redbot_new/redbot/cli.py", line 72, in <module>
main()
File "/data/data/spec_test/redbot_new/redbot/cli.py", line 64, in main
thor.run()
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 116, in run
self._run_fd_events()
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 357, in _run_fd_events
self._fd_event(event, fileno)
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 190, in _fd_event
self._fd_targets[fd].emit(event)
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/events.py", line 89, in emit
ev(*args)
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 158, in handle_writable
self._handle_close()
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 202, in _handle_close
self._close()
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 208, in _close
self.unregister_fd()
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 57, in unregister_fd
self._loop.unregister_fd(self._fd)
File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 339, in unregister_fd
self._epoll.unregister(fd)
OSError: [Errno 9] Bad file descriptor
Makefile:56: recipe for target 'cli' failed
make: *** [cli] Error 1
I used redbot for a while on mac with python 3.9 and had no issues. Today, I tried it out an an Ubuntu 18 machine (python 3.9) and it only works in roughly every third try. The other runs crash with two different errors:
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data/data/spec_test/redbot/bin/redbot_cli", line 74, in
main()
File "/data/data/spec_test/redbot/bin/redbot_cli", line 66, in main
thor.run()
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 116, in run
self._run_fd_events()
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 357, in _run_fd_events
self._fd_event(event, fileno)
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 190, in _fd_event
self._fd_targets[fd].emit(event)
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/events.py", line 89, in emit
ev(*args)
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 158, in handle_writable
self._handle_close()
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 202, in _handle_close
self._close()
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 208, in _close
self.unregister_fd()
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 57, in unregister_fd
self._loop.unregister_fd(self._fd)
File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 339, in unregister_fd
self._epoll.unregister(fd)
OSError: [Errno 9] Bad file descriptor
exception calling callback for <Future at 0x7ff97bc8d340 state=finished returned list> Traceback (most recent call last): File "/home/jannis/.pyenv/versions/3.9.6/lib/python3.9/concurrent/futures/_base.py", line 329, in _invoke_callbacks callback(self) File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/dns/init.py", line 25, in done cb(ff.result()) File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/http/client.py", line 201, in _handle_dns self._initiate_connection() File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/http/client.py", line 220, in _initiate_connection tcp_client.connect_dns( File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 349, in connect_dns self.handle_socket_error(socket.error(err, os.strerror(err))) File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 376, in handle_socket_error self.handle_conn_error(err_type, err_id, err_str) File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 389, in handle_conn_error self.unregister_fd() File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 57, in unregister_fd self._loop.unregister_fd(self._fd) File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 339, in unregister_fd self._epoll.unregister(fd) FileNotFoundError: [Errno 2] No such file or directory