Closed mebel-christ closed 1 year ago
Good catch, thanks for reporting!
Please just feel free to send pull-requests as soon as you have any code! It's always easier to talk about things when there is already some code to reference, even if it isn't the final version or the prettiest solution ;)
In any case, I will try reproducing this. Keep in mind that hitting the bug depends on your local system's file descriptor limit which isn't as tight as yours (1024?) in many systems. That's probably why I have never seen it so far.
Reproduces well for me!
ulimit -H -n 1024
python3 bug.py
# bug.py
import os
import tbot
from tbot.machine import board, connector
class EvalBoard(connector.SubprocessConnector, board.Board):
pass
@tbot.testcase
def test_fds():
tbot.log.message(os.getpid())
for i in range(2000):
with EvalBoard() as b:
tbot.log.message(f"RUN {i}")
b.exec("echo", "Hello tbot!")
if __name__ == "__main__":
test_fds()
[...]
├─RUN 1014
├─RUN 1015
├─RUN 1016
├─RUN 1017
└─Fail. (21.718s)
Traceback (most recent call last):
File "/run/user/1000/Temporary/bug.py", line 21, in <module>
test_fds()
File "/home/rahix/Documents/Development/tbot/tbot/decorators.py", line 62, in wrapped
return tc(*args, **kwargs)
File "/run/user/1000/Temporary/bug.py", line 15, in test_fds
with EvalBoard() as b:
File "/home/rahix/Documents/Development/tbot/tbot/machine/machine.py", line 158, in __enter__
self.ch = self._cx.enter_context(self._connect())
File "/home/rahix/Documents/Development/tbot/tbot/machine/connector/common.py", line 62, in _connect
return channel.SubprocessChannel()
File "/home/rahix/Documents/Development/tbot/tbot/machine/channel/subprocess.py", line 142, in __init__
super().__init__(SubprocessChannelIO())
File "/home/rahix/Documents/Development/tbot/tbot/machine/channel/subprocess.py", line 41, in __init__
self.p = subprocess.Popen(
File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1746, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
Great, I will post the PR ASAP.
Keep in mind that hitting the bug depends on your local system's file descriptor limit which isn't as tight as yours (1024?) in many systems.
TBH I really wonder why this never became a problem up until now. Since you use select() in the SubprocessChannelIO which in turn passes down to the Syscall select that (according to this man page) can only ever handle file-descriptors smaller than 1024.
Lately I did some experiments with tbot powercycling a DUT via SSH which worked like a charm at first but kept failing after a few hundred cycles. After some investigation I found that this seems to occur since tbot does not properly close some file-descriptors. To simplify the problem (and hopefully make it reproducible) I ran tbot with following setup:
config/eval.py
:tc/test_fd.py
:I execute the setup without any additional configuration like this:
newbot -c config.eval tc.test_fd.test_fds
At some point in time (I guess this depends on your machine) this fails with an error message (I saw different kinds of errors, depending on wether I use SubprocessConnector or SSHConnector) like this:I modified the test like this to "freeze" execution at some point in time:
I get this output:
When I inspect the open file-descriptors belonging to the tbot process with
lsof -p 61006
I see the following:It seems like the pseudo-terminals are closed properly (since they get deleted) but the file-descriptors remain open. @Rahix I already found a fix for the described behavior and will happily supply a PR, but could you first try to reproduce my issue and confirm it as a bug? Maybe even I am doing something wrong or unintended with my test as well?