search

DonggeLiu / Legion

A coverage-based software testing tool
MIT License
29 stars 4 forks source link

"OSError: [Errno 24] Too many open files" inside BenchExec #17

Closed DonggeLiu closed 4 years ago

DonggeLiu commented 4 years ago

Every benchmark quickly encounters the following error when running inside BenchExec:

Traceback (most recent call last): File "./Legion.py", line 1465, in print(main()) File "./Legion.py", line 1298, in main run() File "./Legion.py", line 565, in run mcts() File "./Legion.py", line 656, in mcts traces = simulation(node=node) File "./Legion.py", line 1024, in simulation for mutant in mutants if not FOUND_BUG] File "./Legion.py", line 1024, in for mutant in mutants if not FOUND_BUG] File "./Legion.py", line 1096, in binary_execute_parallel report = execute() File "./Legion.py", line 1048, in execute stderr=sp.PIPE, close_fds=True) File "/usr/lib/python3.6/subprocess.py", line 687, in init errread, errwrite) = self._get_handles(stdin, stdout, stderr) File "/usr/lib/python3.6/subprocess.py", line 1197, in _get_handles c2pread, c2pwrite = os.pipe() OSError: [Errno 24] Too many open files

This is strange and troublesome because:

  1. Outside BenchExec, this error can be mitigated by setting ulimit -Sn 10000 or ulimit -n unlimited, but not sure how to do it inside BenchExec
  2. Not sure why this error did not occur in the competition, as we did not change anything about subprocess after the version we submitted.
  3. I tried calling os.system("ulimit -n unlimited") at the beginning of Legion, the same error occurred
  4. I tried to manually close fds (e.g. instr.stdout.close(), instr.stdin.close(), instr.stderr.close()) and kill the subporcess (instr.kill()), none of them worked.
  5. I tried adding the --filesCountLimit 1024 flag to BenchExec, caused the following error:

    23:27:07 seq-mthreaded/pals_STARTPALS_ActiveStandby.1.ufo.BOUNDED-10.pals.yml 2020-02-04 23:27:07 - ERROR - Exception during run execution Traceback (most recent call last): File "/usr/lib/python3/dist-packages/benchexec/localexecution.py", line 264, in run self.execute(currentRun) File "/usr/lib/python3/dist-packages/benchexec/localexecution.py", line 305, in execute files_size_limit=benchmark.config.filesSizeLimit, File "/usr/lib/python3/dist-packages/benchexec/runexecutor.py", line 790, in execute_run **kwargs File "/usr/lib/python3/dist-packages/benchexec/runexecutor.py", line 955, in _execute files_count_limit, files_size_limit, temp_dir, cgroups, pid File "/usr/lib/python3/dist-packages/benchexec/runexecutor.py", line 620, in _setup_file_hierarchy_limit callbackFn=self._set_termination_reason, File "/usr/lib/python3/dist-packages/benchexec/filehierarchylimit.py", line 52, in init assert os.path.isdir(path) AssertionError

DonggeLiu commented 4 years ago

I am not too sure if this error occurred in the competition, as the result HTML file from the competition is different from the one I generated. They look alike, but mine does not have coverage scores. I recall that we had two HTML files for each experiment, one before validation and one after. I think mine is the former and the competition only released the latter. I don't know if this OSError can get propagated to the latter HTML, I can only confirm that the latter HTML from the competition did report this error on the same benchmarks, it reported OUT OF MEMORY instead.

DonggeLiu commented 4 years ago

Solved by calling ulimit -n N where N should be large enough to allow tons of binary executions but smaller than the local hard limit (from ulimit -n -H)