Opening big files leads to Unix EMFILE error

[GirardR1006]

Good morning, I am trying to open multiple files with several hundred of lines, using the following function:

  let launch_solver target formula =
    let open Feather in
    let open Feather.Infix in
    let files = ls target |> collect_lines in
    let times = List.map files
        ~f:(fun x -> let path = target^"/"^x in
             let trimed = cat path |. head (-2) |> collect_stdout in
             echo trimed > path |> run;
             echo formula >> path |> run;
             let now = Caml.Sys.time () in
             process "z3" [path] > devnull |> run;
             let time_facet = (Float.((Caml.Sys.time ()) - now)) in
             time_facet
           )

Here, target is a folder and files has about 273 elements in it, so no big deal. However, the program crashes when calling let trimed = cat path |. head (-2) |> collect_stdout in with the following stacktrace

my_prog: internal error, uncaught exception:
       Unix.Unix_error(Unix.EMFILE, "dup", "")
       Raised by primitive operation at Spawn.spawn in file "src/spawn.ml" (inlined), line 118, characters 6-16
       Called from Spawn.spawn in file "src/spawn.ml", line 125, characters 2-65
       Called from Feather.exec in file "feather.ml", line 147, characters 4-122
       Called from Feather.eval in file "feather.ml", line 165, characters 6-173
       Called from Feather.collect_gen in file "feather.ml", line 288, characters 2-177
       Called from Feather.collect_stdout in file "feather.ml", line 300, characters 33-60
       Called from Dune__exe__Disco.launch_solver.(fun) in file "bin/my_prog/my_prog.ml", line 108, characters 26-65
       Called from Base__List.count_map in file "src/list.ml", line 407, characters 13-17
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.count_map in file "src/list.ml", line 415, characters 47-72
       Called from Base__List.map in file "src/list.ml" (inlined), line 418, characters 15-31
       Called from Dune__exe__Disco.launch_solver in file "bin/my_prog/my_prog.ml", line 106, characters 16-428
       Called from Cmdliner_term.app.(fun) in file "cmdliner_term.ml", line 25, characters 19-24
       Called from Cmdliner.Term.run in file "cmdliner.ml", line 117, characters 32-39
Fatal error: exception Unix.Unix_error(Unix.EMFILE, "dup", "")
Raised by primitive operation at Feather.Unix.dup in file "feather.ml", line 14, characters 12-29
Called from Feather.collect_gen in file "feather.ml", line 290, characters 21-40
Called from Feather.collect_lines in file "feather.ml" (inlined), line 307, characters 25-52
Called from Feather.terminate_child_processes in file "feather.ml", line 442, characters 2-70
Called from Stdlib.at_exit.(fun) in file "stdlib.ml", line 553, characters 62-65
Called from Stdlib.do_at_exit in file "stdlib.ml" (inlined), line 556, characters 20-39
Called from Stdlib.exit in file "stdlib.ml", line 559, characters 2-15
Called from Cmdliner.Term.exit in file "cmdliner.ml" (inlined), line 289, characters 25-72
Called from Dune__exe__Disco in file "bin/my_prog/my_prog.ml", line 184, characters 2-46

According to https://ocaml.org/api/Unix.html, EMFILE error seems to indicate that there are too many open files in my process, however I only open one file at the time. It may come from the "z3" binary, but I am not sure exactly how it may interfere. The problem seems to come from the collect_stdout functions: I tried to replace the head pipe with a custom head function, and it kept crashing. Also, replacing collect_stdout by collect_lines makes the program crash much earlier.

Could you look at it? Thanks in advance for your time.

[Firobe]

There was a file descriptor leak in the spawning process we found with @tmarti2 : can you check if pinning Feather to https://github.com/Firobe/feather.git solves your issue ?

[GirardR1006]

Confirmed that pinning your version solves the issue, thanks again :D

[charlesetc]

Thanks for reporting @GirardR1006 and thanks for the quick fix @Firobe!