Pipes not properly closing after pytraj.energy_analysis.lie?

tuckerburgin commented 3 years ago

I am trying to use pytraj.energy_analysis.lie to analyze a large number of short trajectory files. After exactly 506 calls to the function, I am consistently met with the following traceback:

  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/utils/get_common_objects.py", line 314, in inner
    return f(*args, **kwargs)
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/utils/decorators.py", line 11, in inner
    return f(*args, **kwd)
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/analysis/energy_analysis.py", line 224, in lie
    c_dslist, _ = do_action(traj, command, c_action.Action_LIE)
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/analysis/c_action/__init__.py", line 27, in do_action
    with capture_stdout() as (out, _):
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/externals/wurlitzer.py", line 252, in pipes
    with w:
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/externals/wurlitzer.py", line 156, in __enter__
    pipe = self._setup_pipe('stderr')
  File "/home/tburgin/miniconda3/envs/myenv/lib/python3.6/site-packages/pytraj/externals/wurlitzer.py", line 106, in _setup_pipe
    pipe_out, pipe_in = os.pipe()
OSError: [Errno 24] Too many open files

Note that the OS I'm working with has a soft limit on open files set to 1024:

$ ulimit -Sn
1024

So perhaps at a rate of two pipes per function call (stderr and stdour) plus a few other files here and there, that can explain the problem.

It seems that the stdout/stderr pipes set up by wurlitzer.py are not being properly closed at the end of their usefulness. I have attempted find a solution to this issue myself, but with no luck, so I'm submitting this issue in the hopes that someone with more experience working with pipes will be able to resolve it (or else suggest a workaround!)

Thanks!

hainm commented 3 years ago

hi @tuckerburgin, it's a known issue. Please try to follow this to turn it off. Cheers. https://github.com/Amber-MD/pytraj/issues/1498#issuecomment-558360392

tuckerburgin commented 3 years ago

Aha, I see! Thanks for your help. Silly me, I searched old issues for "pytraj.energy_analysis.lie" but not for just "LIE"!

hainm commented 3 years ago

great.

I searched old issues for "pytraj.energy_analysis.lie" but not for just "LIE"!

I could not search the old thread either (wrong keyword I guess) so I have to browse the old issues. :D

Amber-MD / pytraj

Pipes not properly closing after pytraj.energy_analysis.lie? #1561