PROTEUS (and its submodules) are opening too many files. This is an issue for model runtime but it also means that when many cases of the model are running at once (e.g. in a grid), we can encounter the hard limit on the number of open files.
For example, I am running a grid of 1029 points with 105 processes at a time. After a few minutes of runtime, my user has 49k files open which can be broken down as follows...
45k are associated with Python in some way
9k are associated with scipy
4k are associated with pandas
3k are associated with numpy
3k are associated with l_run_cdf which is due to each SOCRATES call requiring multiple shared libraries
580 are associated with spider
This information can be obtained using the lsof command.
It would be valuable to investigate whether there are resource leaks occurring somewhere. I am sure we can also reduce the number of imported libraries; I can see that scipy is importing its FFT subroutines but they aren't used anywhere in the model.
PROTEUS (and its submodules) are opening too many files. This is an issue for model runtime but it also means that when many cases of the model are running at once (e.g. in a grid), we can encounter the hard limit on the number of open files.
For example, I am running a grid of 1029 points with 105 processes at a time. After a few minutes of runtime, my user has 49k files open which can be broken down as follows...
l_run_cdfwhich is due to each SOCRATES call requiring multiple shared librariesThis information can be obtained using the
lsofcommand.It would be valuable to investigate whether there are resource leaks occurring somewhere. I am sure we can also reduce the number of imported libraries; I can see that scipy is importing its FFT subroutines but they aren't used anywhere in the model.
This might relate to Issue #70.