AndreasHeger
commented
5 years ago Thanks for reporting this - I will check. I observed something similar when using multiprocessing for concurrency. One of the reason I added the ability to use gevent.
Best wishes, Andreas
On 21/12/2018 18:29, Andreas Tille wrote:
Hi, there is a bug report https://bugs.debian.org/841073 for the Debian packaged version which says that the ruffus test suite consumes an extraordinary amount of memory on the build machine. The bug was filed in 2016 for version 2.6.3 of ruffus where a memory consumption of 18GB was measured. The bug reporter did a new measure for the recent version 2.8.1 he gets 38GB so it became worse. I tried to reproduce the issue and can confirm that if I do
|while true; do grep Committed_AS /proc/meminfo; sleep 1; done |
in a separate xterm while running the build time tests I get something like
|Committed_AS: 9807008 kB Committed_AS: 22324324 kB Committed_AS: 33082052 kB Committed_AS: 33082052 kB Committed_AS: 17073508 kB Committed_AS: 41512736 kB Committed_AS: 45553976 kB Committed_AS: 26653296 kB Committed_AS: 5887392 kB |
while the test suite outputs something like
|test_pool_manager.py::TestExecutionEngines::test_pipeline_fails_with_unknown_manager PASSED [ 25%] test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_gevent_manager <- ../../../../../../../usr/lib/python2.7/unittest/case.py SKIPPED [ 50%] test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_multiprocessing PASSED [ 75%] test_pool_manager.py::TestExecutionEngines::test_pipeline_runs_with_multithreading PASSED [100%] ===================== 3 passed, 1 skipped in 2.50 seconds ====================== pytest -v test_cmdline.py ============================= test session starts ============================== platform linux2 -- Python 2.7.15+, pytest-3.6.4, py-1.7.0, pluggy-0.8.0 -- /usr/bin/python2 cachedir: ../../../../../.pytest_cache rootdir: /home/andreas/debian-maintain/salsa/med-team/build-area/python-ruffus-2.8.1, inifile: collecting ... collected 4 items test_cmdline.py::Test_cmdline::test_argparse PASSED [ 25%] test_cmdline.py::Test_cmdline::test_optparse PASSED [ 50%] test_cmdline.py::Test_cmdline::test_verbose PASSED [ 75%] test_cmdline.py::Test_cmdline::test_verbose_abbreviated_path PASSED [100%] |
Please note: I'm definitely not sure that this is the affected test neither that these are the only tests which push the memory limit way more far than other programs but I confirm that the reporter is right and that the issue can be found somewhere close here. I'd suggest some bisecting to find the actual test to find out the reason. Kind regards, Andreas.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cgat-developers/ruffus/issues/104, or mute the thread https://github.com/notifications/unsubscribe-auth/AEOKYO_ZMQXk4A-sIX1GixKxI6zqE9oxks5u7SiegaJpZM4ZekBb.
eamanu
emollier
Hi, there is a bug report for the Debian packaged version which says that the ruffus test suite consumes an extraordinary amount of memory on the build machine. The bug was filed in 2016 for version 2.6.3 of ruffus where a memory consumption of 18GB was measured. The bug reporter did a new measure for the recent version 2.8.1 he gets 38GB so it became worse. I tried to reproduce the issue and can confirm that if I do
in a separate xterm while running the build time tests I get something like
while the test suite outputs something like
Please note: I'm definitely not sure that this is the affected test neither that these are the only tests which push the memory limit way more far than other programs but I confirm that the reporter is right and that the issue can be found somewhere close here. I'd suggest some bisecting to find the actual test to find out the reason. Kind regards, Andreas.