Closed cousins closed 4 years ago
Hi Steve,
You need to set number of threads and/or processes in constants.py, the current version of ISEScan does not support setting it in command line (I will add this option to ISEScan soon, probably this weekend). I would suggest you to set number of processes you want to use, e.g. nproc = 4 or greater if your computer has 4 or more CPU cores. Setting number of threads in python does not make sense sometimes.
Xie
Hi Xie,
Thanks. I tried changing both nthread and ncpu and it has no effect as far as I can tell. It seems to set it at 4 no matter what.
Best,
Steve
On Tue, Apr 7, 2020 at 3:51 PM Zhiqun Xie notifications@github.com wrote:
Hi Steve,
You need to set number of threads and/or processes in constants.py, the current version of ISEScan does not support set it in command line (I will add this option to ISEScan soon, probably this weekend). I would suggest you to set number of processes you want to use, e.g. nproc = 4 or greater if your computer has 4 or more CPU cores. Setting number of threads in python does not make sense sometimes.
Xie
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/xiezhq/ISEScan/issues/19#issuecomment-610586977, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDNXGMX2HD7QSKQWNJ7DZDRLN73HANCNFSM4MCNPOAQ .
Hi Xie,
It seems that maybe the changes in constants.py are not being incorporated. Do I need to do something to get them to take effect? I tried changing the value for tmpdir as well and that is not working. It is still trying to use /tmp which is too small for what we need. I tried changing it to /scratch.
Thanks,
Steve
On Tue, Apr 7, 2020 at 8:46 PM Stephen Cousins steve.cousins@maine.edu wrote:
Hi Xie,
Thanks. I tried changing both nthread and ncpu and it has no effect as far as I can tell. It seems to set it at 4 no matter what.
Best,
Steve
On Tue, Apr 7, 2020 at 3:51 PM Zhiqun Xie notifications@github.com wrote:
Hi Steve,
You need to set number of threads and/or processes in constants.py, the current version of ISEScan does not support set it in command line (I will add this option to ISEScan soon, probably this weekend). I would suggest you to set number of processes you want to use, e.g. nproc = 4 or greater if your computer has 4 or more CPU cores. Setting number of threads in python does not make sense sometimes.
Xie
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/xiezhq/ISEScan/issues/19#issuecomment-610586977, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDNXGMX2HD7QSKQWNJ7DZDRLN73HANCNFSM4MCNPOAQ .
--
Steve Cousins Supercomputer Engineer/Administrator Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 581-3574 Orono ME 04469 steve.cousins at maine.edu
Hi Steve,
ISEScan.py can only utilize multiple CPU cores for FragGeneScan and HMMER which are two CPU-intensive steps (HMMER search is the speed-limit step in ISEScan) in ISEScan. The nproc parameter in constants.py is not for ISEScan users, it is for some testing codes when I developed ISEScan. I will remove nproc and those testing codes in ISEScan soon to avoid confusing users. I will update ISEScan today to add an optional nthread option for isescan.py to allow user to set number of CPU cores to use, and I will remove nthread and nproc paramters from constants.py accordingly.
ISEScan does not set temporary directory like /tmp/ you mentioned. If you run python3 isescan.py NC_012624.fna proteome hmm ISEScan will then write intermediate files and final results to three directories, proteome/, hmm/, prediction/. The final results are in prediction/, some intermediate/temporary files are written to proteome/ and hmm/. Could you give more details on 'It is still trying to use /tmp which is too small for what we need'? The Linux applications usually use /tmp/ for the temporary files because the environment varaible TMPDIR, TEMP or TMP is usually set to /tmp. So, you may try setting one of these environment varabiles to /scratch: export TMPDIR=/scratch/tmp4steve mkdir -p $TMPDIR
If you have lots of genome sequences (either multiple sequences in a genome or one sequence per genome), you can try running multiple isescan.py parallel on your computer as I described in 'How to run a set of genomes in a row' (https://github.com/xiezhq/ISEScan/issues/13).
Xie
Hi Xie,
I tried changing tmpdir in constants.py to /scratch instead of /tmp since our /tmp directory isn't big enough but ISEScan is still sending errors indicating that /tmp is running out of space. What do we need to do to change from /tmp to /scratch?
Thanks,
Steve
On Wed, Apr 8, 2020 at 9:35 PM Zhiqun Xie notifications@github.com wrote:
Hi Steve,
ISEScan.py cannot utilize multiple cpu cores as it is a python3 applications. The nproc and nthread parameters in constants.py are not for ISEScan users, they are for some testing codes in ISEScan. I will remove them soon.
ISEScan does not set temporary directory like /tmp/ you mentioned. If you run python3 isescan.py NC_012624.fna proteome hmm ISEScan will then write intermediate files and final results to three directories, proteome/, hmm/, prediction/. The final results are in prediction/, some intermediate/temporary files are written to proteome/ and hmm/.
If you have lots of genome sequences (either multiple sequences in a genome or one sequence per genome), you can try running multiple isescan.py parallel on your computer as I described in 'How to run a set of genomes in a row' (#13 https://github.com/xiezhq/ISEScan/issues/13).
Xie
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/xiezhq/ISEScan/issues/19#issuecomment-611278876, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDNXGLPVC55UDKNSUZTCATRLUQ6HANCNFSM4MCNPOAQ .
--
Steve Cousins Supercomputer Engineer/Administrator Advanced Computing Group University of Maine System 244 Neville Hall (UMS Data Center) (207) 581-3574 Orono ME 04469 steve.cousins at maine.edu
Sorry, I didn't get your previous message in my email. I will try setting TEMP, TMP, and TMPDIR to see if that works.
ISEScan has been upgraded to v1.7.2.1, and you can conveniently install it by Bioconda (see https://github.com/xiezhq/ISEScan#Bioconda-install). The ISEScan >= v1.7.2 can set number of threads (GeneFragScan and hmmer called by ISEScan will utilize multiple threads and these two steps are the speed-limited steps in ISEScan) by command option --nthread, for example, isescan.py NC_012624.fna proteome hmm --nthread 2.
Xie
The end of constants.py file has:
What is the way to set nthread for a given run? I have tried:
and
but it just shows:
Thanks,
Steve