search

zhangrengang / TEsorter

TEsorter: an accurate and fast method to classify LTR-retrotransposons in plant genomes
https://doi.org/10.1093/hr/uhac017
GNU General Public License v3.0
90 stars 20 forks source link

possible overcommit #59

Open EricDeveaud opened 1 month ago

EricDeveaud commented 1 month ago

Hello,

number of processors to use is either hardcoded (4, 8) either set using multiprocessing.cpu_count()

problem is that multiprocessing.cpu_count() returns the number of available cpu, but this is not the same as the number of cpu available to the process. For example, you can run in a taskset context or a batch scheduler like slurm.

see:

$ nproc
96
$ taskset -c 1 nproc
1
$ taskset -c 1 python3 -c "import multiprocessing; print(multiprocessing.cpu_count())"
96

I would suggest to use len(os.sched_getaffinity(0)) instead of multiprocessing.cpu_count()

$ python3 -c "import os; print(len(os.sched_getaffinity(0)))"
96
$ taskset -c 1 python3 -c "import os; print(len(os.sched_getaffinity(0)))"
1

regards

Eric

zhangrengang commented 1 month ago

Hi Eric, thanks for your review and suggestion. I have changed it to len(os.sched_getaffinity(0)).