search

merenlab / illumina-utils

A library and collection of scripts to work with Illumina paired-end data (for CASAVA 1.7+ pipeline).
GNU General Public License v2.0
89 stars 31 forks source link

avoid oversubscription #32

Closed EricDeveaud closed 1 year ago

EricDeveaud commented 1 year ago

related to https://github.com/merenlab/illumina-utils/issues/31

Eric

meren commented 1 year ago

Hi Eric,

Thank you very much for the PR!

I looked at the code, and I think an approach like this may be a better solution here:

try:
    ncpu_avail = len(os.sched_getaffinity(0))
except AttributeError:
    ncpu_avail = multiprocessing.cpu_count()

Since this is what I get on a Mac system:

>>> import multiprocessing
>>> multiprocessing.cpu_count()
6
>>> import os
>>> os.sched_getaffinity(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'os' has no attribute 'sched_getaffinity'
EricDeveaud commented 1 year ago

agree with this enhancement. sorry I neglected to check on mac os.

regards

Eric

meren commented 1 year ago

Can you still make commits to this branch or would you like me to update it?

semiller10 commented 1 year ago

Thanks a lot for the fix and info, Eric. Looks good after Meren's astute catch.

EricDeveaud commented 1 year ago

I can update, and will

meren commented 1 year ago

Thank you for these changes, Eric! Since @semiller10 also approves, I'm merging the branch. We should perhaps release a new stable version to make it available officially.

I'll try to take care of it next week :)