peterwittek / somoclu

Massively parallel self-organizing maps: accelerate training on multicore CPUs, GPUs, and clusters
https://peterwittek.github.io/somoclu/
MIT License
268 stars 70 forks source link

Restricting core usage #179

Open PaMartini opened 1 week ago

PaMartini commented 1 week ago

Is there a way to specify the cores that are used during training? I am using the Python interface to Somoclu. I tried:

import psutil
import os

cores = [0,1,3]

p = psutil.Process(os.getpid())
p.cpu_affinity(cores)

os.environ["OMP_PROC_BIND"] = 'TRUE'
os.environ["OMP_PLACES"] = ','.join(f'{{{core}}}' for core in cores)

... script that uses Somoclu to train a SOM ...

But this does not work for me ... (I guess the parallelization of Somoclu ignores the set parameters.) (Also, I know that this is possible via taskset, but I would prefer to specify the cores in the python code.)

xgdgsc commented 1 week ago

What if setting a global env variable instead of inside python? Or start a new process with those env variables?

PaMartini commented 5 days ago

Isn't

os.environ["OMP_PROC_BIND"] = 'TRUE'
os.environ["OMP_PLACES"] = ','.join(f'{{{core}}}' for core in cores)

setting a global env variable?

How would you do this: "Or start a new process with those env variables?"

The reason for my request was that i would like to have the cores on which SOM training with Somoclu is run as an input parameter to my python function. If this is not possible due to Somoclus parallelization beeing out of reach for Pyton, I guess the best option would be to use taskset from the commandline.

xgdgsc commented 5 days ago

Like https://stackoverflow.com/a/4453495/1136027