search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

SelfCal worker determines total number of cpus not available cpus when unset in input #1507

Open PeterKamphuis opened 1 year ago

PeterKamphuis commented 1 year ago

When in the SelfCal worker the number of cpus is not specified caracal determines the number of cpus through psutil.cpu_count() in these lines:

https://github.com/caracal-pipeline/caracal/blob/29cd2a18905e7c6924f4aac1d9a6943ee37c87b5/caracal/workers/selfcal_worker.py#L283-L287

However psutil.cpu_count() determines the actual number of cpus on the hardware not necessarily the number of cpus caracal can use (see https://psutil.readthedocs.io/en/latest/index.html?highlight=cpu_count#psutil.cpu_count). This can especially lead to trouble when the amount of available cpus is limit by slurm or some other job manager.

The correct way would be (according to psutil):

ncpu = len(psutil.Process().cpu_affinity())

paoloserra commented 3 months ago

@PeterKamphuis any progress with this? Thanks!

PeterKamphuis commented 3 months ago

@paoloserra No, but I shall try again to make some time for it.