Open hariszaf opened 1 year ago
Currently is using the number of threads
from config.yaml
, guess it could be refined using a new parameters instead of the threads
From my observations: each chunk gets a diamond job with config.yml:threads allocated, e.g. on a 32 thread machine each of 5 chunks will each be allocated 32 threads - massively oversubscribing. But I dont think they run concurrently, at least analysis of one chunk didnt seem to start until one of the other 4 chunks had finished. I'm not sure about this the actual behaviour of cwl-runner/the linux scheduler, or just an artefact of the logging. If each chunk is allocated a total_num_of_threads/number_of_chunks number of cpus (32/5 in previous example), all diamond chunk analyses seem to run concurrently.
Diamond asks for 5 * causing overload.
Thorough description of its behavior could benefit the wf.