dereneaton / ipyrad

Interactive assembly and analysis of RAD-seq data sets
http://ipyrad.readthedocs.io
GNU General Public License v3.0
70 stars 39 forks source link

Structure nreps 10 for k=[2, 3, …] on 16 Core CPU will only use 10 CPUs for each K #549

Closed kindofausername closed 4 months ago

kindofausername commented 4 months ago

Is there a way to use all CPU Cores for structure so that all ipcluster cores are occupied?

Starting

kpop = [2, 3]
nreps = 10

struct.run(nreps=10, kpop=[2, 3], auto=True)

will first calculate 10 reps of kpop 2 and thereafter 10 reps of kpop 3.

Is there also a way to calculate the higher kpops before the lower once?

kindofausername commented 4 months ago

My mistake arose from using the following code:

## submit N replicates of each test to run on parallel client
kvalues = [5, 4, 3]

for kpop in kvalues:
    struct.run(kpop=kpop, nreps=10, ipyclient=ipyclient)

This resulted in only 10 replicates running at a time for each kpop value in the sequence.

However, when you use the following code without the for-loop, all cores are occupied, allowing for parallel execution across the different kpop values:


struct.run(kpop=[5, 4, 3], nreps=10, ipyclient=ipyclient)

This modification ensures that the function struct.run is called once with a list of kpop values, rather than multiple times with a single kpop value, thereby utilizing all available cores for parallel processing of all specified kpop values simultaneously.