search

jcus0006 / mtdcovabm

Distributed Covid 19 Agent Based Model modelled on Maltese data.
0 stars 0 forks source link

Dask SSHCluster - Start a different number of workers on each host based on a config file #14

Closed jcus0006 closed 6 months ago

jcus0006 commented 11 months ago

The worker_options param of SSHCluster includes a property "n_workers", however, when set, this would start the same number of workers on each host. This assumes that all the nodes within the cluster share the same resources, which is not always the case.

Need to find a way to use a custom configuration file (can be built myself) that instructs Dask to start a different number of workers on each node.

jcus0006 commented 6 months ago

This was achieved by adding "n_workers" as 1, and passing as many instances of the same IP address, as we want workers on the same node.