JuliaParallel / ClusterManagers.jl

Other
235 stars 74 forks source link

Add the `ExistingProcessManager` cluster manager #148

Closed DilumAluthge closed 3 years ago

DilumAluthge commented 3 years ago

Closes #147

Summary

This pull request adds the ExistingProcessManager cluster manager. Because this is a non-breaking feature, I have bumped the version number from 0.4.0 to 0.4.1.

This is a resubmission of #147.

Example usage

First, run the following command four times. It can be on the same machine or on different machines.

julia --worker=1234567890abcdef &
Make note of the hosts and port numbers that are printed to stdout. For this example, we will suppose that the hosts and port numbers that were printed to stdout were as follows: Host Port
192.168.1.151 9684
192.168.1.151 9685
192.168.1.151 9686
192.168.1.151 9687

Now, open a Julia session and run the following: (in the workers array, replace the hosts and port numbers with the hosts and port numbers that you received in the previous step)

julia> import ClusterManagers, Distributed

julia> Distributed.cluster_cookie("1234567890abcdef")

julia> workers = [
       ("192.168.1.151", 9684),
       ("192.168.1.151", 9685),
       ("192.168.1.151", 9686),
       ("192.168.1.151", 9687),
       ]

julia> Distributed.addprocs(ClusterManagers.ExistingProcessManager(workers))

Motivation

The idea here is that for whatever reason, you have already manually started the Julia worker processes by doing julia --worker=my_cluster_cookie. Those workers are now running, and you want to add them as processes so that Distributed is aware of them. This is what the ExistingProcessManager cluster manager allows you to do.

Constructors

There are two ways to construct a ExistingProcessManager:

  1. Pass a vector of tuples, where the first element of each tuple is the host, and the second element of each tuple is the port number.
  2. Pass a vector of Distributed.WorkerConfigs.
DilumAluthge commented 3 years ago

I need this functionality in production relatively soon, so I've fleshed this out a bit, added tests and docs, and put it into a dedicated package: https://github.com/bcbi/ExistingProcessManagers.jl

I do still think that this would be useful for users of ClusterManagers.jl. But either way, it is already available in ExistingProcessManagers.jl.

DilumAluthge commented 3 years ago

I'm closing this PR as stale.

The functionality is available in the ExistingProcessManager.jl package (https://github.com/bcbi/ExistingProcessManagers.jl), which is now registered in the General registry.