how to run parallel computation on remote clusters requiring ssh authentication

I want to run a parallel function on remote clusters yet i only manage to run it on my local cluster

I have 3 different clusters with 3 different host addresses. i tried to use the pathos package to run the computation on all 3 clusters together. Yet i only manage to use only the local cluster. all 3 clusters require SSH authentication with user_name and password. I didn't find a way to pass to the pathos api the user name and password of the remote clusters.

I even tried to open a tunnel from localhost:special_port to one of the remote clusters and run the pathos on server = localhost:special_port but it does not working ( pathos still runs only on local cluster).

what is the right way to do the operation described above?

from pathos.pools import ParallelPool as ppool
import sshtunnel
import paramiko

    with sshtunnel.open_tunnel(
            (remote_host, 22),
        ssh_username=ssh_username,
        ssh_password=ssh_password,
        remote_bind_address=(remote_host, 22),
        local_bind_address=('', 12345)) as tunnel:
        client = paramiko.SSHClient()
        client.load_system_host_keys()
        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        client.connect("127.0.0.1", 12345, username=ssh_username, password=ssh_password)
p = ppool(processes=80, servers=("132.67.140.121:22",), )

        x = [i for i in range(10000)]
        print(p.servers)
        y = p.amap(inc, x)

        z = y.get()
        client.close()

def inc(x):
    time.sleep(0.1)
    return x**2

uqfoundation / pathos

how to run parallel computation on remote clusters requiring ssh authentication #170