Support port forwarding in SSHCluster

jacobtomlinson commented 4 years ago

Some users may have SSH access to a machine and want to use SSHCluster with it, but are not able to access ports other than 22 due to firewall restrictions.

It would be nice to be able to forward the scheduler ports locally through the SSH tunnel.

This would require:

Add kwargs to SSHCluster to specify local port forwards
Set up port forward in scheduler object after connection is made (docs)
Set the scheduler address to the locally forwarded port here

Example

>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
...     ["host1", "host2", "host3", "host4"],
...     forward_scheduler_port=8786,
...     forward_dashboard_port=8787
... )
>>> client = Client(cluster)

There is an assumption here that the restrictive firewall rules would not limit worker-scheduler and worker-worker communication.

mrocklin commented 4 years ago

Yeah, I'm in favor. Port forwarding is part of my routine when using Dask, and I always have to look up how to do it on the command line.

On Wed, Jun 10, 2020 at 9:13 AM Jacob Tomlinson notifications@github.com wrote:

Some users may have SSH access to a machine and want to use SSHCluster with it, but are not able to access ports other than 22 due to firewall restrictions.

It would be nice to be able to forward the scheduler ports locally through the SSH tunnel.

This would require:

Add kwargs to SSHCluster to specify local port forwards

Set up port forward in scheduler object https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L170 after connection is made (docs https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.SSHClientConnection.forward_remote_port )

Set the scheduler address to the locally forwarded port here https://github.com/dask/distributed/blob/6ee7f899d1bc686ba4910c08509c7f2e5345506d/distributed/deploy/ssh.py#L206

Example

from dask.distributed import Client, SSHCluster>>> cluster = SSHCluster( ... ["host1", "host2", "host3", "host4"], ... forward_scheduler_port=8786, ... forward_dashboard_port=8787 ... )>>> client = Client(cluster)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/3881, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTDTZ3FOKD6PADPC4RDRV6WJHANCNFSM4N2RCKSQ .

quasiben commented 4 years ago

@drabastomek you may be interested in the PR -- I believe the Azure PR: https://github.com/dask/dask-cloudprovider/pull/67 was using some port forwarding tricks

drabastomek commented 4 years ago

Thanks @quasiben! I believe we finally worked out the PF. I will be checking in the changes today.

moritzaugustin commented 4 years ago

is there an estimated date when this will be finally merged via https://github.com/dask/distributed/pull/3895 ?

i would very much like to test the port forwarding feature in my dask setup (tens of servers) where i currently rely on manual port forwarding management

TomAugspurger commented 4 years ago

@moritzaugustin seems like https://github.com/dask/distributed/pull/3895 has some merge conflicts and test failures. I'm guessing @jacobtomlinson would be fine with you picking up that pull request if you're interested.

dask / distributed

Support port forwarding in SSHCluster #3881