fetching progress from resource manager shouldn't rely on SSH tunnel

Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services

http://packages.python.org/mrjob/

Other

2.62k stars 586 forks source link

fetching progress from resource manager shouldn't rely on SSH tunnel #2194

Open coyotemarin opened 4 years ago

coyotemarin commented 4 years ago

When we set up an SSH tunnel to the resource manager, we use the tunnel to check the job's progress and log/print it to the user.

Now that we're checking the YARN resource manager for pooling (see #2191) over SSH, we should use the same capability to fetch progress, so it works whether or not the user opens up a tunnel.