ngrok vs Argo tunnel

[demotomohiro]

I recently added Argo Tunnel support to remocolab that uses free version. I compared latency and bandwidth of ngrok and Argo Tunnel. Latency vary over time and average latencies of them were almost same. But bandwidth of argo tunnel was always better than ngrok. These result can be different from your case.

Current remocolab uses ngrok in default (ngrok is used when you didn't specify tunnel argument). But I'm planning to change the default to argo tunnel if it works well in most of users. Do you think default value of tunnel argument should be argo tunnel or ngrok?

I think argo tunnel is easier to use than ngrok for most of users because it doesn't require creating account. But it requires copy of cloudflared in client machine.

And ngrok seems less secure than argo tunnel. ngrok gives fixed hostname with random port. Tools like nmap might find your ssh server and tools likes ssh-brute might find out colab user's password if you didn't use public key auth and changed the password to weak one. Argo tunnel generates host name with random 4 words and finding your host name with brute force attack would be very hard.

How to measure ssh latency

My code is based on hzpc-joostk's answer in https://serverfault.com/questions/807910/measure-total-latency-of-ssh-session . I added time.sleep(6) because it reported much longer time without it. Follwing command create a file deleteme in current directory. You can delete it after the measurement.

Command to measure ssh latency when Argo tunnel is used: (change your-host-name-here.trycloudflare.com part to your argo tunnel hostname)

python -m timeit -n 10 --setup "import subprocess, time; p = subprocess.Popen(['ssh', '-oUserKnownHostsFile=deleteme', '-oProxyCommand=./cloudflared access ssh --hostname %h', 'colab@your-host-name-here.trycloudflare.com', 'cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0); time.sleep(6)" "p.stdin.write(b'z'); assert p.stdout.read(1) == b'z'"

Command to measure ssh latency when ngrok is used: (Change -p argument and 0.tcp.jp.ngrok.io part according to your case)

python -m timeit -n 10 --setup "import subprocess, time; p = subprocess.Popen(['ssh', '-oUserKnownHostsFile=deleteme', '-p 12345', 'colab@0.tcp.jp.ngrok.io', 'cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0); time.sleep(6)" "p.stdin.write(b'z'); assert p.stdout.read(1) == b'z'"

Results of argo tunnel: (I did factory reset runtime each time I run the command) 3:00 10 loops, best of 5: 195 msec per loop 10 loops, best of 5: 70 msec per loop 10 loops, best of 5: 73 msec per loop 15:00 10 loops, best of 5: 71.4 msec per loop 10 loops, best of 5: 128 msec per loop 21:00 10 loops, best of 5: 241 msec per loop 10 loops, best of 5: 287 msec per loop 9/22 0:00 10 loops, best of 5: 209 msec per loop

average latency: 159.3

Result of ngrok latency: 9/21 3:00 10 loops, best of 5: 56.9 msec per loop 10 loops, best of 5: 179 msec per loop 10 loops, best of 5: 244 msec per loop 15:00 10 loops, best of 5: 183 msec per loop 10 loops, best of 5: 115 msec per loop 21:00 10 loops, best of 5: 202 msec per loop 10 loops, best of 5: 116 msec per loop 9/22 0:00 10 loops, best of 5: 183 msec per loop

average latency: 159.8625

How to measure ssh bandwidth

I used scp to measure upload/download bandwidth. Follwing python3 code generates 64MB random content file used in scp:

import random
with open('testrandom', 'wb') as f:
  for i in range(1024*1024*8):
    f.write(random.getrandbits(64).to_bytes(8, 'big'))

scp for Argo tunnel:

# Uploading
scp -o UserKnownHostsFile=deleteme -oProxyCommand="./cloudflared access ssh --hostname %h" testrandom colab@:your-host-name-here.trycloudflare.com.

# Downloading
scp -o UserKnownHostsFile=deleteme -oProxyCommand="./cloudflared access ssh --hostname %h" colab@your-host-name-here.trycloudflare.com:testrandom testrandom2

scp for ngrok: (Use capital 'P' for port)

# Uploading
scp -o UserKnownHostsFile=deleteme -P 12345 testrandom colab@0.tcp.jp.ngrok.io:.

# Downloading
scp -o UserKnownHostsFile=deleteme -P 12345 colab@0.tcp.jp.ngrok.io:testrandom testrandom2

Result of argo tunnel bandwidth: 5:00 up: 5.1MB/s 00:12 down: 5.9MB/s 00:10 15:00 up: 5.1MB/s 00:12 down: 5.3MB/s 00:12 21:00 up: 5.0MB/s 00:12 down: 4.7MB/s 00:13 up: 4.7MB/s 00:13 down: 5.1MB/s 00:12 9/22 0:00 up: 5.1MB/s 00:12 down: 5.4MB/s 00:11

Result of ngrok bandwidth: 5:00 up: 1.6MB/s 00:41 down: 1.6MB/s 00:41 15:00 up: 2.6MB/s 00:24 down: 2.6MB/s 00:24 21:00 up: 1.5MB/s 00:41 down: 1.5MB/s 00:43 up: 2.6MB/s 00:24 down: 2.5MB/s 00:25 9/22 0:00 up: 1.5MB/s 00:41 down: 1.5MB/s 00:41

[tejasvi]

TIL Argo Tunnel. I think it should be made the default, possibly with a warning about the change in the next couple of versions.

[demotomohiro]

Argo tunnel works but ngrok doesn't work on google colab now. If ngrok never works on colab, I will make Argo tunnel the default.

[demotomohiro]

I have changed the default tunnel from ngrok to Argo Tunnel because ngrok stop working on colab at least since november 30. https://github.com/demotomohiro/remocolab/pull/80

[demotomohiro]

I have tested remocolab with ngrok today and it worked.