balena-os / leviathan

A distributed hardware testing framework
https://balena-os.github.io/leviathan
Apache License 2.0
22 stars 6 forks source link

replace socat method for ssh tunneling with balena tunnel + ssh tunnel commands #1100

Closed rcooke-warwick closed 5 months ago

rcooke-warwick commented 5 months ago

This PR https://github.com/balena-os/leviathan/pull/1098 failed to resolve the issue with the ssh connection to the DUT failing at the end of the test suites. The exact reason is still unknown but restarting the socat tunnel unstuck the tests. There was some unexpected behaviour with the socat command being used to forward the DUT ports, so instead I've tried to replace it with balena tunnel + standalone port forwarding using ssh

Explanation: Instead of using: socat tcp-listen:22222,reuseaddr,fork "system:ssh <USER>@<BALENA_SSH_PROXY> -p 22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /root/id host <WORKER_UUID> timeout 600 /usr/bin/nc localhost <WORKER_PORT>" to set up port forwarding from the worker to the core container, and then doing the same on the worker to crate a similar tunnel from worker to DUT, we just use :

balena tunnel <UUID> <CORE_PORT 8888>:<WORKER_SSH_PORT (22222)>

This makes the worker accessible on core via localhost:8888 - core now has SSH access to the worker.

Then, we use the command:

ssh -L <CORE_PORT>:<DUT_IP>:<DUT_PORT> -p <CORE_PORT 8888> <USER>@127.0.0.1 -N

This forwards DUT_PORT to CORE_PORT - because its being done over the balena tunnel from the core to workers SSH port. It will DUT_PORT accessible at core localhost:

This makes the SSH connection not hang during the test suites where it did hang when using the socat method. It also simplifies the implementation, and also uses our existing tunneling method (balena tunnel) instead of a custom implementation.

acostach commented 5 months ago

lgtm