connection interrupted when multiple identities

[qrkourier]

• There's an undiagnosed problem with the latest build from the main branch (pre-release commit 986122d305ebcf75ff32c21da3cdcc62793af777).
• I noticed that my tunneled SSH session was consistently terminated a few seconds after it began.
• I was able to reconnect immediately after the session was terminated.
• The problem only occurs when I run the client tunneler with --identity-dir, not with --identity.
• The problem did not recur when I ran the latest release version with --identity-dir (v0.21.0 is stable).
• The problem did recur with the binary from the Ubuntu 22 Jammy Debian package and the binary build from main.
• The tunneler did not crash or restart on the client or hosting side at the time the connection was interrupted.
• The hosting tunneler performs normally and is a MIPS build of v0.21.0.
• The CloudZiti network is v0.27.5.

I collected these client tunneler error logs at the moment the connection was interrupted.

May 13 10:29:26 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       30.698]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[cb2c8d5d-e747-4292-ad8e-e82eb35a3087.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:26 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       30.699]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[74adeb39-8def-4d34-a2b9-a0c96f6a7871.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:30 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       35.059]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[minicontroller.ziti] request failed: -110(connection timed out)
May 13 10:29:31 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       35.763]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[cb2c8d5d-e747-4292-ad8e-e82eb35a3087.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:31 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       35.769]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[74adeb39-8def-4d34-a2b9-a0c96f6a7871.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.853]   ERROR ziti-sdk:connect.c:945 connect_reply_cb() conn[1.7/Connecting] failed to connect [-20/not a directory]
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.853]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.854]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:374 on_ziti_write() ziti_write(ziti_conn[0x5600a00a6b20]) failed: invalid state
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.875]   ERROR ziti-sdk:channel.c:483 dispatch_message() ch[3] received message without conn_id or for unknown connection ct[ED72] conn_id[6]
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.883]   ERROR ziti-sdk:channel.c:461 dispatch_message() ch[3] could not find waiter for reply_to = 2
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.883]   ERROR ziti-sdk:channel.c:483 dispatch_message() ch[3] received message without conn_id or for unknown connection ct[ED70] conn_id[7]
May 13 10:29:35 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       39.903]   ERROR ziti-sdk:channel.c:483 dispatch_message() ch[3] received message without conn_id or for unknown connection ct[ED72] conn_id[7]
May 13 10:29:36 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       40.822]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[cb2c8d5d-e747-4292-ad8e-e82eb35a3087.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:36 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       40.826]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[74adeb39-8def-4d34-a2b9-a0c96f6a7871.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:41 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       45.984]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[cb2c8d5d-e747-4292-ad8e-e82eb35a3087.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:41 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       45.986]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[74adeb39-8def-4d34-a2b9-a0c96f6a7871.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:46 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       51.052]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[cb2c8d5d-e747-4292-ad8e-e82eb35a3087.production.netfoundry.io] request failed: -103(software caused connection abort)
May 13 10:29:46 kpop4 ziti-edge-tunnel[2167918]: (2167918)[       51.054]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[74adeb39-8def-4d34-a2b9-a0c96f6a7871.production.netfoundry.io] request failed: -103(software caused connection abort)

Next steps toward a diagnosis include

• re-triggering the issue with a different service and hosting tunneler to further isolate the client tunneler as the cause.
• collect debug logs

[qrkourier]

I can reproduce the same client-side tunneler issue with a ZEDS (v0.26.5) service using the release binary (v0.21.0) as a hosting tunneler. This rules out any potential that the issue was caused by the MIPS hosting tunneler or the CloudZiti network where I first encountered the issue. I've attached the DEBUG log from this test.

ziti-edge-tunnel-0.21.0-29-g986122d.log

[qrkourier]

• The issue did not recur when started with --identity-dir, and only one identity was present in the dir.

[qrkourier]

• The issue recurs immediately when a particular identity from a different network is present in the identity dir. The issue ceases to recur when that identity is removed from the identity dir and the tunnel restarted.

[ekoby]

is it still an issue? @qrkourier

[qrkourier]

I'm unable to trigger this issue with ZET 0.21.5 installed with the Debian x64 package.