mscdex / ssh2

SSH2 client and server modules written in pure JavaScript for node.js
MIT License
5.51k stars 663 forks source link

Support for Bun runtime #1416

Closed rgillan closed 3 weeks ago

rgillan commented 3 weeks ago

We are currently using ssh2 as a production server under nodejs (which is awesome). As part of a major performance based overhaul of our software we are implementing the Bun runtime. The team there are well underway in implementing support for V8 APIs, although we are running into a problem with the node-gyp/bindings which does not get compiled when ssh2 is installed via Bun. Just wanted some help/guidance as to where to dig as the lack of compiled bindings (sshcrypto.node) is appearing to cause issues. If we open up a ssh client to our server code under nodejs, we see the following:

[6208826.861727083] Custom crypto binding available
[6208826.861727083] Local ident: 'SSH-2.0-ssh2js1.15.0'
[6208826.861727083] Remote ident: 'SSH-2.0-OpenSSH_8.6'
[6208826.861727083] Outbound: Sending KEXINIT
[6208826.861727083] Inbound: Handshake in progress
[6208826.861727083] Handshake: (local) KEX method: curve25519-sha256@libssh.org,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,kex-strict-s-v00@openssh.com
[6208826.861727083] Handshake: (remote) KEX method: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
[6208826.861727083] Handshake: KEX algorithm: curve25519-sha256
[6208826.861727083] Handshake: (local) Host key format: rsa-sha2-512,rsa-sha2-256,ssh-rsa
[6208826.861727083] Handshake: (remote) Host key format: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
[6208826.861727083] Handshake: Host key format: rsa-sha2-512
[6208826.861727083] Handshake: (local) C->S cipher: aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,chacha20-poly1305@openssh.com
[6208826.861727083] Handshake: (remote) C->S cipher: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
[6208826.861727083] Handshake: C->S Cipher: chacha20-poly1305@openssh.com
[6208826.861727083] Handshake: (local) S->C cipher: aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,chacha20-poly1305@openssh.com
[6208826.861727083] Handshake: (remote) S->C cipher: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
[6208826.861727083] Handshake: S->C cipher: chacha20-poly1305@openssh.com
[6208826.861727083] Handshake: (local) C->S MAC: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[6208826.861727083] Handshake: (remote) C->S MAC: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[6208826.861727083] Handshake: C->S MAC: <implicit>
[6208826.861727083] Handshake: (local) S->C MAC: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[6208826.861727083] Handshake: (remote) S->C MAC: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[6208826.861727083] Handshake: S->C MAC: <implicit>
[6208826.861727083] Handshake: (local) C->S compression: none,zlib@openssh.com,zlib
[6208826.861727083] Handshake: (remote) C->S compression: none,zlib@openssh.com,zlib
[6208826.861727083] Handshake: C->S compression: none
[6208826.861727083] Handshake: (local) S->C compression: none,zlib@openssh.com,zlib
[6208826.861727083] Handshake: (remote) S->C compression: none,zlib@openssh.com,zlib
[6208826.861727083] Handshake: S->C compression: none
[6208826.861727083] Received DH Init
[6208826.861727083] Generating signature ...
[6208826.861727083] Outbound: Sending KEXECDH_REPLY
[6208826.861727083] Outbound: Sending NEWKEYS
[6208826.861727083] Inbound: NEWKEYS
[6208826.861727083] Handshake completed
[6208826.861727083] Outbound: Sending EXT_INFO
[6208826.861727083] Inbound: Received SERVICE_REQUEST (ssh-userauth)
[6208826.861727083] Outbound: Sending SERVICE_ACCEPT (ssh-userauth)
[6208826.861727083] Inbound: Received USERAUTH_REQUEST (none)

If we run the same code with bun, you can see that the "custom crypto binding available" section does not fire, which is checking what's available in openssh on the machine we believe:

[9.552599917] Outbound: Sending KEXINIT
[9.552599917] Inbound: Handshake in progress
[9.552599917] Handshake: (local) KEX method: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,kex-strict-s-v00@openssh.com
[9.552599917] Handshake: (remote) KEX method: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
[9.552599917] Handshake: KEX algorithm: ecdh-sha2-nistp256
[9.552599917] Handshake: (local) Host key format: rsa-sha2-512,rsa-sha2-256,ssh-rsa
[9.552599917] Handshake: (remote) Host key format: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
[9.552599917] Handshake: Host key format: rsa-sha2-512
[9.552599917] Handshake: (local) C->S cipher: aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
[9.552599917] Handshake: (remote) C->S cipher: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
[9.552599917] Handshake: C->S Cipher: aes128-ctr
[9.552599917] Handshake: (local) S->C cipher: aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
[9.552599917] Handshake: (remote) S->C cipher: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
[9.552599917] Handshake: S->C cipher: aes128-ctr
[9.552599917] Handshake: (local) C->S MAC: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[9.552599917] Handshake: (remote) C->S MAC: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[9.552599917] Handshake: C->S MAC: hmac-sha2-256-etm@openssh.com
[9.552599917] Handshake: (local) S->C MAC: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[9.552599917] Handshake: (remote) S->C MAC: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[9.552599917] Handshake: S->C MAC: hmac-sha2-256-etm@openssh.com
[9.552599917] Handshake: (local) C->S compression: none,zlib@openssh.com,zlib
[9.552599917] Handshake: (remote) C->S compression: none,zlib@openssh.com,zlib
[9.552599917] Handshake: C->S compression: none
[9.552599917] Handshake: (local) S->C compression: none,zlib@openssh.com,zlib
[9.552599917] Handshake: (remote) S->C compression: none,zlib@openssh.com,zlib
[9.552599917] Handshake: S->C compression: none
[9.552599917] Received DH Init
[9.552599917] Generating signature ...
[9.552599917] Outbound: Sending KEXECDH_REPLY
[9.552599917] Outbound: Sending NEWKEYS
[9.552599917] Socket ended
[9.552599917] Socket closed

The ssh client connection throws an error and closes the socket:

ssh_dispatch_run_fatal: Connection to 192.168.42.102 port 614: invalid format

The test machine is a RPi5 running Ubuntu 24.04 LTS (GNU/Linux 6.8.0-1008-raspi aarch64). nodejs version is v18.19.1. Bun version is latest (1.1.27). Any help debugging what's happening would be very much appreciated.

Jarred-Sumner commented 3 weeks ago

Probably a bug in Bun rather than ssh2

rgillan commented 3 weeks ago

Probably a bug in Bun rather than ssh2

@Jarred-Sumner thought it could be still, but from walking through the ssh2 code it should be ok with the binding not being available and wondering where else to look. Will keep digging

mscdex commented 3 weeks ago

This project only supports node.js. You're probably better off asking the Bun folks instead as they will know more about the status of any issues with node.js/V8 compatibility.

mscdex commented 3 weeks ago

Side note: there should be a line logged regardless of the custom binding availability that says whether it's available or not. If you're not seeing that at all, then something is really messed up in Bun.

mscdex commented 3 weeks ago

One last thing: note that different algorithms are being negotiated between the two test cases, so it's not an apples-to-apples comparison.

rgillan commented 3 weeks ago

All good, was just looking for where to debug further. Will start from the missing debug log, cheers

rgillan commented 3 weeks ago

so will dig further with the bun team to get to the bottom of this. The different negotiation is due to curve25519 not being in the crypto implementation yet I expect, but will have to dig further to see where the process is failing. We have both nodejs and bun versions running now without custom bindings so hopefully will converge on the issue quickly. Thanks for your help.

mscdex commented 3 weeks ago

FWIW you can explicitly set algorithms in the connection config object. That might help you have a more direct comparison between node and bun.

rgillan commented 3 weeks ago

FWIW you can explicitly set algorithms in the connection config object. That might help you have a more direct comparison between node and bun.

Thanks, have just done that and now have everything matched. Logging the outbound buffer contents at KEXECDH_REPLY and there's a difference, so will hopefully find out where the mismatch is

rgillan commented 3 weeks ago

@mscdex just to close this out. By digging through the resultant binary packet (buffer) structure we found an issue with the buffer.utf8Write implementation, which has already been fixed. With the latest bun, simply replacing the utf8Write calls (136 locations) with the current nodejs buffer.write api at least all the ssh2 server functionality works as expected. Thanks for all your help getting this resolved.

supermario commented 1 week ago

Hey @rgillan – think I'm having similar issues. A little confused by your resolution, you mentioned "which has already been fixed" but then also note your approach of making replacements for utf8Write calls. What am I missing?

With the latest released bun and ssh2 I'm still experiencing issues which don't occur when run on node. Does your comment suggest you vendored ssh2 with the described replacements to make it work for bun?

rgillan commented 1 week ago

@supermario if you upgrade to bun 1.1.27 (sorry) the buffer.utf8Write issue is fixed. We have done a clean bun install after deleting the node_modules folder and both ssh2 and cpu-features are latest release NPM. For our use case (a ssh server with lots of moving parts/capabilities, environment variable passing, remote shell, tunnels etc.) it fixed the problem. Did find that unless we deleted the node_modules folder it didn't update everything but think that was more our environment than anything untowards. Happy to help you debug if it still doesn't work for you.