search

rapier1 / hpn-ssh

HPN-SSH based on OpenSSH
https://psc.edu/hpn-ssh-home
Other
302 stars 41 forks source link

Java clients #30

Closed conorgriffin closed 1 year ago

conorgriffin commented 3 years ago

This is a bit of a shot in the dark, but I've read your papers/presentations and justification for HPN-SSH and I'd love to use it, but we're bound at the moment to using a Java library. At the moment I'm using JScape but it's not performing very well compared to openssh and so I'm going to evaluate alternatives. Are you aware of any Java libraries that endeavour to do what you've done with HPN-SSH? Or which Java SSH libraries are best for high throughput? Many thanks for your time if you get a chance to respond. Otherwise good luck with the new funding, I'll be watching this space with interest.

Conor

rapier1 commented 3 years ago

Take a look at http://www.jcraft.com/jsch/. They say they support None Cipher Switching which is part of the HPN-SSH enhancements. I'ver never used it so I can't give you any idea of it's performance characteristics or usage.

On 1/18/21 5:18 PM, Conor Griffin wrote:

This is a bit of a shot in the dark, but I've read your papers/presentations and justification for HPN-SSH and I'd love to use it, but we're bound at the moment to using a Java library. At the moment I'm using JScape but it's not performing very well compared to openssh and so I'm going to evaluate alternatives. Are you aware of any Java libraries that endeavour to do what you've done with HPN-SSH? Or which Java SSH libraries are best for high throughput? Many thanks for your time if you get a chance to respond. Otherwise good luck with the new funding, I'll be watching this space with interest.

Conor

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/rapier1/openssh-portable/issues/30, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66E6P44TX45FEIE42CDS2SXUXANCNFSM4WH4WPMQ.

rapier1 commented 3 years ago

Also, if it doesn't implement dynamic receive buffers you may be able to statically define large buffers - which will help with bulk data transport. You'll need to dig through the source code to find the variable definition that they are using for that. It may be obvious but my guess is that the buffer will be defined to either 64K or 2048K. If you redefine it to something like 32MB that could help with inbound transfers. It may screw up interactive sessions with over buffering issues.

I don't know java all that well so I don't even know if the kernel parameters (like the tcp connections receive buffer) is available to the JVM. Sorry I can't be of more help on this.

conorgriffin commented 3 years ago

Yeah Jsch is one I'm going to check out thanks. The one I'm using right now has a default download buffer of 32K and only a single outstanding request! They do allow you to enable "pipelining" where you can have multiple parallel requests and their default is 8. This is still almost 10x slower than vanilla sftp/openssh. Upping the the buffer to 64K improves things a little, but going beyond that results in truncated files being 'successfully' downloaded, i.e. the transfer succeeds and doesn't throw any errors but the file is incomplete 🤷 . Also, upping the number of parallel requests seems to offer no improvement once you go above 10.

Anyway, I'm going to experiment with a few libraries that allow these settings to be manipulated and see how the performance compares. Thanks for your time. 🍻

rapier1 commented 3 years ago

Best of luck. I also guess I should reach out to some of these developers at some point.

Chris

On 1/19/21 2:37 PM, Conor Griffin wrote:

Yeah Jsch is one I'm going to check out thanks. The one I'm using right now has a default download buffer of 32K and only a single outstanding request! They do allow you to enable "pipelining" where you can have multiple parallel requests and their default is 8. This is still almost 10x slower than vanilla sftp/openssh. Upping the the buffer to 64K improves things a little, but going beyond that results in truncated files being 'successfully' downloaded, i.e. the transfer succeeds and doesn't throw any errors but the file is incomplete 🤷 . Also, upping the number of parallel requests seems to offer no improvement once you go above 10.

Anyway, I'm going to experiment with a few libraries that allow these settings to be manipulated and see how the performance compares. Thanks for your time. 🍻

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rapier1/openssh-portable/issues/30#issuecomment-763080877, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66GYPW3YAJ5AYIEGPDTS2XNQZANCNFSM4WH4WPMQ.

jgoldverg commented 1 year ago

Hello so I was planning on opening a similar request, about if there could be plans to write a Java client? I could even potentially consider as I have the immense need for a stable, and performance oriented SSH/SCP/SFTP client. Sadly Java is missing alternatives to JSch. SSHJ is of worthy note as it has attempted to do this but the performance of that library is way below JSch.

Currently the Java community in general is stuck using JSch as the only SSH/SCP/SFTP client and it has massive limitations and has not been properly kept up to date as the last update to the Maven Repository was in 2018!

@conorgriffin in my experience JSch does perform well(when compared to rclone doing an Sftp transfer) but due to the massive lack of documentation the library becomes very difficult to tune.

rapier1 commented 1 year ago

Honestly, that would be a heavy lift as it would essentially be rewriting large portions of HPN-SSH in Java - which is not a language I have much experience with. I would love to work with someone on it but I am not the right person to do it myself. That said, it may be possible use libssh.a with JNI to get started but that's mostly me just thinking out loud.

jgoldverg commented 1 year ago

Personally I do come from the Java background which is why I would feel ok in attempting to take this on but I want to address what I am looking and if we could create some kind roadmap.

My use case, is I want to use the HPN ssh client in my Java app to communicate to openssh & hpn ssh servers where I would be sending large files HPNS implementation of sftp and scp which hopefully will result in higher throughput.

From my trivial understanding of the project I would only need the client to fundamentally keep doing sftp, scp transfers but to saturate(lets say a chameleon cloud node with a 10Gb link) then I would need the hpn server and client running or can I still get benefits with just the client?

In my usecase of saturating links on the client end then I would only need to port the client to Java correct?

tomaswolf commented 1 year ago

Java discussions are probably a bit out of place here, but nevertheless: the original JSch 0.1.55 is long abandoned. It does not support modern keys (no ed25519, for instance). It also has numerous bugs.

There is a fork which fixed some bugs, added support for modern keys, and lots of other things. But it's still JSch, and it has inherited some of JSch's nasty bugs.

That said: JSch is not the only Java library for SSH. There is, for instance, Apache MINA sshd. It's not perfect either, but works well and is an actively maintained open-source project of the ASF.