search

pkg / sftp

SFTP support for the go.crypto/ssh package
BSD 2-Clause "Simplified" License
1.52k stars 380 forks source link

SFTP Using Library significantly slower than SFTP native on host #464

Closed daghaian closed 3 years ago

daghaian commented 3 years ago

I am banging my head against to wall to try and understand what would cause such a significant degradation in performance when using this package vs native SFTP.

Using Native SFTP image

Using SFTP Package

image

I'm using concurrent reads/writes both enabled and attempting to use the ReadFrom method. I am running with version 1.13.2 (latest as far as I'm aware). Any assistance would be appreciated

puellanivis commented 3 years ago

There has so far not so much focus on performance and optimization of this library. It Works™, and that’s 99% good enough for the cases when this library needs to be used. If someone has access to the native client, I imagine they would almost always choose that one first.

That said, there are a few things that could be outside of the code causing slow copying. Right now, we are disabling concurrent writes by default, which means that latency significantly affects performance. Over a localhost adapter, 1Gbit, or 10Gbit, this is not really significant, but over the internet, this can lead to significantly worse performance.

If you’re comfortable with the added risks of concurrent writes: you can use the UseConcurrentWrites client option. I’m in the process of overhauling the client, and will probably take a new look at both the reading and writing code, hopefully getting a more robust solution.

daghaian commented 3 years ago

Thank you for your prompt response. It turns out that the real performance capability comes from the reader.IO that is used. I was pruning through the code for rclone (https://github.com/rclone/rclone) which is using your package but with a custom asynchronous reader and the speeds achievable are significantly faster.

puellanivis commented 3 years ago

👍 Good to hear.

saluru commented 3 years ago

Unsubscribe

On Wed, Aug 25, 2021, 6:40 PM daghaian @.***> wrote:

I am banging my head against to wall to try and understand what would cause such a significant degradation in performance when using this package vs native SFTP.

Using Native SFTP [image: image] https://user-images.githubusercontent.com/16483722/130878043-1a03e1b8-1e20-48e0-900c-16203d75294b.png

Using SFTP Package

[image: image] https://user-images.githubusercontent.com/16483722/130878116-b22f6e8a-35f6-4f59-8222-9a54499ea2b9.png

I'm using concurrent reads/writes both enabled and attempting to use the ReadFrom method. I am running with version 1.13.2 (latest as far as I'm aware). Any assistance would be appreciated

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pkg/sftp/issues/464, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSV4QK3PXTKMXRAEECCEPLT6V5OXANCNFSM5C2FBHTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .