Multi-part download for SFTP.

[ghost]

I had to experience a lot of cases when single connection is slow.
For example 5-10 times less in comparison with possible network speed.
Unfortunately it is usual case for worldwide connection.

But it can be solved if you will add multipart downloading.
So multiple connections can be opened to download the same file simultaneously.
Chunks even can be buffered and then written to file on a disk.
Or it can be fully async with writing data to mapped file in random way.

Whatever, just use multiple network connections. Similar feature is available in BitKinex, but it is This feature is very important!

It is implemented in lftp for Linux. Or one more idea: https://github.com/jbd/msrsync.

[VictorVG]

You look at the speed of your connection with the provider, but you need to look at the capabilities of the entire chain to the remote host, and not all servers allow multiple connections from one IP. Usually in such cases, the server blocks the client considering this a DoS attack.

[ghost]

You look at the speed of your connection with the provider, but you need to look at the capabilities of the entire chain to the remote host

It is exactly what I mentioned.
Whole chain to remote host allows 100/300/500 Mbit depending on provider (tested by iperf and speedtest).
But single connection is limited for example at 50/75/100 Mbit.
Usually problem is related to upload speed.

Usually in such cases, the server blocks the client considering this a DoS attack.

My experience is different.
If we are talking about SSH/SFTP then I can say it is my server, so I can do what I want there. )) If you mentioned connection in general, for example FTP or HTTP, then I can say I never got such issues from public servers.
For example I used aria2c for HTTP/FTP for multipart download and never got any bans.

[VictorVG]

If this is your server, then you can configure it for any parameters, if it’s a stranger, no. For example, a Realtek server - there is not only that they don’t send data without specifying a valid e-mail, they also specifically limit the speed at the level of 50-60 Kbps. Do not believe, check it yourself. And this practice exists on most commercial servers, so there is no point in speeding it up. Only an eternal ban will be written out.

[ghost]

I do not know anything about Realtek but I tried to download some their drivers and it was Akamai CDN.
But file too small to start 2nd connection (if possible of course, I do not know, but most likely).

12/10 23:57:22 [NOTICE] Downloading 1 item(s)
[#c65dd4 27MiB/43MiB(64%) CN:1 DL:11MiB ETA:1s]
12/10 23:57:26 [NOTICE] Download complete: C://RealManage.Console.Setup.zip

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
c65dd4|OK  |    12MiB/s|C://RealManage.Console.Setup.zip

Do not believe, check it yourself. And this practice exists on most commercial servers, so there is no point in speeding it up.

But as I said I never got such problems.
Probably we are using different sites :)

---

Though it is not the topic of this issue. I just want to consider possibility to add mulipart download for sftp.
I do not need to download sftp from 3rd parties. I need to download for example backup through secure channel.

In other words I got the same problem with SFTP in NetBox as one solved with GetRight, DownThemAll and other downloaders for HTTP(S).
In first post I provided some attempts to solve it in Windows/Linux world.
So it will be great to get in NetBox!

[VictorVG]

hardhub

Akamai CDN is a structure that adds to all its advertising, well, these are trifles, the main thing is that it places precisely some popular data, and they are not interested in anything else. But if you met a couple of quick cases, then this exists everywhere. :)

A few years ago, a recording appeared on the Mozilla Fundation bug tracker:

"I suggest removing XUL, NNP support from SeaMonkey because I don’t use them, which means nobody needs them, and transfer the project to the Chorome core - I use it and I like it, which means I need to throw Gecko and replace it with WebKit since I don’t have Gecko need it, so nobody needs it anymore! "

Enough :) Firewood broke .... Do you propose to repeat the same thoughtlessly? :)

Multithreading is a good thing, but not thoughtlessly, and where it really will benefit in the form of reducing the time it takes to complete a task. Much more useful would be the possibility of asynchronous operation of the background queue so that you do not have to keep the NetBox panel open while waiting for the next 4 - 5 GB ISO from UNIX servers.

[ghost]

Much more useful would be the possibility of asynchronous operation of the background queue so that you do not have to keep the NetBox panel open while waiting for the next 4 - 5 GB ISO from UNIX servers.

No, I need absolutely different thing. I want to get single file but with maximum speed.
It is OK to open 5 instances of far manager to get what you described.
But there is no way to get what I need in Far Manager.

[VictorVG]

There is, and there is no need to rewrite anything - CURL 7.66.0 and newer - it has multithreading support. I use it myself -https://github.com/VictorVG/FarUE3_x64/releases and https://github.com/VictorVG/FarUE3_x86/releases - F2 - D - C - cURL submenu. Latest update is progress - I'm doing it on the basis of b5525 now - in b5524 minor flaws got out, and I had to bother with updating the compilers ...

[ghost]

But it is not in NetBox.
I think it will be good idea to make it as a part of NetBox and include in official distribution of far manager.

[VictorVG]

You are offered a tool that solves your problem now, but you say, “I don’t want it! I want it as I want it!”. Sources are open, offer a patch. No - use what you have.

[ghost]

Of course it is an option (independently good or not good for me or others).
But I see no reason why a NetBox solution cannot be proposed and discussed.
Somebody generates ideas, somebody implements it, somebody makes improvements and so on.
In any case we need to start with something.

[VictorVG]

So far, apart from the vague idea of "that it was!" There is no formulation of the problem. Try to formulate it more clearly so that you can understand what you are offering and what paths lead to achieving the goal.

And in the form "I have an idea, but I really don’t understand what I want, but you have to make it as I want!" we run into a dead end, and certainly the best idea will be rejected right away, because no one will waste time implementing something that the proposal author himself does not understand ...

[ghost]

"I have an idea, but I really don’t understand what I want, but you have to make it as I want!"

It sounds like you are kidding me. Or maybe you have not tried to understand the situation enough well.
Main idea is to maximize bandwidth usage downloading files via SFTP.
Current implementation of NetBox allows to resume transferring of file starting at specific offset (seek).
So in theory it is possible to start transferring using N connections initially.

For example we have on server 10 GB file.
And in settings of NetBox we have something like this: maximum connections: [ 5 ].
So when downloading (or uploading) will be started NetBox will open 5 connections.
It will download (or upload) at the same time 5 chunks of the same file.
It can query first 2 GB in one connection, next 2 GB (3rd and 4th) in second connection and so on.
Exactly like it is made by download managers like DownThemAll, GetRight and others (of course using HTTP/HTTPS).

Does it make sense now?

[VictorVG]

There will be a probable problem with FTP/SFTP/FTPS - the server must support this feature because it is he who selects the points in the file body from where the data transfer starts, and not all servers support this feature. Therefore, it is worth looking at the statistics of the availability of such support by existing servers. Otherwise, it may turn out to be a wasted effort.

[ghost]

I do not have such statistics, but I think it will be supported on all Linux distros and by all Windows SSH servers.
So probably any SSH host... I personally tested it on CentOS 7+, Fedora 29+, BitviseSSH server.
I am talking about SFTP only. And issue was named: "Multi-part download for SFTP".
As I know resuming for FTP is not supported at all.

So quick answer SFTP - 100%, FTP - 0%. But this issue is only about SFTP.

[VictorVG]

According to RFC 959, FTP allows the file to be downloaded from the point of disconnection, but does not allow parallel connection to the file, but although your proposal is based on RFC 959, you are mistakenly trying to separate the SFTP protocols - "SSH other FTP" and FTPS - "SSL other FTP" and FTP, although in both cases, and I previously used it for you, encrypted secure protocol traffic is transmitted through FTP protocol tools.

The protocol RFC 4253 The Secure Shell (SSH) Transport Layer Protocol itself only organizes a secure connection over insecure data channels. A protocol feature is encryption of the entire transmitted traffic, including passwords, using the selected algorithm.

Therefore, in the current version, your proposal is not working.

[ghost]

All that is ping-pong of terms and comments.
But I think you understand that if I will open 5 far managers I will be able to download the same file at the same time over SFTP.
And SFTP is available absolutely in all OpenSSH distros.
So I am sure you understand what I mentioned and if you think you can specify it more preciously then you are welcome! :)