Simple downloader in v2.0.19 was significantly faster with realdebrid than the new internal downlaoder

[privatize0965]

With the old simple downloader I'd get about 1GB/s in download speed and sometimes, depending on rdb servers I guess, even 1,5GB/s.

With ne new internal downloader I'm only getting 500MB/s at a max. No matter how many parallel connections and chunks are set.

I tried to reacreate the simple downoaders behavior by setting chunks and connection both to 1, but that didn't help at all. Also tried many different combinations, but as I said, download speed are never going anywhere over 500MB/s with this. Same thing for Aria2c. That's why I've always used simple downloader. It just worked the best.

Is there anything I'm missing? How exactly did the simple downloader work that made speeds differ so much?

This is also not just bad timing or currently struggling servers. Downgrading to v2.0.19 and simple downloader immediately gets me those speeds again.

What version are you using? v2.0.22

Wat OS are you running? Ubuntu 22.04.2 LTS

[rogerfar]

If you put Parallel connections and Parallel chunks to 0, then it should behave as the old downloader. Straight do disk.

I made a small test app here, you will need the .NET6 runtime to run it: net6.0.zip

You can change the appsettings.json to mess with the settings and DownloadList.json to add or change download paths.

If I run it with the default settings (no parallel) it gives me about 35MB/s which is also the max on my internet connection.

[privatize0965]

I have to run this on the Linux terminal. So No gui. Would you be able to provide a version of this without windows specific functions that also works in a console?

I'm running this on a cloud vm with ubuntu as OS.

System.NotSupportedException: Task bar progress only works on Windows
    at ShellProgressBar.ProgressBarOptions.set_EnableTaskBarProgress(Boolean value)
    at Downloader.Sample.Program.Initial() in C:\Users\Roger\Downloads\Downloader-master\src\Samples\Downloader.Sample\Program.cs:line 55
    at Downloader.Sample.Program.Main() in C:\Users\Roger\Downloads\Downloader-master\src\Samples\Downloader.Sample\Program.cs:line 36
END

Edit:

Maybe you just need to set EnableTaskBarProgress to false for this to work in a linux terminal

[jagvirdehal]

@privatize0965 you could access the web portal with an SSH tunnel in PuTTY. Just create a local tunnel with port 6500 and you should be able to access it

[privatize0965]

@privatize0965 you could access the web portal with an SSH tunnel in PuTTY. Just create a local tunnel with port 6500 and you should be able to access it

I was talking about the testing tool @rogerfar has posted in the reply above, not rdt-client itself. There is no webinterface. It's just a small commandline utility, which uses a feature that is window only, so I can't run that to test the different settings, which is the whole purpose of the tool. I don't have a windows desktop, which wouldn't be useful anyway for my testing purposes, but much less do I have a windows server.

[privatize0965]

So I just tried settings parallel connections and chunks to 0. With those settings, I'm getting ~150MB/s only. I'm getting about 3x that, by settings it to 4/4, but thats still far from the speed I'm getting with the old simple downloader, wich still downloads with over 1GB/s. Just tried both versions again.

[rogerfar]

I made a new test app, the other one was from the Downloader itself, this one has both the new Downloader and the old one. Downloadtest.zip

Make sure to edit "settings.json", none of the settings apply to the old downloader (except for URL obviously).

There might be 1 explanation, the old downloader from 2.0.19 and before wasn't very well reporting speeds back, as the throttle part wasn't working either very well, there could very well be a calculation/display error.

[privatize0965]

EDIT: For easier testing, you can recreate this by starting a simple http server on your local machine to download from. python -m http.server 51515 in the directory with your testfiles to download. Then set the download url to something like this http://127.0.0.1:51515/testfile.mkv. This way, you are not limited by your internet connection and not depended on the server you download from. Then just monitor the loopback interface (lo for linux).

---

Thanks. I'll test this when I find some time and give you some feedback.

@rogerfar So I did some testruns and recorded some data. The rates presented are also verified by watching external tools like iftop and bmon while downloading

Simpledownloader has indeed the highest download rate wich is also pretty much consistant throughout the complete download duration (for me between 1.6Gbps and 1.8Gbps or 200MB/s - 225MB/s).

old downloader

{
 `"UseOldDownloader": true,
  "Url": "https://45.download.real-debrid.com/speedtest/testDefault.rar",
  "BufferBlockSize": 8000,
  "ChunkCount": 1,
  "MaximumBytesPerSecond": 0,
  "MaxTryAgainOnFailover": 5,
  "ParallelDownload": true,
  "ParallelCount": 0,
  "Timeout": 3000,
  "RangeDownload": false,
  "RangeLow": 0,
  "RangeHigh": 0,
  "MinimumSizeOfChunking": 512,
  "ReserveStorageSpaceBeforeStartingDownload": true
}

> time dotnet ./DownloadTest.dll

Downloading with old downloader (HttpClient)
Complete

Five runs:
1. dotnet ./DownloadTest.dll  41.54s user 44.53s system 175% cpu 49.070 total
2. dotnet ./DownloadTest.dll  37.93s user 45.25s system 170% cpu 48.863 total
3. dotnet ./DownloadTest.dll  39.12s user 44.58s system 172% cpu 48.632 total
4. dotnet ./DownloadTest.dll  39.05s user 45.20s system 179% cpu 46.961 total
5. dotnet ./DownloadTest.dll  40.85s user 45.23s system 173% cpu 49.535 total

Result: 10GB/48s = 208.33 MB/s = 1.66Gbps

curl

I can also reproduce similar rates with curl

curl https://45.download.real-debrid.com/speedtest/testDefault.rar -o test.rar
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100 10.0G  100 10.0G    0     0   229M      0  0:00:44  0:00:44 --:--:--  221M

Result: 229 MB/s = 1.832 Gbps

new downlaoder

The new downloader behaves kinda weird.

First of, it's eating system memory like crazy when downloading big files. I minitored my system, which only has 8GB of ram, and it start shoveling data to swap like crazy while downloading. This might be part of the cause for slow speeds. The system gets bottlenecked by memory and io.

I found setting "ReserveStorageSpaceBeforeStartingDownload": true prevents this, kind of, but only sometimes. The option does not work, when the file is already existing on disk or at least, does not prevent the downloader downloading into memory first. It only ever works, without eating all the system memory is when the destination file does not exist beforehand.

But even with this, I'm getting wildly different download rates

Downloading with Downloader (WebClient)
Complete

1. dotnet ./DownloadTest.dll  117.76s user 80.67s system 245% cpu 1:20.82 total
2. dotnet ./DownloadTest.dll  183.97s user 144.34s system 88% cpu 6:12.87 total 
3. dotnet ./DownloadTest.dll  173.84s user 122.73s system 97% cpu 5:03.82 total

{
  "UseOldDownloader": false,
  "Url": "https://45.download.real-debrid.com/speedtest/testDefault.rar",
  "BufferBlockSize": 8000,
  "ChunkCount": 0,
  "MaximumBytesPerSecond": 0,
  "MaxTryAgainOnFailover": 5,
  "ParallelDownload": false,
  "ParallelCount": 0,
  "Timeout": 3000,
  "RangeDownload": false,
  "RangeLow": 0,
  "RangeHigh": 0,
  "MinimumSizeOfChunking": 512,
  "ReserveStorageSpaceBeforeStartingDownload": true

}

I managed to get to a max of 220MB/s with the new downloader with the settings below. So 4 downloads, 2 chunks and ReserveStorageSpaceBeforeStartingDownload = true. But even then, rates might start high for a couple of seconds and then the speed drops down significantly and then flucuates between very low speed like ~5MB/s and some rates higher up like 140MB/s.

{
  "UseOldDownloader": false,
  "Url": "https://45.download.real-debrid.com/speedtest/testDefault.rar",
  "BufferBlockSize": 8000,
  "ChunkCount": 2,
  "MaximumBytesPerSecond": 0,
  "MaxTryAgainOnFailover": 5,
  "ParallelDownload": true,
  "ParallelCount": 4,
  "Timeout": 3000,
  "RangeDownload": false,
  "RangeLow": 0,
  "RangeHigh": 0,
  "MinimumSizeOfChunking": 512,
  "ReserveStorageSpaceBeforeStartingDownload": true

}

All in all, I feel like this might me an upstream issue with https://github.com/bezzad/Downloader and big files. Which, I'd guess, most rdt-client users are dealing with.

This might also be way more proncounced on systems with beefier internet connections and comparativly low amount of memory. The server I'm testing on has a 10Gbps connection and "only" 8GB of ram. So, when this thing is download ~200MB/s to memory, things become problematic. But this only seems to be part of the problem as stated above, even with ReserveStorageSpaceBeforeStartingDownload = true, and monitoring low memory usage, the download rates still never really reacht whay curl or the old downloader are capable of.

I'm not sure how to properly test this further without the source of you test tool. I'd like to play around with it a bit. Also, if you need some more data or testing, feel free to ask

[ddpruel]

I just downgraded to 2.0.19 and tried the simple downloader for the first time. I also find that I am getting speeds far higher than any other method.

[rogerfar]

@privatize0965 thanks for testing that. I think it might be an issue of Downloader using still the old obsolete and deprecated WebClient instead of HttpClient. I will do some research and see if there is a better downloader out there.

[rogerfar]

I have created a new downloader from scratch that should be a lot better than the current options, supports bandwidth throttling and multiple threads too.

I create an update to the test app, if anyone wants to test out the 3 different downloaders that would be appreciated. DownloadTest.zip

[sylvaindd]

I have created a new downloader from scratch that should be a lot better than the current options, supports bandwidth throttling and multiple threads too.

I create an update to the test app, if anyone wants to test out the 3 different downloaders that would be appreciated. DownloadTest.zip

Hi, could you please publish the version on Docker Hub with a custom tag so we can test ? I just tried 2.0.26 too, i get 5-50Mpbs, with 2.0.19 I get 500-1000Mbps

[ctriaan]

2.0.26 works perfectly for me, maxing out my 200Mbps line.

Also, the whole process is really fast - unpacking, uploading into Sonarr, etc.

[ddpruel]

I'm unfortunately still experiencing significantly slower speeds than 2.0.19. 80MB/s on the old version vs 3MB/s on the latest, even when playing with parallel download settings.

[ctriaan]

@ddpruel , my settings below. General Maximum parallel downloads 3 Maximum unpack processes 3

Download client Download speed (in MB/s) (only used for the Internal Downloader) 0 Parallel connections per download (only used for the Internal Downloader) 0

I noticed that, when I tried 3 Parallel connections per download, the speed shown was 1/3 of the real download speed reported by my router.

[sylvaindd]

I think last release fixed it for me. I'd even say it's now faster than in 2.0.19.

Note that it's now in MB/s (I had it in Mbps in .19), it averages around 80-90MB/s which is what I'd expect.

Thanks !