search

SteamRE / DepotDownloader

Steam depot downloader utilizing the SteamKit2 library.
GNU General Public License v2.0
1.96k stars 264 forks source link

Slow downloading speed all the time... #127

Closed PaulNevada closed 4 years ago

PaulNevada commented 4 years ago

Steam Client is downloading at full 100 Mbit/sec (my bandwidth) and then I shut it down and DepotDownloader's speed is around 10-15 Mbit/sec or even less. The same AppID and everything else. It looks like the download rate is capped by something. It's 10 times slower. What is that? I tried -max-downloads and -max-servers - no help... And, how can I debug it with the needed args? Thanks!

azuisleet commented 4 years ago

The answer is still the same as https://github.com/SteamRE/DepotDownloader/issues/107#issuecomment-642264622

PaulNevada commented 4 years ago

How to debug it? You have to investigate that issue - it does exist while all is just fine with the networking on my side...

azuisleet commented 4 years ago

You need to try the suggested parameters.

PaulNevada commented 4 years ago

You need to try the suggested parameters.

I tried them both - in any combinations - it didn't help at all...

PaulNevada commented 4 years ago

How can I debug it with the needed args? I am quite a new c-user... I can build it, but how to start it from the VS with the args?

PaulNevada commented 4 years ago

???

PaulNevada commented 4 years ago

What proofs do I need to provide to you and how? I'll make them. Just tell me, please.

PaulNevada commented 4 years ago

?

voided commented 4 years ago

@paulnevada1983 to confirm, you've tried the -max-downloads flag that was recommended?

PaulNevada commented 4 years ago

Yes, as well as the other parameter which is -max-servers... The speed is always the same - 10 Mbit/sec, BUT! I figured out that if the game's content is presented as tons of small files (1-100 MB) it works fast enough and when the program moves to the big one(s) it starts 8-10 times slower. The same games' builds via Steam Client on the same PC and its hard disks are being performed fast as it should. Trying to investigate the code these days...

Bladehawke commented 4 years ago

I did some testing here and saw 180+ Mbps with -max-downloads 16

PaulNevada commented 4 years ago
  1. What app, depot, manifest ID?
  2. What country are you from?
  3. Did you notice any differences between runs with that parameter on/off? Thanks!
Bladehawke commented 4 years ago
  1. -app 275850 -depot 275851 -manifest 7464613103538631399
  2. USA
  3. I was getting more like 20-30 Mbps with it off
PaulNevada commented 4 years ago

Can you try the same runs (off and on) but for the -app 883710 -depot 883711 in case you own it, please? Latest build is ok. You see, this one contains big file - 22 GB - it's needed to be checked as well...

Bladehawke commented 4 years ago

Getting like 10-30 (variable) on the large file.

I haven't examined the source as I know very little about network programming, but I'd guess that what's happening is that the max-downloads argument allows it to download that many files in parallel, while Steam is using more of a download manager approach and chunking the file. It downloads (as an example) 4-6 different pieces of the large file at the same time (like IDM) while I'm thinking only one thread per file here.

PaulNevada commented 4 years ago

10-30 Mbit/sec? I think this Tool works very close to the Steam Client, but, not the same way for sure... And it does support chunking while downloading... And what's the real difference?

Bladehawke commented 4 years ago

Yes, and that's with the -max-downloads 16

azuisleet commented 4 years ago

Can you give https://github.com/SteamRE/DepotDownloader/tree/chunks-as-tasks a shot?

Bladehawke commented 4 years ago

I'm getting > 150 Mbps with that using @paulnevada1983 's requested depot Looks like a fix for his issue.

PaulNevada commented 4 years ago

Can you give https://github.com/SteamRE/DepotDownloader/tree/chunks-as-tasks a shot?

Do I have to run it with some above args on?

PaulNevada commented 4 years ago

For me official 2.3.6 works even better then this one...

Bladehawke commented 4 years ago

The chunks-as-tasks branch has an apparent (not real) stall as it allocates the files early on, but then with -max-downloads 16 I was getting over 150Mbps and up to 300 (I have 400Mbps total bandwidth). If you watch this with Task Manager, you'll see the disk usage spike while network usage is almost non-existent when it's allocating the space.

azuisleet commented 4 years ago

I can either make the pre-allocation more visible (like showing a message) or fold it into the download process silently (the previous behavior). I believe Steam does the former.

The important piece is if you your network connection is being saturated when the pre-allocation is complete. is that happening?

Bladehawke commented 4 years ago

I could probably saturate the connection by increasing my -max-downloads.
I think just a message that it's pre-allocating would be adequate. It took me a minute to realize what it was doing the first time I played with it, but it's definitely a huge improvement in both download speed and total time for large files over the master branch.

PaulNevada commented 4 years ago

More information will be great. As I see it doesn't work for me at all - here's the question - are DD with SteamKit2 some emulators of how the Steam Client work naturally?

azuisleet commented 4 years ago

I have added the messages for pre-allocation and validation for the I/O tasks. What part isn't working?

DepotDownloader attempts to follow the processes that Steam does when downloading files.

PaulNevada commented 4 years ago

Are these ones live here or have you added them for future release? And is SteamKit2 itself is the same as DD - a try to emulate all the things? Thanks!

azuisleet commented 4 years ago

The changes are pending in https://github.com/SteamRE/DepotDownloader/tree/chunks-as-tasks

Once it's confirmed that speeds are improved, I can merge it for a future release.

PaulNevada commented 4 years ago

And is SteamKit2 itself as well as DD - a try to emulate all the things about Steam Client?

AmGona commented 4 years ago

The changes are pending in https://github.com/SteamRE/DepotDownloader/tree/chunks-as-tasks

Once it's confirmed that speeds are improved, I can merge it for a future release.

I am dumb.. how do i download and use the chunks as tasks depotdownloader branch?

KhaosVoid commented 4 years ago

Can confirm that branch "chunks-as-tasks" is indeed faster compared to master. Huge improvement as network traffic shown in Task Manager shows constant receive activity when downloading a large file. In particular, the file from a game I was downloading was a 45GB data pack. When using 2.3.6 build, max servers and max downloads set to 64, my download speed averaged 10-14Mbps with occasional 16-18Mbps spikes. Using the "chunks-as-tasks" branch with the same settings, my speed stayed around 680-740Mbps downloading the same file.

@AmGona You need to download the source and compile yourself. You'll need to publish to a local directory as just compiling as Release will only compile the DepotDownloader project and the dependent assemblies from the NuGet packages will not be present.

azuisleet commented 4 years ago

This was merged into ac97c011077f8476beee540f04c8a3e220744233. A new release is pending.