search

beyond-all-reason / pr-downloader

console downloader for spring maps/games written in c++
GNU General Public License v2.0
0 stars 8 forks source link

Improve pr-downloader speed #8

Closed p2004a closed 2 years ago

p2004a commented 2 years ago

Three changes:

  1. Removed select wait from 1s to 10ms: I've noticed that sometimes it just unnecessarily waits for 1s without progressing, maybe at some point all passed fds are already ready and select doesn't generate any notification.
  2. Stop doing full validation of whole Sdp file after each update. Each file building Sdp is already verified in the WriteData.
  3. Don't initialize steam download if there aren't any files to download.

About 2:

The current code will catch situation where both, at the same time:

  1. File is corrupted
  2. pr-downloader crashes/is killed between SafeCloseFile and fileSystem->removeFile

It's highly unlikely scenario.

The current code also catches when there is a silent on disc corruption, or there is power outage and somehow files are inconsistent. (Note that when downloading file using CFile, it's first being saved as tmp, and only then moved to destination so there is quite fine protection).

Given that e.g. maps/engine downloaded via HTTP don't have similar protection at all, it makes sense to drop this second consistency validation.

Overall savings: 1: 2-3s, 2: ~9s (from ramdisk), 3: ~100ms.

With it:

time ./src/pr-downloader --filesystem-writepath /tmp/pr-downloader-test --rapid-download byar:test | tail -n 1
[Info] /home/p2004a/Workspace/BAR/pr-downloader/src/main.cpp:215:main():Download complete!
./src/pr-downloader --filesystem-writepath /tmp/pr-downloader-test  byar:test  0,09s user 0,05s system 65% cpu 0,208 total
tail -n 1  0,00s user 0,00s system 1% cpu 0,208 total
Beherith commented 2 years ago

There have been previous instances of file corruption, though as far as I can tell, it also happened to some maps too. But knowing that, im very open to merging this, just to see if it does ok in the real world. I merged quite a few other changes to prd recently.

Beherith commented 2 years ago

Can the validation maybe be enabled via cmd arg?

p2004a commented 2 years ago

There is already a separate --rapid-validate flag that validates consistency of the whole pool, so it can be triggered this way.

Beherith commented 2 years ago

ok thanks!