Huge RAM usage when checking a big torrent

[GoogleCodeExporter]

What steps will reproduce the problem?
1. This bug was reported by a qBittorrent user.
2. The user tried to check a 60GB torrent with qBittorrent and it consumed all 
his RAM (4GB) in a few minutes.
3. The user even made a video to show the effect (plays fine in VLC):
http://blaxpirit.org.ua/screenshot/qbittorrent-bug.avi

What is the expected output? What do you see instead?
It should never use that much memory, no matter what the torrent size is

What version of the product are you using? On what operating system?
libtorrent v0.15svn on Windows 7 64bits (compiled with msvc 9.0)

Please provide any additional information below.

Original issue reported on code.google.com by dch...@gmail.com on 14 Jun 2010 at 3:14

[GoogleCodeExporter]

The video doesn't show which application used the memory. Could it be the 
system caching the file?

You could suggest turning off file caching (a session settings).

Was qBittorrent built as a 64 bit application?

Original comment by arvid.no...@gmail.com on 14 Jun 2010 at 4:39

[GoogleCodeExporter]

Arvid > No qBittorrent was build as 32bits.
Is the session setting sessionSettings.cache_size ?

BlaXpirit > sessionSettings.cache_size can be altered from advanced properties 
in qBittorrent. Please try lowering the value and see if it changes anything.

Original comment by dch...@gmail.com on 14 Jun 2010 at 4:44

[GoogleCodeExporter]

No. my point was that since the video didn't show which application was using 
the RAM, it might be the Operating system itself, for file caching. This is 
controlled via disk_io_read_mode, disk_io_write_mode. If you disable disk cache 
for all files for both read and write, the OS won't use any RAM for cache.

which configuration was libtorrent built with?

Original comment by arvid.no...@gmail.com on 15 Jun 2010 at 3:59

[GoogleCodeExporter]

Libtorrent was compiled with:
bjam -q --without-python --toolset=msvc variant=release link=static 
runtime-link=shared logging=none geoip=static dht-support=on boost=source

Boost v1.42 was compiled with:
bjam -q --with-filesystem --with-thread --toolset=msvc variant=release 
link=static runtime-link=shared

Original comment by dch...@gmail.com on 15 Jun 2010 at 6:38

[GoogleCodeExporter]

Issue 89 has been merged into this issue.

Original comment by arvid.no...@gmail.com on 19 Jun 2010 at 2:28

[GoogleCodeExporter]

Good news. Someone else is able to reproduce it. Apparently, he is using 64bits 
too, this may not be a coincidence.

Original comment by dch...@gmail.com on 19 Jun 2010 at 2:33

[GoogleCodeExporter]

what storage mode is used? sparse, allocate or compact?

Original comment by arvid.no...@gmail.com on 19 Jun 2010 at 2:45

[GoogleCodeExporter]

i've removed some stuff and now it checks files and does not eat all of my free 
rams ;}
can anyone confirm it's fixes issue and is it okey to do that?

bool file::open(std::string const& path, int mode, error_code& ec)
{
.....

const static open_mode_t mode_array[] =
{
// read_only
{GENERIC_READ, FILE_SHARE_READ, OPEN_EXISTING}, // , FILE_FLAG_RANDOM_ACCESS
// write_only
{GENERIC_WRITE, FILE_SHARE_READ, OPEN_ALWAYS}, // , FILE_FLAG_RANDOM_ACCESS
// read_write
{GENERIC_WRITE | GENERIC_READ, FILE_SHARE_READ, OPEN_ALWAYS}, // , 
FILE_FLAG_RANDOM_ACCESS

....

Original comment by radioman...@gmail.com on 19 Jun 2010 at 7:14

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

   int allocation_mode = libtorrent::storage_mode_sparse;

   add_torrent_params p; 
   p.storage_mode = (storage_mode_t)allocation_mode;
   p.paused = true;
   p.duplicate_is_error = false;
   p.seed_mode = true; // waits for pressing 'j'
   p.auto_managed = true;

Original comment by radioman...@gmail.com on 19 Jun 2010 at 7:29

[GoogleCodeExporter]

In my opinion this is not really a bug. The FILE_FLAG_RANDOM_ACCESS flag will 
cause the Windows NT Cache Manager to create a memory-mapped file and map the 
file being checked into the process address space. The cache manager on Vista 
and above is completely different than XP and below. The OS recognizes that the 
RAM was available and will utilize it to the fullest extent. It should also be 
noted that the NT cache manager *should* recognize when resources are low and 
use a smaller-size mapping.

As Arvid pointed out this OS caching behavior can be disabled. However on 
operating systems above Windows XP I would recommend allowing the OS to control 
the system resources and caching behavior.

Removing the FILE_FLAG_RANDOM_ACCESS flag will have a negative impact on file 
checking speed. If this is not important then feel free to disable the caching 
behavior.

Best Wishes,
-David Delaune

Original comment by david.de...@googlemail.com on 20 Jun 2010 at 1:22

[GoogleCodeExporter]

so if i have 32GB ram, and lets recheck my startrek, it uses all ram ;D nuts

Original comment by radioman...@gmail.com on 20 Jun 2010 at 1:37

[GoogleCodeExporter]

would you rather all your ram go to waste, not being used for anything?

It doesn't make any sense for an OS to _not_ use all free ram for caching.

I will stop trying to reproduce this then, thanks for the explanation David!

Original comment by arvid.no...@gmail.com on 20 Jun 2010 at 2:50

• Changed state: Invalid

[GoogleCodeExporter]

...i bet that users will be 'just happy' ;}

Original comment by radioman...@gmail.com on 20 Jun 2010 at 3:08

[GoogleCodeExporter]

Arvid, according to user feedback, it seems that libtorrent v0.14.10 is NOT 
affected by this issue. Is that normal?

Also, even though it is reported here that this behavior is normal, it makes 
the computer lag like hell, which does not look normal to me.

I think I will revert to v0.14.10 (again)...

Original comment by dch...@gmail.com on 20 Jun 2010 at 3:11

[GoogleCodeExporter]

or you can use my 'fix' and still use trunk until final fix arrives ;}

Original comment by radioman...@gmail.com on 20 Jun 2010 at 3:47

[GoogleCodeExporter]

but it become 'invalid' issue ;D we have been left only with the hope

Original comment by radioman...@gmail.com on 21 Jun 2010 at 5:51