Non working cache when session_settings::disable_os_cache

[GoogleCodeExporter]

I set in torrent session:
disk_io_write_mode = session_settings::disable_os_cache;
disk_io_read_mode = session_settings::disable_os_cache;

because when caching files libtorrent use >2GB of RAM.
But with this settings caching non working. When i add new torrent(files 
already exist) whith torrent_info:

.storage_mode = storage_mode_sparse;
.auto_managed = false;
.paused = true;

Then i started torrent and torrent state changes in the following 
sequence(alert->message()):
1) resumed 
2) state changed to: checking (q) 
3) state changed to: checking 
4) state changed to: checking (q) 
5) paused 
6) resumed 
7) state changed to: checking 
8) state changed to: checking (q) 
9) paused 

And caching does not occur.

PS: I tested this in 0.15.7 ,0.15.8 ,RC_0_15 branch in Windows 7 64bit

Original issue reported on code.google.com by Lacro...@gmail.com on 3 Oct 2011 at 6:52

[GoogleCodeExporter]

if you set the os cache settings to enabled, then it works?

Original comment by arvid.no...@gmail.com on 3 Oct 2011 at 4:40

[GoogleCodeExporter]

yes

Original comment by Lacro...@gmail.com on 4 Oct 2011 at 2:20

[GoogleCodeExporter]

i can't confirm this, at least on my custom storage (which is derived from 
libtorrent's 0.15.3). i will test default storage now

ps. does op mean that files are not being rechecked? i.e. we have some files 
downloaded - we add torrent with these files - libtorrent doesn't see the 
downloaded data. do i understand correctly?

Original comment by rain87...@gmail.com on 7 Oct 2011 at 9:49

[GoogleCodeExporter]

can't confirm anyway. tested with
disk_io_write_mode = session_settings::disable_os_cache;
disk_io_read_mode = session_settings::disable_os_cache;
aligned and unaligned files are checked correctly. tested on win 2003 server x64

Original comment by rain87...@gmail.com on 7 Oct 2011 at 10:00

[GoogleCodeExporter]

"ps. does op mean that files are not being rechecked? i.e. we have some files 
downloaded - we add torrent with these files - libtorrent doesn't see the 
downloaded data. do i understand correctly?"

libtorrent trying checking files, but can not do this and paused torrent.

Original comment by Lacro...@gmail.com on 7 Oct 2011 at 11:22

[GoogleCodeExporter]

Now i make build with defines:

TORRENT_STATS
TORRENT_DISK_STATS
TORRENT_STORAGE_DEBUG
TORRENT_VERBOSE_LOGGING
TORRENT_LOGGING

I started with:
disk_io_write_mode = session_settings::disable_os_cache;
disk_io_read_mode = session_settings::disable_os_cache;

And add already downloaded torrent. In "libtorrent_logs0\main_session.log" i 
see this messages:

Oct 07 17:25:09 spawning network thread
Oct 07 17:25:09 open listen port
Oct 07 17:25:09 done starting session
fastresume data for Warehouse_13_Season_3 rejected: The operation completed 
successfully ret:-1
Oct 07 17:25:20: fatal disk error [ error: Invalid attempt to access a memory 
address torrent: Warehouse_13_Season_3 ]
Oct 07 17:25:20: fatal disk error [ error: Invalid attempt to access a memory 
address torrent: Warehouse_13_Season_3 ]

PS. I translate messages from my Russian system language.

Original comment by Lacro...@gmail.com on 7 Oct 2011 at 11:30

[GoogleCodeExporter]

Do you know what filesystem the files are on?
Are they accessed over a network share of some kind or anything like that?
does this reproduce reliably for other torrents or just this one?

Original comment by arvid.no...@gmail.com on 7 Oct 2011 at 5:15

[GoogleCodeExporter]

Files are located on the local NTFS filesystem. Thit problem reproduce at all 
torrent when enable disable_os_cache.

Original comment by Lacro...@gmail.com on 8 Oct 2011 at 5:50

[GoogleCodeExporter]

hydri, we have just made a qBittorrent release with libtorrent v0.15.8 and 
disk_io_write_mode = session_settings::disable_os_cache;
disk_io_read_mode = session_settings::disable_os_cache;

Windows users are now reporting a lot of I/O errors as soon as the torrent data 
is being written to the disk. Error messages vary: "The parameter is incorrect" 
or "Reached the end of file.".

Original comment by dch...@gmail.com on 9 Oct 2011 at 3:34

[GoogleCodeExporter]

Expanding on Comment 9, see this report for more info:
https://bugs.launchpad.net/qbittorrent/+bug/871328

Original comment by hammered...@gmail.com on 9 Oct 2011 at 4:27

[GoogleCodeExporter]

yes, the problem seems to be in storage::write_unaligned. read_unaligned works 
fine (by me), but write doesn't

Original comment by rain87...@gmail.com on 10 Oct 2011 at 7:52

[GoogleCodeExporter]

i have digged slightly inside libtorrent's storage. the problems are in 
storage::write_unaligned

first is - when lt tries to read data from file, to get buffer aligned (the gap 
between aligned_start and file_offset)
- it does not count, that file may not have any data in it (yes, this is what 
windows is talking about when says "file end..."). so i've rewritten this part 
of code as

    size_type actual_file_size = file_handle->get_size(ec);
    if(actual_file_size < 0)
    {
        TORRENT_ASSERT(ec);
        return actual_file_size;
    }

    // allocate a temporary, aligned, buffer
    disk_buffer_holder aligned_buf(*disk_pool(), disk_pool()->allocate_buffers(
        num_blocks, "write scratch"), num_blocks);
    file::iovec_t b = {aligned_buf.get(), aligned_size};
    size_type ret;
    if(aligned_start < actual_file_size) // we have something to read
    {
        if(actual_file_size < aligned_start + aligned_size) // but not that much
            b.iov_len = actual_file_size - aligned_start;
        ret = file_handle->readv(aligned_start, &b, 1, ec);
        b.iov_len = aligned_size;
        if (ret < 0)
        {
            TORRENT_ASSERT(ec);
            return ret;
        }
    }

second problem is - when lt writes data back to file, it always writes 
aligned_size bytes, despite file may be smaller
so i've made
    // write the buffer back to disk
    if(file_offset + size > actual_file_size) // file will grow -> don't allow to grow more than required
        b.iov_len = size + file_offset - aligned_start;
    ret = file_handle->writev(aligned_start, &b, 1, ec);
    b.iov_len = aligned_size; // not sure if this is neccessary

with these fixes the trouble seems to be gone, tested on several torrents

Original comment by rain87...@gmail.com on 10 Oct 2011 at 9:50

[GoogleCodeExporter]

last condition is probably incorrect, although it will work in most cases. more 
correct is
    if(aligned_start + aligned_size > actual_file_size) // file will grow -> don't allow to grow more than required
        b.iov_len = std::max(size + file_offset, actual_file_size) - aligned_start;

Original comment by rain87...@gmail.com on 10 Oct 2011 at 10:35

[GoogleCodeExporter]

I've been trying to reproduce this without any success. maybe the condition 
just isn't common enough for me to have seen it on just a small number of 
torrents.

All buffers must be full memory pages, the last block in a file must be written 
as a full page. The size of the file can be adjusted later, but it has to be 
re-opened for that, in buffered mode.

Original comment by ar...@rasterbar.com on 11 Oct 2011 at 8:05

[GoogleCodeExporter]

The first patch seems good though. I'll check that in. thanks!

Original comment by arvid.no...@gmail.com on 11 Oct 2011 at 8:46

[GoogleCodeExporter]

Arvid, it does not seem to happen for all torrents. We have torrents attached 
to https://bugs.launchpad.net/qbittorrent/+bug/871328 that we are sure trigger 
I/O errors for users. Maybe this helps.

Original comment by dch...@gmail.com on 11 Oct 2011 at 8:51

[GoogleCodeExporter]

dunno, as for me - problem is 100% reproduceable on every non-singlefile 
torrent with disk_io_read_mode = session_settings::disable_os_cache;
write_unaligned simply fails at every first write, because file is zero-sized 
(even if file is not empty, write_unaligned will fail anyway when file will 
have to grow, because write_unaligned will attempt to read beyond eof)

all this goes to windows, i haven't tested this on linux (and actually don't 
see any sense to turn linux caching off, it works not that buggy as windows')

Original comment by rain87...@gmail.com on 11 Oct 2011 at 8:59

[GoogleCodeExporter]

Arvid, what's the status on this? We really need this since os caching seems 
buggy on Windows 7 64bits.

Original comment by dch...@gmail.com on 1 Nov 2011 at 7:46

[GoogleCodeExporter]

tried the torrent attached to the launchpad thread a while back, without being 
able to reproduce the problem. I think it might be caused by either me running 
in a VM (and maybe the driver isn't picky about the alignment) or running XP 
(I'm pretty sure the alignment requirements apply to XP as well, they're well 
documented).

I checked in the patch to ignore errors when the reading before writing 
unaligned would fall outside of the end-of-file (as posted by rain87). If 
someone that can reproduce these problems could tell me exactly which operation 
that fails, and with what error code, it might be easy to figure out how to fix 
it. I suppose I could try getting a win7 VM.

Original comment by ar...@rasterbar.com on 2 Nov 2011 at 4:39

[GoogleCodeExporter]

i can reproduce this with 100% rate. function which fails is 
storage::write_unaligned, at line
size_type ret = file_handle->readv(aligned_start, &b, 1, ec);
in file::readv ReadFileScatter fails, and last error code is 38 which means 
ERROR_HANDLE_EOF
38 (0x26)
Reached the end of the file
according to msdn

i can provide any additional info you need

Original comment by rain87...@gmail.com on 2 Nov 2011 at 11:56

[GoogleCodeExporter]

Thanks! I think the patch I just checked in fixes this. Please let me know if 
there are any other issues that appear after this one.

Original comment by arvid.no...@gmail.com on 3 Nov 2011 at 4:57

[GoogleCodeExporter]

nice, as i understand, you have accepted the first patch, that fixes 
ERROR_HANDLE_EOF. but there is one more problem - as i've already mentioned, 
libtorrent always writes aligned_size bytes. this means that ALL unaligned 
files in torrent will have wrong size on disk (except some rare cases, when 
file's end is piece-aligned). i have solved this in my second short patch by 
reducing b.iov_len before calling to file_handle->writev - and this worked fine 
for me. you noticed that
>the last block in a file must be written as a full page
okay, but then we have to adjust file size after we have written this last 
page. if second patch looks inappropriate for you, please solve this issue in 
the more preferable way

Original comment by rain87...@gmail.com on 3 Nov 2011 at 7:43

[GoogleCodeExporter]

There was code in to fix the size issue, but I think it wasn't working. I did 
check in an alternative code path that uses NtSetInformationFile() to set the 
file size without closing the handle. Are you still seeing the issue of files 
being too big?

Original comment by arvid.no...@gmail.com on 4 Nov 2011 at 4:48

[GoogleCodeExporter]

no, i haven't checked the trunk yet, i will try it today and report results. i 
thought you ignored second issue, sorry

Original comment by rain87...@gmail.com on 4 Nov 2011 at 7:42

[GoogleCodeExporter]

i have tested https://libtorrent.svn.sourceforge.net/svnroot/libtorrent
on w2k3x64 the problem has gone. but on winxp32 i get file_error_alert with 
error code 87 (ERROR_INVALID_PARAMETER) for file, which is in the middle of the 
torrent. this file is 47 bytes, but it takes 4096 on disk (first 47 bytes are 
the file itself, and some trash until 4096). in the moment i'm unable to track 
down where does this error come from (i don't have development tools on my 
winxp), but if you have no idea what does this error mean, i will track it a 
bit later

Original comment by rain87...@gmail.com on 4 Nov 2011 at 9:31

[GoogleCodeExporter]

The error means that I'm passing in invalid arguments to a system call. My 
guess would be that the function that's failing with this error is 
NtSetInformationFile(). Now, I think I'm passing in good arguments, so, setting 
a breakpoint on that call and looking at the values of the variables passed 
into it would probably reveal what the error is.

Original comment by arvid.no...@gmail.com on 4 Nov 2011 at 11:58

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

sorry for large delay in response, was able to test just now. error occures in
storage::readwritev -> file_handle->set_size(file_iter->size, ec); -> 
SetFilePointerEx

here is screenshot with stack trace 
http://img5.imageshack.us/img5/5707/qttempti4814.png

actually i don't understand, why does this error happen - all parameters seems 
to be valid, m_file_handle is valid too, because GetFileSizeEx(m_file_handle, 
&cur_size) just before works correctly. maybe SetFilePointerEx fails because 
file is opened in unbuffered mode?

i've set a breakpoint at NtSetInformationFile in file::writev, and found out 
that NtSetInformationFile is called for the first file in the torrent, and 
works correctly - it limits file size to correct value. but it isn't called for 
the file in the middle of the torrent - because file_size is zero, because size 
is 4096

have you any idea?

Original comment by rain87...@gmail.com on 9 Nov 2011 at 9:32

[GoogleCodeExporter]

thanks, this should be fixed now.

Original comment by arvid.no...@gmail.com on 13 Nov 2011 at 5:05

[GoogleCodeExporter]

yes, this seems to be fixed. tried on my test torrents on both 2k3 and xp, 
worked fine with
    settings.disk_io_read_mode = session_settings::disable_os_cache;
    settings.disk_io_write_mode = session_settings::disable_os_cache;
will now try to find win7 to test at

Original comment by rain87...@gmail.com on 14 Nov 2011 at 9:36

[GoogleCodeExporter]

tested win7 x64 - okay too. hope the bug has really gone, but sure it has to be 
tested on more systems

Original comment by rain87...@gmail.com on 14 Nov 2011 at 11:53

[GoogleCodeExporter]

Where to get the fixed version for testing?

Original comment by Lacro...@gmail.com on 15 Nov 2011 at 2:28

[GoogleCodeExporter]

get it from svn
svn co https://libtorrent.svn.sourceforge.net/svnroot/libtorrent/trunk

Original comment by rain87...@gmail.com on 15 Nov 2011 at 8:17

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I tested the new version with on client_test.cpp with:
        settings.disk_io_write_mode = session_settings::disable_os_cache;
        settings.disk_io_read_mode = session_settings::disable_os_cache;

in few torrents, but my problem not fixed. The situation is similar as in the 
first post. But when disable_os_cache is disable, the memory is no longer 
released in large quantities. And I could not run the client in debug mode with 
options:
TORRENT_STATS
TORRENT_STORAGE_DEBUG
TORRENT_VERBOSE_LOGGING
TORRENT_LOGGING

because i catch segmentation faults whith this options. And with the option 
TORRENT_DISK_STATS program not building.

Original comment by Lacro...@gmail.com on 16 Nov 2011 at 10:52

[GoogleCodeExporter]

Lacro, can you exactly describe the case you are testing and provide a torrent 
which experiences this issue every time?

as i understand, the situation is following:
1. you have some large file downloaded (~2 gb)
2. you run your libtorrent-based app with
        settings.disk_io_write_mode = session_settings::disable_os_cache;
        settings.disk_io_read_mode = session_settings::disable_os_cache;
3. you add the torrent with downloaded file
4. torrent pauses (probably due to some disk error), and data is not being 
rechecked, showing zero downloaded bytes

am i right?

if i would able to reproduce the error, i would able to track it's source down

ps. i have never experienced read errors - only write errors. so your problem 
probably differs from the one we have been working at in this thread (although 
this thread is your :) )

Original comment by rain87...@gmail.com on 16 Nov 2011 at 11:05

[GoogleCodeExporter]

1. you have some large file downloaded (~2 gb) -- No. Now i tested in two 
torrents:
- One file 700Mb
- Few(22) files 350Mb

2. Yes

3. Yes

4. Now torrent always in checking queue state(in client_test.cpp show [all 
(1)][downloading (0)][non-paused (0)][seeding (0)][queued (1)][stopped 
(0)][checking (1)])

Original comment by Lacro...@gmail.com on 16 Nov 2011 at 11:24

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Now I build client_test.cpp with bjam(early I used qmake with own .pro file) 
and i see this error on console:
ERROR: 'Invalid attempt to access a memory address' in f:\......avi

PS Message translate from russian

Original comment by Lacro...@gmail.com on 16 Nov 2011 at 11:38

[GoogleCodeExporter]

can you look for your error message here 
http://freebasic-world.narod.ru/er_rorread.html and provide an error code? 
error message translation does not give much information :)

Original comment by rain87...@gmail.com on 16 Nov 2011 at 11:54

[GoogleCodeExporter]

Status log from client_test:

1) [all (1)][downloading (1)][non-paused (1)][seeding (0)][queued 
(0)][stopped(0)][checking (1)]

2) [all (1)][downloading (0)][non-paused (0)][seeding (0)][queued 
(1)][stopped(0)][checking (1)]

3) [all (1)][downloading (1)][non-paused (1)][seeding (0)][queued 
(0)][stopped(0)][checking (1)]

4) [all (1)][downloading (0)][non-paused (0)][seeding (0)][queued 
(1)][stopped(0)][checking (1)]

Original comment by Lacro...@gmail.com on 16 Nov 2011 at 11:54

[GoogleCodeExporter]

rain87: Error code 998

Original comment by Lacro...@gmail.com on 16 Nov 2011 at 11:55

[GoogleCodeExporter]

my guess would be that it is 998 - ERROR_NOACCESS
998 (0x3E6)
Invalid access to memory location.

Original comment by rain87...@gmail.com on 16 Nov 2011 at 11:59

[GoogleCodeExporter]

Yes, all right

Original comment by Lacro...@gmail.com on 16 Nov 2011 at 12:03

[GoogleCodeExporter]

nice. this is definitely different error from one which arvid has fixed in 
write_unaligned. i will try to reproduce it on my system - but actually i doubt 
that i will. according to statistic among our users, this is very rare error

Original comment by rain87...@gmail.com on 16 Nov 2011 at 12:15

[GoogleCodeExporter]

Tested with lt trunk on Win7x64 and the bug that rain87 and arvid were 
discussing seems fixed here and I don't see any other issues with large file 
downloads. 

Original comment by caluml...@gmail.com on 17 Nov 2011 at 2:54

[GoogleCodeExporter]

rain87: now i tested on WinXP, all works fine, probably, this problem only for 
Win7x64

Original comment by Lacro...@gmail.com on 17 Nov 2011 at 6:34

[GoogleCodeExporter]

ok, i'm now porting my app to libtorrent 16, when it will be done i'll test 
your case on win7x64 (hope i will do it today)

Original comment by rain87...@gmail.com on 17 Nov 2011 at 9:06

[GoogleCodeExporter]

Arvid is this fix going to be applied to the 0.15 branch? 

Original comment by caluml...@gmail.com on 17 Nov 2011 at 11:41

[GoogleCodeExporter]

I put it in both trunk and RC_0_15. I assumed people were mostly testing it in 
0.15.

and I'm starting to feel confident enough to cut a 0.15.9 release with this.

Original comment by arvid.no...@gmail.com on 19 Nov 2011 at 6:50