Open GoogleCodeExporter opened 8 years ago
yeah, I would imagine these errors are related. thanks for the detailed report,
I'll take a look
Original comment by arvid.no...@gmail.com
on 9 Jan 2013 at 12:47
which version of libtorrent is this? I take it 0.16.x, right?
Original comment by arvid.no...@gmail.com
on 9 Jan 2013 at 2:50
Originally I bumped into this with 0.16.6, but also tried 0.16.5 and 0.16.4
while I could still reproduce this.
Original comment by Daymansm...@gmail.com
on 9 Jan 2013 at 8:48
I would imagine it's related to files opened in OVERLAPPED mode, and
specifically where the operation is launched with an overlapped structure,
hence asking for it to be asynchronous.
My guess is that this is synonymous with ERROR_IO_PENDING so I've added
ERROR_CANT_WAIT to be handled the same way.
You don't happen to know which actual system call causes this error, do you?
in libtorrent_aio I improved the disk I/O error reporting infrastructure to
include the low-level operation that failed as well.
Original comment by arvid.no...@gmail.com
on 10 Jan 2013 at 3:18
> You don't happen to know which actual system call causes this error, do you?
Unfortunately I don't.
Building fails with the patch applied.
http://libtorrent.svn.sourceforge.net/viewvc/libtorrent/branches/RC_0_16/src/fil
e.cpp?view=patch&r1=7840&r2=7839&pathrev=7840
src\file.cpp(1311) : error C2446: '==' : no conversion from 'long' to 'HANDLE'
Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast
src\file.cpp(1311) : error C2040: '==' : 'HANDLE' differs in levels of
indirection from 'long'
src\file.cpp(1313) : error C2065: 'last_error' : undeclared identifier
src\file.cpp(1521) : error C2446: '==' : no conversion from 'long' to 'HANDLE'
Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast
src\file.cpp(1521) : error C2040: '==' : 'HANDLE' differs in levels of
indirection from 'long'
src\file.cpp(1523) : error C2065: 'last_error' : undeclared identifier
Original comment by Daymansm...@gmail.com
on 10 Jan 2013 at 12:55
ah, yes. fixed.
Original comment by arvid.no...@gmail.com
on 11 Jan 2013 at 4:50
Does this still happen in 0.16.7 ?
Original comment by arvid.no...@gmail.com
on 21 Jan 2013 at 5:28
[deleted comment]
[deleted comment]
Still an issue with 0.16.7.
Finally got some traces.
---------------
0 libtorrent::file::writev file.cpp 1593 0x7fed9e01d30
1 libtorrent::default_storage::write_unaligned storage.cpp 1345 0x7feda004542
2 libtorrent::default_storage::readwritev storage.cpp 1219 0x7feda003bf8
3 libtorrent::default_storage::writev storage.cpp 959 0x7feda002b0b
4 libtorrent::piece_manager::write_impl storage.cpp 1803 0x7feda006650
5 libtorrent::disk_io_thread::flush_range disk_io_thread.cpp 607 0x7feda14ca2d
6 libtorrent::disk_io_thread::flush_contiguous_blocks disk_io_thread.cpp 471 0x7
feda14c094
7 libtorrent::disk_io_thread::thread_fun disk_io_thread.cpp 2123 0x7feda152c95
8 boost::_mfi::mf0<void,libtorrent::disk_io_thread>::operator() mem_fn_template.
hpp 50 0x7feda1646fa
9 boost::_bi::list1<boost::_bi::value<libtorrent::disk_io_thread * __ptr64>
>::operator()<boost::_mfi::mf0<void,libtorrent::disk_io_thread>,boost::_bi::list
0> bind.hpp 254 0x7feda164097
10 boost::_bi::bind_t<void,boost::_mfi::mf0<void,libtorrent::disk_io_thread>,boo
st::_bi::list1<boost::_bi::value<libtorrent::disk_io_thread * __ptr64> >
>::operator() bind_template.hpp 21 0x7feda16286d
11 boost::asio::detail::win_thread::func<boost::_bi::bind_t<void,boost::_mfi::mf
0<void,libtorrent::disk_io_thread>,boost::_bi::list1<boost::_bi::value<libtorren
t::disk_io_thread * __ptr64> > > >::run win_thread.hpp 117 0x7feda16268a
12 boost::asio::detail::win_thread_function win_thread.ipp 118 0x7fed9d92a4b
13 beginthreadex MSVCR100D 0x5fd472e5
14 beginthreadex MSVCR100D 0x5fd472a4
15 BaseThreadInitThunk kernel32 0x765f652d
16 RtlUserThreadStart ntdll 0x76cec521
---------------
And
---------------
0 libtorrent::file::writev file.cpp 1593 0x7fed9e01d30
1 libtorrent::default_storage::write_unaligned storage.cpp 1345 0x7feda004542
2 libtorrent::default_storage::readwritev storage.cpp 1219 0x7feda003bf8
3 libtorrent::default_storage::writev storage.cpp 959 0x7feda002b0b
4 libtorrent::piece_manager::write_impl storage.cpp 1803 0x7feda006650
5 libtorrent::disk_io_thread::flush_range disk_io_thread.cpp 607 0x7feda14ca2d
6 libtorrent::disk_io_thread::thread_fun disk_io_thread.cpp 2269 0x7feda153813
7 boost::_mfi::mf0<void,libtorrent::disk_io_thread>::operator() mem_fn_template.
hpp 50 0x7feda1646fa
8 boost::_bi::list1<boost::_bi::value<libtorrent::disk_io_thread * __ptr64>
>::operator()<boost::_mfi::mf0<void,libtorrent::disk_io_thread>,boost::_bi::list
0> bind.hpp 254 0x7feda164097
9 boost::_bi::bind_t<void,boost::_mfi::mf0<void,libtorrent::disk_io_thread>,boos
t::_bi::list1<boost::_bi::value<libtorrent::disk_io_thread * __ptr64> >
>::operator() bind_template.hpp 21 0x7feda16286d
10 boost::asio::detail::win_thread::func<boost::_bi::bind_t<void,boost::_mfi::mf
0<void,libtorrent::disk_io_thread>,boost::_bi::list1<boost::_bi::value<libtorren
t::disk_io_thread * __ptr64> > > >::run win_thread.hpp 117 0x7feda16268a
11 boost::asio::detail::win_thread_function win_thread.ipp 118 0x7fed9d92a4b
12 beginthreadex MSVCR100D 0x5fd472e5
13 beginthreadex MSVCR100D 0x5fd472a4
14 BaseThreadInitThunk kernel32 0x765f652d
15 RtlUserThreadStart ntdll 0x76cec521
---------------
Original comment by Daymansm...@gmail.com
on 22 Jan 2013 at 5:28
I've tried ignoring those errors completely with attached patch. But it didn't
go very well.
I was able to finish the torrent, but after reboot and recheck it had some
pieces missing.
Roughly 1% of data was corrupt (~489 MiB) with 47.8 GiB torrent size and 16 MiB
chunk size. This is around 30 corrupted chunks, scattered across 28 corrupted
files with 43 files fully downloaded. Each file has a size of 533 MiB - 1.1 GiB.
Original comment by Daymansm...@gmail.com
on 23 Jan 2013 at 9:00
Attachments:
> I've tried ignoring those errors completely with attached patch. But it
didn't go very well.
> I was able to finish the torrent, but after reboot and recheck it had some
pieces missing.
> Roughly 1% of data was corrupt (~489 MiB) with 47.8 GiB torrent size and 16
MiB chunk size. This is around 30 corrupted chunks, > scattered across 28
corrupted files with 43 files fully downloaded. Each file has a size of 533 MiB
- 1.1 GiB.
Strangely enough I cannot reproduce this with mingw64 build. Downloaded 40 GiB
w/o issues.
I will remove my patch and revert original fix
http://libtorrent.svn.sourceforge.net/viewvc/libtorrent/branches/RC_0_16/src/fil
e.cpp?r1=7844&r2=7826&view=patch to see if mingw is affect at all.
This might get ugly if this is a compiler bug.
Original comment by Daymansm...@gmail.com
on 23 Jan 2013 at 6:39
my first attempt at handling this explicitly excluded mingw with #ifdefs,
because that error code is not defined in the mingw headers and would fail to
build for me otherwise.
Thanks for the traces, that's extremely helpful! Is that the only place you see
this error happen?
It points to GetOverlappedResult() failing. I have a potential fix, could you
try the attached patch?
Also, are you downloading onto an exotic filesystem by any chance? (by which I
mean anything other than ntfs).
Original comment by arvid.no...@gmail.com
on 24 Jan 2013 at 5:12
Attachments:
> Thanks for the traces, that's extremely helpful! Is that the only place you
see this error happen?
Looks like it's the only place. I've set breakpoints at lines 1356 and 1593.
But only the last one was triggered.
> I have a potential fix, could you try the attached patch?
Seems to be working. Running with original fix reverted and wait.diff applied.
Downloaded around 50 GiB w/o any issues.
> Also, are you downloading onto an exotic filesystem by any chance? (by which
I mean anything other than ntfs).
Nope. Just NTFS.
The only 'exotic' (in windows world) thing I can think of is that I'm running
native 64-bit version. But the same happens with official qBittorrent builds,
which are 32-bit.
Strange thing is that mingw was never affected (maybe because it's using posix
threads?).
Original comment by Daymansm...@gmail.com
on 24 Jan 2013 at 8:55
It appears to be caused by the bug mentioned in one of the comments at the
bottom of:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms683209(v=vs.85).aspx
Here's what that report says:
Windows 7: Resolves a race condition where a multi-threaded app using
GetOverlappedResult can return without resetting the event in the overlapped
structure, causing the next call to this function to return prematurely.
Windows Vista (default): Provides the behavior with the race condition that
applications may have a dependency on. Applications wishing to avoid this race
prior to the Windows 7 behavior should wait on the overlapped event and when
signaled, call GetOverlappedResult with bWait == FALSE. "
Original comment by arvid.no...@gmail.com
on 25 Jan 2013 at 5:48
I've checked in that patch now. thanks for the report.
Original comment by arvid.no...@gmail.com
on 25 Jan 2013 at 5:52
Unfortunately this bug hit me again. On the same spot with the same backtrace.
I'm attaching the backtrace with the contents of OVERLAPPED structure if this
is of any help.
I've also tried to embed compatibility manifest in both torrent.dll and
qbittorrent.exe as described in
http://msdn.microsoft.com/en-us/library/dd371711%28VS.85%29.aspx w/o any
success.
The command I use to build libtorrent is:
bjam -j4 -q --toolset=msvc --prefix=%INST_DIR% boost=system boost-link=shared
link=shared runtime-link=shared variant=release debug-symbols=off
resolve-countries=on full-stats=on export-extra=off ipv6=on dht-support=on
character-set=unicode geoip=static encryption=openssl windows-version=vista
threading=multi address-model=64 host-os=windows target-os=windows
embed-manifest=on architecture=x86 warnings=off warnings-as-errors=off
inlining=full optimization=speed "cflags=/O2 /GL /favor:blend"
"linkflags=/NOLOGO /OPT:REF /OPT:ICF=5 /LTCG"
"include=%BUILDROOT%\OpenSSL\OpenSSL64\include"
"include=%BUILDROOT%\Boost\Boost64\include"
"library-path=%BUILDROOT%\OpenSSL\OpenSSL64\lib"
"library-path=%BUILDROOT%\Boost\Boost64\lib" "define=BOOST_ALL_NO_LIB" install
----
qBittorrent is built with BOOST_ASIO_ENABLE_CANCELIO, which is not enabled by
default, but is enabled in libtorrent Jamfile so I turn it on.
I use windows-version=vista with libtorrent and define _WIN32_WINNT=0x0600 when
building qBittorrent, so they match.
Original comment by Daymansm...@gmail.com
on 8 Feb 2013 at 9:21
Attachments:
are you using RC_0_16 or 0.16.7?
RC_0_16 has one more fix which I'm pretty confident in.
Original comment by arvid.no...@gmail.com
on 9 Feb 2013 at 9:52
I'm using 0.16.7 patched with
http://libtorrent.svn.sourceforge.net/viewvc/libtorrent/branches/RC_0_16/src/fil
e.cpp?r1=7844&r2=7932
Are there other fixes beyond file.cpp?
Original comment by Daymansm...@gmail.com
on 9 Feb 2013 at 11:02
[deleted comment]
I've tried RC_0_16 with same results.
I've also tried checking WaitForSingleObject return value, and it's always ok -
zero.
Original comment by Daymansm...@gmail.com
on 9 Feb 2013 at 12:41
that was the patch I had in mind. I think I may have made two check-ins,
because I missed a few uses of GetOverlappedResult() the first time.
It's odd that GetOverlappedResult() would return can't wait when it's not asked
to wait (i.e. last argument is false).
Original comment by arvid.no...@gmail.com
on 9 Feb 2013 at 10:05
0.16.8 still has the issue.
So far only the workaround from old qBt bugtracker helps: archive the file in
question, delete original, unpack, recheck and go.
Doesn't help:
1. Restart
2. Disabling antivirus (well, I had to try)
3. Using 'disable_os_cache'
I'll try to use preallocation instead of sparse files, maybe this will help,
I'm just out of ideas.
Original comment by Daymansm...@gmail.com
on 10 Feb 2013 at 9:29
Huh. Using preallocation instead of sparse files actually throws
ERROR_IO_PENDING, so those two cases are really connected.
I made a trace (along with a trace from another thread, which may or may be not
helpful), but I cannot pinpoint the exact source of file alert produced,
probably comes from a thread. I'd be glad to set breakpoints, but I need the
exact location.
My guess would be that two or more threads are trying to write same blocks.
Original comment by Daymansm...@gmail.com
on 13 Feb 2013 at 2:27
Attachments:
Original comment by arvid.no...@gmail.com
on 13 Feb 2013 at 5:37
I realized I forgot to handle errors properly in my latest patch. Here's a fix
for that which I'm putting in right now:
Index: src/file.cpp
===================================================================
--- src/file.cpp (revision 8011)
+++ src/file.cpp (working copy)
@@ -1342,7 +1342,12 @@
CloseHandle(ol.hEvent);
return -1;
}
- WaitForSingleObject(ol.hEvent, INFINITE);
+ if (WaitForSingleObject(ol.hEvent, INFINITE) == WAIT_FAILED)
+ {
+ ec.assign(GetLastError(), get_system_category());
+ CloseHandle(ol.hEvent);
+ return -1;
+ }
DWORD num_read;
if (GetOverlappedResult(m_file_handle, &ol, &num_read, false) == 0)
{
@@ -1582,7 +1587,12 @@
CloseHandle(ol.hEvent);
return -1;
}
- WaitForSingleObject(ol.hEvent, INFINITE);
+ if (WaitForSingleObject(ol.hEvent, INFINITE) == WAIT_FAILED)
+ {
+ ec.assign(GetLastError(), get_system_category());
+ CloseHandle(ol.hEvent);
+ return -1;
+ }
DWORD num_written;
if (GetOverlappedResult(m_file_handle, &ol, &num_written, false) == 0)
{
Also, setting a breakpoint in storage::set_error() may pin-point where the
error is coming from.
Original comment by arvid.no...@gmail.com
on 13 Feb 2013 at 5:59
I'm attaching the trace with BP in storage.cpp before the link gets lost in IRC
logs.
Original comment by Daymansm...@gmail.com
on 13 Feb 2013 at 7:35
Attachments:
I'm attaching stats from disk-stats=on logging=default build
Original comment by Daymansm...@gmail.com
on 6 Aug 2013 at 10:49
Attachments:
We are still getting users reporting this problem with 0.16.17:
http://forum.deluge-torrent.org/viewtopic.php?f=12&t=48473&p=201695#p201695
What do you need in order to fix this issue?
Original comment by caluml...@gmail.com
on 23 Sep 2014 at 7:32
The solution works for me, but please fix this annoying issue.
Original comment by john.a...@gmail.com
on 24 Sep 2014 at 8:20
which solution works for you?
my understanding is that all proposed solutions in this thread made it into
libtorrent quite a while ago.
Original comment by arvid.no...@gmail.com
on 24 Sep 2014 at 8:27
I think he meant that the archive/unpack solution (workaround).
As reported above, Deluge is still experiencing this issue in latest versions
using 0.16.17.
Original comment by jackofal...@gmail.com
on 17 Oct 2014 at 6:19
Client: 1.3.9
libtorrent: 0.16.18.0
Still happening...
Original comment by kokau...@gmail.com
on 11 Nov 2014 at 2:30
Client: Deluge 1.3.10
libtorrent: 0.16.18
Still happening.
Original comment by old...@gmail.com
on 17 Nov 2014 at 12:34
people who experience this, do you have disk_io_write_mode set to
disable_os_cache by any chance? or disable_os_cache_for_aligned_files?
If so, would you mind trying out setting it to enable_os_cache and see if that
solves it?
Original comment by arvid.no...@gmail.com
on 18 Nov 2014 at 3:12
How do you change these options in windows?
Original comment by old...@gmail.com
on 18 Nov 2014 at 7:32
That is the case in Deluge:
http://git.deluge-torrent.org/deluge/tree/deluge/core/core.py?h=1.3-stable#n102
To change it you would need to download and modify that file then drop it into
relevant location of Deluge install.
Arvid what is the recommendation for that setting in >=0.16? I know it was
changed to prevent memory issue in 0.15:
https://code.google.com/p/libtorrent/issues/detail?id=166
Original comment by caluml...@gmail.com
on 18 Nov 2014 at 9:24
Actually just remembered that using the ltconfig plugin should allow you to
change these settings too.
Original comment by caluml...@gmail.com
on 18 Nov 2014 at 9:28
the main reason for these settings to exist was because of windows causing
excessive caching of files. This was fixed in 0.16.x.
http://blog.libtorrent.org/2012/05/windows-disk-cache/
Original comment by arvid.no...@gmail.com
on 18 Nov 2014 at 9:36
I am trying to use Ltconfig since I don't know how to use git link.
Using ltconfig I have line disk_io_write_mode with value 2. Should I try value
1 instead?
Thanks
Original comment by old...@gmail.com
on 18 Nov 2014 at 9:39
No it should be 0
http://www.rasterbar.com/products/libtorrent/manual.html#session-customization
enable_os_cache = 0,
disable_os_cache_for_aligned_files = 1,
disable_os_cache = 2
Original comment by caluml...@gmail.com
on 18 Nov 2014 at 9:52
Offtopic: And because of the ever-growing RAM usage because of OS
caching(Windows) qbittorrent gave in and exposes an option for users to disable
os cache. The problem(growing RAM usage) mainly manifested with seeders that
had a lot of traffic.
Original comment by hammered...@gmail.com
on 18 Nov 2014 at 12:22
Setting enable_os_cache through ltconfig apparently solved the problem. I hope
that now I won't have a RAM eater as well.
Original comment by old...@gmail.com
on 18 Nov 2014 at 2:16
if you're on a recent 0.16.x version, you shouldn't have that problem. (the
details are in the blog post, and it was fixed in the 0.16.x branch). When this
was fixed, the default value for both of those were changed to 0 as well.
The option (and feature) has been removed from trunk, but still in 1.0.x.
I wonder if it would make sense to rip it out of 0.16.x and up too.
Original comment by arvid.no...@gmail.com
on 18 Nov 2014 at 10:09
No arvid. Users of qbt with 0.16.16(or 0.16.17) reported that ever-growing RAM
problem.
Usually Windows 7 users that also had very active seeding traffic. Builds with
disable_os_cache seemed to solve the problem, so I exposed it as a checkbox in
the advanced settings.
Original comment by hammered...@gmail.com
on 18 Nov 2014 at 10:34
hammered999: do you have a link to a recent report?
Original comment by arvid.no...@gmail.com
on 19 Nov 2014 at 5:54
@arvidn here are some really recent ones:
Comment 1:
https://github.com/qbittorrent/qBittorrent/issues/1457#issuecomment-68320007
Comment 2:
https://github.com/qbittorrent/qBittorrent/issues/1457#issuecomment-68439062
qbt v3.1.11 uses 0.16.18+svn10417
Original comment by hammered...@gmail.com
on 31 Dec 2014 at 1:22
Original issue reported on code.google.com by
Daymansm...@gmail.com
on 8 Jan 2013 at 1:22