search

qbittorrent / qBittorrent

qBittorrent BitTorrent client
https://www.qbittorrent.org
Other
28.68k stars 4.01k forks source link

Crashes at regular intervals with current libtorrent RC #17949

Open oorzkws opened 2 years ago

oorzkws commented 2 years ago

qBittorrent & operating system versions

qBittorrent version: v4.5.0beta1 (64-bit) Libtorrent version: 2.0.7.0 Qt version: 6.4.0 Boost version: 1.80.0 OpenSSL version: 3.0.5 zlib version: 1.2.12 zlib version: 1.2.12
 OS version: Windows Server 2022 Version 21H2 10.0.20348 x86_64

What is the problem?

This issue may be invalid if you don't support libtorrent RC

qBittorrent is regularly crashing while running in the background. This is happening approx every 4 hours.

Caught signal: SIGSEGV


 0# 0x00007FF7F502FE43 in qbittorrent
 1# 0x00007FF7F502F539 in qbittorrent
 2# 0x00007FF7F50299AF in qbittorrent
 3# 0x00007FF7F5750DA0 in qbittorrent
 4# 0x00007FF7F57A1718 in qbittorrent
 5# 0x00007FF7F5736030 in qbittorrent
 6# _chkstk in ntdll
 7# RtlVirtualUnwind2 in ntdll
 8# KiUserExceptionDispatcher in ntdll
 9# QBitArray::setBit in Qt6Core
10# QBitRef::operator= in Qt6Core
11# 0x00007FF7F51127C8 in qbittorrent
12# 0x00007FF7F5066D0B in qbittorrent
13# 0x00007FF7F506DEA9 in qbittorrent
14# 0x00007FF7F507BE5B in qbittorrent
15# QMetaCallEvent::placeMetaCall in Qt6Core
16# QObject::event in Qt6Core
17# QApplicationPrivate::notify_helper in Qt6Widgets
18# QApplication::notify in Qt6Widgets
19# QCoreApplication::notifyInternal2 in Qt6Core
20# QCoreApplicationPrivate::sendPostedEvents in Qt6Core
21# QWindowsGuiEventDispatcher::sendPostedEvents in Qt6Gui
22# QEventDispatcherWin32::processEvents in Qt6Core
23# QWindowsGuiEventDispatcher::processEvents in Qt6Gui
24# QEventLoop::exec in Qt6Core
25# QCoreApplication::exec in Qt6Core
26# 0x00007FF7F501A85E in qbittorrent
27# 0x00007FF7F5027D95 in qbittorrent
28# 0x00007FF7F5766637 in qbittorrent
29# 0x00007FF7F5733192 in qbittorrent
30# BaseThreadInitThunk in KERNEL32
31# RtlUserThreadStart in ntdll

Steps to reproduce

  1. Build 4.5.0beta1 against libtorrent RC
  2. ???
  3. Crash

Additional context

Built on bb2caabd613a51a25f1057f0093036132e565e16 with libtorrent RC.

Newlines are broken pasting from the crash info popup, by the way.

Log(s) & preferences file(s)

Log

glassez commented 2 years ago

@oorzkws Your stack trace is broken. Did you forget to put .pdb file in the folder from where you run qBittorrent?

oorzkws commented 2 years ago

@oorzkws Your stack trace is broken. Did you forget to put .pdb file in the folder from where you run qBittorrent?

The pdb is there unless it doesn't follow symlinks

Edit: Your comment may have pointed to part of the issue though, let me undo all the linkages and see if the crashes continue.

userdocs commented 2 years ago

current rc would be 2.0.8?

oorzkws commented 2 years ago

Hm, I was relying on the version relayed by qBittorrent but it was pulled from git ~2 days ago: https://github.com/oorzkws/qBittorrent/actions/runs/3328197105/jobs/5503913940

userdocs commented 2 years ago

Maybe you need to build it again

https://github.com/microsoft/vcpkg/commit/857f979c59c0e902b4d20fdfac8bba2977ba656c

oorzkws commented 2 years ago

Rebuilt on 2.0.8.0, will report back if things change.

oorzkws commented 2 years ago

Same thing, curious that the symbols aren't catching. qBittorrent has crashed Please file a bug report at http://bugs.qbittorrent.org and provide the following information:

qBittorrent version: v4.4.5 (64-bit) Libtorrent version: 2.0.8.0 Qt version: 6.4.0 Boost version: 1.80.0 OpenSSL version: 3.0.5 zlib version: 1.2.12 OS version: Windows Server 2022 Version 21H2 10.0.20348 x86_64 Caught signal: SIGSEGV

 0# 0x00007FF7EA840603 in qbittorrent
 1# 0x00007FF7EA83FCF9 in qbittorrent
 2# 0x00007FF7EA83A16F in qbittorrent
 3# 0x00007FF7EAF628B0 in qbittorrent
 4# 0x00007FF7EAFB30C8 in qbittorrent
 5# 0x00007FF7EAF47B40 in qbittorrent
 6# _chkstk in ntdll
 7# RtlVirtualUnwind2 in ntdll
 8# KiUserExceptionDispatcher in ntdll
 9# QBitArray::setBit in Qt6Core
10# QBitRef::operator= in Qt6Core
11# 0x00007FF7EA9230A8 in qbittorrent
12# 0x00007FF7EA87753B in qbittorrent
13# 0x00007FF7EA87E6C9 in qbittorrent
14# 0x00007FF7EA88C6CB in qbittorrent
15# QMetaCallEvent::placeMetaCall in Qt6Core
16# QObject::event in Qt6Core
17# QApplicationPrivate::notify_helper in Qt6Widgets
18# QApplication::notify in Qt6Widgets
19# QCoreApplication::notifyInternal2 in Qt6Core
20# QCoreApplicationPrivate::sendPostedEvents in Qt6Core
21# QWindowsGuiEventDispatcher::sendPostedEvents in Qt6Gui
22# QEventDispatcherWin32::processEvents in Qt6Core
23# QWindowsGuiEventDispatcher::processEvents in Qt6Gui
24# QEventLoop::exec in Qt6Core
25# QCoreApplication::exec in Qt6Core
26# 0x00007FF7EA82AC4E in qbittorrent
27# 0x00007FF7EA838553 in qbittorrent
28# 0x00007FF7EAF78147 in qbittorrent
29# 0x00007FF7EAF44CA2 in qbittorrent
30# BaseThreadInitThunk in KERNEL32
31# RtlUserThreadStart in ntdll

Is there a way I can configure it to dump on crash?

Chocobo1 commented 2 years ago

The pdb is there unless it doesn't follow symlinks

Just to clarify, .pdb file should not be renamed nor replaced by symlink (nor moved to a different directory). It must be the same regular file and at the same place as the qbt executable.

oorzkws commented 2 years ago

I undid all the symlinks and am still seeing the same crashes. Trace is the same as above, no symbols.

Chocobo1 commented 2 years ago

I undid all the symlinks and am still seeing the same crashes. Trace is the same as above, no symbols.

Strange, crash dump is fine for me when triggered with an artificial crash. Anyway, I guess you'll have to hook it up with a debugger.

oorzkws commented 2 years ago 
(9660.424): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
Qt6Core!QBitArray::setBit+0x48:
00007ffa`dd6dd8e8 884201          mov     byte ptr [rdx+1],al ds:00007ffa`dda1fa5c=00

image Sorry, I'm unable to copy from this window - my windbg skills aren't the best.

glassez commented 2 years ago

@oorzkws Do you use release or debug mode when build qBittorrent?

oorzkws commented 2 years ago

@oorzkws Do you use release or debug mode when build qBittorrent?

The actions file is copied (almost) verbatim from the CI on the main repo, so I would presume release?

      - name: Build qBittorrent
        run: |
          cmake `
            -B build `
            -G "Ninja" `
            -DCMAKE_BUILD_TYPE=RelWithDebInfo `
            -DCMAKE_CXX_FLAGS="/WX" `
            -DCMAKE_EXPORT_COMPILE_COMMANDS=ON `
            -DCMAKE_TOOLCHAIN_FILE="${{ env.RUNVCPKG_VCPKG_ROOT }}/scripts/buildsystems/vcpkg.cmake" `
            -DBOOST_ROOT="${{ env.boost_path }}" `
            -DLibtorrentRasterbar_DIR="${{ env.libtorrent_path }}/lib/cmake/LibtorrentRasterbar" `
            -DMSVC_RUNTIME_DYNAMIC=OFF `
            -DQT6=ON `
            -DVCPKG_TARGET_TRIPLET=x64-windows-static-release `
            -DVERBOSE_CONFIGURE=ON `
            --graphviz=build/target_graph.dot
          cmake --build build --target qbt_update_translations
          cmake --build build
glassez commented 2 years ago

@oorzkws Well, in any case, it would be nice to know more details about the accompanying circumstances. As far as I understand, you use this build in your productive environment, right? How many torrents do you have in a session (as far as I remember, a lot)? How many of them are downloading (i.e. incomplete)? How many torrents (added using magnet link) still don't have downloaded metadata? Do you add/remove torrents or did all the above crashes happen with the same torrent set?

It also seems mysterious to me that it happens after some more or less certain time from the moment of launch. I wonder what it could be related with?

glassez commented 2 years ago

I believe I detect the local cause of reported crash so I can fix it. But the problem is that it most likely indicates a logical error somewhere in the code, so it would be highly desirable to detect it too.

xavier2k6 commented 2 years ago

@glassez

Going by the OP's log, he has over 35k torrents & very few performance warnings...nothing related to failures etc.

One thing that I do note from the log was that it states 4.4.5 in portable mode, original crash report was for 4.5.0beta1 & then second crash report is for 4.4.5 but with all the latest dependencies......

something doesn't seem right here....

Is there any crash dumps related to qBittorrent in %LOCALAPPDATA%\CrashDumps?

oorzkws commented 2 years ago

Sorry I compiled with a fake version because a tracker whitelists clients in a restrictive fashion. Nothing in CrashDumps, I do have WER dumps on.

oorzkws commented 2 years ago

@oorzkws Well, in any case, it would be nice to know more details about the accompanying circumstances. As far as I understand, you use this build in your productive environment, right? How many torrents do you have in a session (as far as I remember, a lot)? How many of them are downloading (i.e. incomplete)? How many torrents (added using magnet link) still don't have downloaded metadata? Do you add/remove torrents or did all the above crashes happen with the same torrent set?

It also seems mysterious to me that it happens after some more or less certain time from the moment of launch. I wonder what it could be related with?

36,166 as of writing. All have the private flag set, no magnets. DHT/PeX disabled. Around 10 added per day from a watch folder, maybe 1GB per day of total data. Around 100 downloading, though most (all, unless the above point has an active download) of those are stalled with a tracker error (deleted tracker side, requires manual cleanup) No torrents added manually, UI never opened for most of these crashes.

I'll edit in exact numbers once the client finishes loading this time. It's worth noting that I have 9 other clients on the same version that aren't adding downloads (seeding only, all torrents complete) that never crash.

Edit: Can't add full counts, it crashed before the recheck finished. I've stopped putting torrents in the watch folder and I'll see if the crashes stop.

glassez commented 2 years ago

@oorzkws Could you try #17963? It just prints some diagnostic info to the log and prevents app from being crashed at this point.

oorzkws commented 2 years ago

@oorzkws Could you try #17963? It just prints some diagnostic info to the log and prevents app from being crashed at this point.

(C) 2022-11-01T05:19:12 - Logical error encountered. Please report to developers. Torrent: 'Geoff Emberling - Social Theory in Archaeology and Ancient History [Retail].pdf'. Metadata exists: No. Native file index: 0. File index: -1.

Edited to add screenshot image

I haven't the foggiest why this is how it is, the torrent should both have a tracker and the private flag set.

glassez commented 2 years ago

@oorzkws

  1. So since it is private torrent it isn't allowed to download metadata from other peers or am I wrong?
  2. What does happen if close and start qBittorrent again?
  3. Do you use .fastresume files or SQLite?

I would be strongly interested to get the resume data of this torrent for investigating.

ghost commented 2 years ago

So since it is private torrent it isn't allowed to download metadata from other peers or am I wrong?

I guess this is a wrong assumption because I can easily download metadata from other peers on a private torrent provided it has a working tracker and at least one peer is connectable and has the metadata. So here it could be possible that none of the peers has metadata as I can only see peers and no seeds on the transfer list. Or it could be that none of the peers are connectable(NATed).

glassez commented 2 years ago

So since it is private torrent it isn't allowed to download metadata from other peers or am I wrong?

I guess this is a wrong assumption because I can easily download metadata from other peers on a private torrent

Okay, that's not the most important thing for me in this situation.

The processing of this torrent behaves not as expected, and I could sort it out by being able to reproduce it locally. Unfortunately, I have no more ideas how to do this, other than to try exactly the same resume data that @oorzkws has.

oorzkws commented 2 years ago

@glassez Sqlite. I've emailed you an exported partial database.

glassez commented 2 years ago

@glassez Sqlite. I've emailed you an exported partial database.

I'm sorry, the data you sent me is invalid (it may be exported incorrectly). The most reliable way is to convert your data to "fastresume" format, and then find the appropriate .fastresume and .torrent files there and send me.

P.S. Can I use the email address you send me files from to correspond with you?

P.P.S. Could you also send the file downloaded by this torrent?

oorzkws commented 2 years ago

P.S. Can I use the email address you send me files from to correspond with you?

P.P.S. Could you also send the file downloaded by this torrent?

Yes and yes. I'll convert from sqlite but that will probably take a significant amount of time - I'll send as soon as it finishes.

oorzkws commented 2 years ago

I'm not sure if it's related but it takes days to exit qBittorrent gracefully. It seems to be constantly writing to the sqlite database during, albeit at only a few KB/s.

Chocobo1 commented 2 years ago

I'm not sure if it's related but it takes days to exit qBittorrent gracefully. It seems to be constantly writing to the sqlite database during, albeit at only a few KB/s.

I'm interested in this. Which qbt version? Still at bb2caabd613a51a25f1057f0093036132e565e16 in the opening post?

oorzkws commented 2 years ago

I'm not sure if it's related but it takes days to exit qBittorrent gracefully. It seems to be constantly writing to the sqlite database during, albeit at only a few KB/s.

I'm interested in this. Which qbt version? Still at bb2caab in the opening post?

Now 3a2e73cc94fc5694beb3ef28a0f4557c4e93e0e3 with #17963 on top.

Chocobo1 commented 2 years ago

Now https://github.com/qbittorrent/qBittorrent/commit/3a2e73cc94fc5694beb3ef28a0f4557c4e93e0e3 with https://github.com/qbittorrent/qBittorrent/pull/17963 on top.

Does the slow exit also happens with bb2caabd613a51a25f1057f0093036132e565e16?

If not, then it could be due to PR #17900. If you can help with some debugging, please change https://github.com/qbittorrent/qBittorrent/blob/5f2d80786132eaf0569fa75e4937aafc57b3f302/src/base/bittorrent/sessionimpl.cpp#L2939-L2940 to 

LogMsg(u"xxx Alert type: "_qs + QString::fromUtf8(a->what()));  // add this line
handleAlert(a);

And post your log file after qbt exits. I'm interested with all the lines that start with xxx Alert type: as they might show what is stalling the exit loop (a recurring pattern, or something else).

oorzkws commented 2 years ago

Sounds good, I'll do that once I've produced the files for glassez.

oorzkws commented 2 years ago

Client has been loading for 24h now after removing all but the bugged torrent. Not sure if it'll finish.

glassez commented 2 years ago

Client has been loading for 24h now after removing all but the bugged torrent. Not sure if it'll finish.

Did I get it right? Have you deleted all torrents except the corrupted ones, closed qBittorrent, then started it again and it has been trying to start for 24 hours now? This is hardly a regular situation. I wouldn't keep waiting.

I clearly need to investigate the resume data of this torrent.

oorzkws commented 2 years ago

Client has been loading for 24h now after removing all but the bugged torrent. Not sure if it'll finish.

Did I get it right? Have you deleted all torrents except the corrupted ones, closed qBittorrent, then started it again and it has been trying to start for 24 hours now? This is hardly a regular situation. I wouldn't keep waiting.

I clearly need to investigate the resume data of this torrent.

That's the process I followed, yeah. The database also hasn't changed size which is curious, should I just upload the whole db (~350mb)?

glassez commented 2 years ago

The database also hasn't changed size which is curious

SQLite doesn't compress file automatically (by default). You can try to perform VACUUM command on it using some SQLite management tool or just send it me via some service that allows you to send 350mb file (or try to compress it with 7zip etc.).