BSODs - Githubissues

ameshkov commented 6 years ago

ID	Windows version	Antivirus software (with version)	AdGuard version	Minidumps	Other links	Comment
1	Win 10 x64 v1803	Kaspersky	6.3.974.3223 Beta		original issue	analysis
2	Win 10	ESET 11.2.49 SSL scanning enabled	6.3 Release	DumpAnalysis.txt.zip	forum	-
3	Win7x64 7601	ESET SS - 9.0.429.1	6.3.1399.4073	dump		Выбило при работе в Firefox 55.0.3
4	Win 10 Pro x64	Eset Nod Antivirus 11.2.49.0	6.3.1399.4073	dump	ID 1978569	-

prolium commented 6 years ago

Continuing to the issue #2276, a BSOD occurred again with the latest version 6.3.1374.4023 RC. Should I upload the minidump and kernel dump files here ?

vozersky commented 6 years ago

@wk-952 you can use this url to upload them: https://www.dropbox.com/request/N2oxbmPcYJ0fpAodbMoL

ameshkov commented 6 years ago

@wk-952 another option would be to send it to devteam@adguard.com

prolium commented 6 years ago

@vozersky It's asking for a valid e-mail address, is that important/necessary ? Sorry, I didn't use dropbox before.

EDIT: I'll send a MEGA link to the above-mentioned e-mail address.

vozersky commented 6 years ago

Oh, not sure. You can just go ahead and upload it somewhere comfortable and then send the url to devteam@adguard.com Or send them directly

prolium commented 6 years ago

All right, I've sent the e-mail with the link to both the minidump and kernel dump files.

ameshkov commented 6 years ago

Regarding issue ID=1

@wk-952

Dump analysis:

The issue arises somewhere deep inside NETIO/TCPIP probably due to a corrupted TCP context.
The problematic connection belongs to qbittorrent.exe, endpoint 124.183.43.143:256
The driver properly handles this connection and issues a command to not filter this connection.

It's hard to say what's causing it but there's no mistake in what's done by the driver.

In theory, the issue could be caused by one of the following:

Kaspersky WFP driver
Stack size limit might be not big enough as you have two WFP drivers + enabled driver verifier.

Possible solutions:

Disable driver verifier and see how it works without it. OR
Disable Teredo. OR
Disable AdGuard for qbittorrent.exe (settings -> filtered apps)

prolium commented 6 years ago

qBittorrent was already removed from filtered apps before this issue occurred, although I have to say, it seems that the crash occurs more frequently when qBittorrent is kept downloading the whole night. ~~Why is AdGuard still filtering connections from it ? That seems odd!~~

Teredo seems to be already disabled, this is what i get from CMD:

> netsh interface teredo show state
Teredo Parameters
---------------------------------------------
Type                    : disabled
Server Name             : win1711.ipv6.microsoft.com.
Client Refresh Interval : 30 seconds
Client Port             : unspecified
State                   : offline
Error                   : none

Regarding driver verifier, as mentioned here, the issue happened many times before driver verifier was turned on, actually this crash and another one from a faulty LAN driver from Killer Network (already solved now) were both the reason I had to turn on driver verifier.
Not sure about Kaspersky though, I'm aware that it's very common to have issues with it, but I'm not sure how to verify this. If you have any suggestions please tell me.

I will keep Kaspersky on, but will disable driver verifier along with AdGuard's service for some days, if a crash occurred then it's highly probable a problem in Kaspersky. Otherwise, I'll keep driver verifier turned off, disable Kaspersky, and leave AdGuard's service tuned on, if a crash occurred then it's definitely a problem in AdGuard. If none of that worked, then the only reasonable assumption would be that the mix of Kaspersky along with AdGuard is somewhat impossible.

If you have other suggestions or info, please tell me. Thanks for the support.

Sorrovv commented 6 years ago

Since I updated to AdGuard for Windows 6.3 for this issue, the system has crashed with BSODs several times. Even I disabled WFP driver, the system still crashed with BSODs. I try to get the minidump file but I can't find Windows 10 BSOD dmp files and no folder C:\WINDOWS\Minidump exists. The system crashed too often, so I have no choice but to uninstall AdGuard for Windows 6.3 RC. By contrast, AdGuard for Windows 6.2 doesn't cause BSODs.

Platform: Windows 10 version 1803 64-bit Antivirus software: Kaspersky Internet Security 18.0.0.405(h) AdGuard version: 6.3.1374.4023 RC

prolium commented 6 years ago

@Sorrovv You may find a large crash dump file (around 1.5GB) in C:\Windows named MEMORY.DMP. If you found that file, try compressing it with 7-zip, with the following parameters:

Pic

![image](https://user-images.githubusercontent.com/11916365/42525310-1d6745cc-8473-11e8-8c51-ec44a9bc9e3d.png)

You'll get a decent compression ratio (around 12%), so the output/compressed file would be ~100MB, then you can send it to above-mentioned e-mail by @ameshkov and @vozersky.

Sorrovv commented 6 years ago

@wk-952 I wish I could help, but now I can't find the memory dump file either. I think it may have been deleted by CCleaner. On the other hand, it seems to me that I don't use virtual memory, so minidump files can not be created. Thanks for your reply.

prolium commented 6 years ago

So, I've tried the following: 1) Disabled driver verifier, but left Kaspersky and AdGuard running. That resulted in the same crash. 2) Disabled driver verifier and Kaspersky (also made sure its system hooks are disabled), but only left AdGuard running. That also resulted in the same crash.

EDIT: In both cases qBittorrent was running.

Should I upload/send those 2 dumps ?

Something that seemed interesting to me was that the RVA of the instruction (at crash) as reported by WhoCrashed was always consistent adgnetworkwfpdrv+0xCC0E, here's the full report:

Log

``` Crash dump directories: C:\Windows C:\Windows\Minidump On Thu 12/07/2018 9:40:56 AM your computer crashed or a problem was reported crash dump file: C:\Windows\Minidump\071218-24031-01.dmp This was probably caused by the following module: ntoskrnl.exe (nt+0x198430) Bugcheck code: 0x3B (0xC0000005, 0xFFFFF80BE0E0A5C7, 0xFFFFF404936CE270, 0x0) Error: SYSTEM_SERVICE_EXCEPTION file path: C:\Windows\system32\ntoskrnl.exe product: Microsoft® Windows® Operating System company: Microsoft Corporation description: NT Kernel & System Bug check description: This indicates that an exception happened while executing a routine that transitions from non-privileged code to privileged code. This appears to be a typical software driver bug and is not likely to be caused by a hardware problem. The crash took place in the Windows kernel. Possibly this problem is caused by another driver that cannot be identified at this time. On Thu 12/07/2018 9:40:56 AM your computer crashed or a problem was reported crash dump file: C:\Windows\MEMORY.DMP This was probably caused by the following module: adgnetworkwfpdrv.sys (adgnetworkwfpdrv+0xCC0E) Bugcheck code: 0x3B (0xC0000005, 0xFFFFF80BE0E0A5C7, 0xFFFFF404936CE270, 0x0) Error: SYSTEM_SERVICE_EXCEPTION file path: C:\Windows\system32\drivers\adgnetworkwfpdrv.sys product: Adguard for Windows company: description: Adguard WFP network driver x64 for Windows 8 and later Bug check description: This indicates that an exception happened while executing a routine that transitions from non-privileged code to privileged code. This appears to be a typical software driver bug and is not likely to be caused by a hardware problem. A third party driver was identified as the probable root cause of this system error. It is suggested you look for an update for the following driver: adgnetworkwfpdrv.sys (Adguard WFP network driver x64 for Windows 8 and later, ). Google query: adgnetworkwfpdrv.sys SYSTEM_SERVICE_EXCEPTION ```

Another thing, I have VMware Workstation which has its own network driver, does it add its own WFP driver ?

ameshkov commented 6 years ago

Disabled driver verifier and Kaspersky (also made sure its system hooks are disabled), but only left AdGuard running. That also resulted in the same crash.

Have you actually uninstalled Kaspersky? Simply disabling it might be not enough, their driver can be still registered.

Another thing, I have VMware Workstation which has its own network driver, does it add its own WFP driver ?

No, that shouldn't be a problem.

ameshkov commented 6 years ago

I guess I know how to resolve this issue on our side at least partly -- we should move the code that is responsible for checking new connections to the driver. Currently, it notifies the user mode and the decision (continue filtering or not) is made there. This logic adds complexity and as your case shows it may cause troubles indirectly.

Regarding the dumps, if the issue is reproducible with Kaspersky uninstalled, I'd like to take a look at the dump.

If it's not, then it'd be better to wait for an updated driver.

Also, there's one more thing that could be a temporary solution: tray menu -> advanced -> settings -> uncheck "use localhost for injections". That checkbox makes AG intercept connections on a later stage (after the BSOD happens in your case).

Sorrovv commented 6 years ago

Hello. Since AdGuard for Windows 6.3 has officially released, I updated to version 6.3.1399.4073, and the system still crashed. I found the dmp files this time and I've sent an email to devteam@adguard.com.

ameshkov commented 6 years ago

@Sorrovv received your minidump, thank you! Meanwhile, an updated driver is likely to come this week (https://github.com/AdguardTeam/AdguardForWindows/issues/2246#issuecomment-404524111)

Sorrovv commented 6 years ago

@ameshkov I'm glad to hear that. Thank you for your hard work.

vozersky commented 6 years ago

added 1 more issue

ameshkov commented 6 years ago

@Sorrovv @wk-952 Guys, here is a test build for both of you: https://uploads.adguard.com/Setup_nightly_6.3.1435.exe

It contains changes from my comment above plus it might help with the issue @Sorrovv is facing (not 100% guarantee though, but we need to see if it helps).

prolium commented 6 years ago

@ameshkov I should keep the driver verifier disabled, right ?

ameshkov commented 6 years ago

@wk-952 yep, it'd be better to keep it disabled

Sorrovv commented 6 years ago

@ameshkov It has been a day and this test build doesn't cause BSODs so far. It seems to me that it really helps to solve this issue. Thank you.

ameshkov commented 6 years ago

@Sorrovv thank you! Fingers crossed, let's see how it goes on monday

Sorrovv commented 6 years ago

@ameshkov It's already Monday. There's no BDOD and everything seems to be OK. Thank you!

ameshkov commented 6 years ago

Awesome, thank you for testing!

prolium commented 6 years ago

I left the PC running since 20/7 with both Kaspersky and AdGuard running, also I let qBittorrent download during the whole period. No BSOD occurred. Thank you very much.

skipik commented 6 years ago

@ameshkov Этот фикс может влиять на скорость открытия страниц? Просто на предыдущей ночной (1400) всё было довольно быстро, а на 1435 прямо заметно медленнее.

ameshkov commented 6 years ago

@wk-952 thank you!

@skipik в нем изменен порядок слоев в WFP драйвере, что, в теории, влиять на скорость не должно. В инструментах разработчика видно какой именно запрос теперь медленнее работает?

skipik commented 6 years ago

@ameshkov На фейсбуке дольше всех грузится https://www.facebook.com/ajax/bz. Визуально стало заметно, что после обновления AdGuard 1400 -> 1435 все страницы медленнее отрисовываться стали. Если нужно какой-то лог записать\прислать, то дайте знать - сделаю.

ameshkov commented 6 years ago

@skipik нужно два HAR-файла, записанных с релизной версией и с последней найтли.

https://support.zendesk.com/hc/en-us/articles/204410413-Generating-a-HAR-file-for-troubleshooting

skipik commented 6 years ago

@ameshkov Готово, отправил на почту devteam. Могу сказать, что визуально прямо чувствуется, как браузер "задышал", надеюсь, что логи это подтвердят. :)

ameshkov commented 6 years ago

@skipik I was able to reproduce the slowdown issue with KIS18. It seems that it does not work well in this configuration (the only way to make them work together in this nightly is to uncheck "use localhost injections" in the advanced settings).

ameshkov commented 6 years ago

@wk-952 @skipik @Sorrovv

Guys, another nightly that is supposed to address the Kaspersky issue: https://uploads.adguard.com/Setup_nightnly_6.3.1446.exe

There're some serious changes inside the WFP driver so I am afraid it needs to be tested

BlagoYar commented 6 years ago

Не знаю, виноват ли в этом AdGuard (не особо разбираюсь в минидампах) http://multi-up.com/1204669

Windows 7 x64 AdGuard 6.3.1339.4073

ameshkov commented 6 years ago

@BlagoYar нет, тут что-то с драйвером файловой системы

fffff880`037326f8 fffff800`033bd4b3 : 00000000`00000019 00000000`00000003 fffff800`0341a940 00000000`00000000 : nt!KeBugCheckEx
fffff880`03732700 fffff800`0335e235 : fffff880`00000000 fffff880`03732890 00000000`00000000 fffff800`00000000 : nt!ExFreePool+0x4fb
fffff880`037327f0 fffff800`032f0f21 : fffffa80`06406c60 00000000`00000000 fffff880`018b9d80 fffffa80`0e8de400 : nt!FsFilterAllocateCompletionStack+0x35
fffff880`03732830 fffff800`032c06d7 : 00000000`00000000 fffff800`00000280 fffffa80`06460030 fffff880`018b9d80 : nt! ?? ::FNODOBFM::`string'+0x22ff1
fffff880`03732860 fffff800`032c04ed : 00000000`00001000 fffff880`03732ba8 fffffa80`03578bb0 fffffa80`00000000 : nt!FsRtlAcquireFileForModWriteEx+0xa3
fffff880`03732b10 fffff800`032c0988 : 00000000`00000000 fffffa80`0e8de4d0 fffffa80`00000000 fffff8a0`032d2018 : nt!MiGatherMappedPages+0x735
fffff880`03732c10 fffff800`03521df6 : fffffa80`03d10660 00000000`00000080 fffffa80`03cedb10 eb026d8d`44000efc : nt!MiMappedPageWriter+0x198
fffff880`03732d00 fffff800`032786e6 : fffff800`03405e80 fffffa80`03d10660 fffffa80`03d10b50 80850fc0`85fffdf9 : nt!PspSystemThreadStartup+0x5a
fffff880`03732d40 00000000`00000000 : fffff880`03733000 fffff880`0372d000 fffff880`03732890 00000000`00000000 : nt!KxStartSystemThread+0x16

skipik commented 6 years ago

@ameshkov It seems that slowdown problem is fixed in this new 1446 build.

prolium commented 6 years ago

The nightly build 6.3.1446 is stable for me, PC was running for almost 2 days with Kaspersky enabled. No BSODs so far.

EDIT: I've upgraded Kaspersky yesterday to 2019, I'll report in 2 days from now if it causes BSOD.

vadimplSPb commented 6 years ago

Ночные версии достаточно ставить поверх предыдущей, или сначала необходим полный деинсталл? (На двух компах примерно месяц сплошные BSOD, направили в эту ветку)

skipik commented 6 years ago

@vadimplSPb Настройки - общие - поменять канал обновлений на Nightly и зайти в "О программе" после этого.

ameshkov commented 6 years ago

@wk-952 awesome, then there's a great chance that we'll include this new driver version in the hotfix update.

Guys, once the issue is confirmed to be resolved, we will clean up the thread (comments will be backed up) and mark issues 1 and 2 as resolved.

Sorrovv commented 6 years ago

I've updated Adguard for Windows to the latest nightly build and upgraded Kaspersky Internet Security to version 2019 four days ago and there's no BDOD so far. Thank you.

vadimplSPb commented 6 years ago

Аналогично, более суток последняя ночная + KIS2018, включая опции WFP-драйвера и фильтрации https. Ни одного BSOD (и на сертификаты сайтов не ругается). Нехорошо, что техподдержка, фактически, перемещается из официльного ветки форума напрямую к разработчикам. Если уж разработчки напрямую общаются с юзерами (а хорошо ли это?), то пусть это будет в одном месте.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

AdguardTeam / AdguardForWindows

BSODs #2246

Dump analysis: