Closed Digiover closed 2 years ago
Same problem for me: https://github.com/microsoft/msquic/issues/2188. I had reported this to MSRC but it was closed as it wasn't a security issue.
Yes, we've decided it wasn't a big enough issue to be a security issue. It happened to already be fixed on the latest versions. We'll likely still push a fix to older versions soon enough. But if there's ever a chance of a crash being a security issue, please don't open it on GitHub and follow the security instructions to report it to MSRC.
Yes, we've decided it wasn't a big enough issue to be a security issue. It happened to already be fixed on the latest versions. We'll likely still push a fix to older versions soon enough. But if there's ever a chance of a crash being a security issue, please don't open it on GitHub and follow the security instructions to report it to MSRC.
Thanks. How do we know this is a security issue? Or are all access violations (c0000005) security issues?
(you may close this issue with a comment)
Yes, we've decided it wasn't a big enough issue to be a security issue. It happened to already be fixed on the latest versions. We'll likely still push a fix to older versions soon enough. But if there's ever a chance of a crash being a security issue, please don't open it on GitHub and follow the security instructions to report it to MSRC.
Well, it renders quic completely useless on WS2022 as the machine will crash, so it's a significant issue I'd say. When you talk about "latest versions", do you mean WS2022 or msquic? I have fully updated WS2022 and the issue still exists. What would be the timeline to get this fixed on WS2022? If this is taking long, is there a way to manually update msquic driver? I only see the DLLs available on releases. And which commit has fixed this bug?
How do we know this is a security issue? Or are all access violations (c0000005) security issues?
Any crash has a good chance of being a security issue if we can figure out how it might be triggered remotely, or even if it can't but there's a good chance of it happening randomly (i.e. race condition).
It seems to me then that it is a security issue? As it can be triggered remotely with HTTP requests. Also: what about my questions above? Thank you.
It can't be easily exploited on purpose by an attacker though. That's why it's not significant enough to be considered a security issue.
Also, feel free to email me if you'd like me to look at any future issues to let you know if it could be a security issue, if you're unsure.
Alright, fair enough.
When you talk about "latest versions", do you mean WS2022 or msquic? I have fully updated WS2022 and the issue still exists. What would be the timeline to get this fixed on WS2022? If this is taking long, is there a way to manually update msquic driver? I only see the DLLs available on releases.
The latest MsQuic version. We still plan to service WS2022 with the fix. Just got to work out the logistics. There's no way for someone to update Windows drivers out of band right now.
@nibanks Please keep us posted on the WS2022 roll-out for the fix.
Will do.
So now that it's closed, any update on roll out date @nibanks?
I'm working on it. 😄 I will definitely let you know when I have a roll out date.
@digiover and @nuklon we got the fix approved for servicing and should hopefully be available in a month or two.
Great, thanks @nibanks. Will you provide an update when it's actually serviced? So we don't have to guess.
Definitely.
FYI @Digiover and @Nuklon the fix should be out with the Windows April monthly update, next month. It will require manual update, or will automatically happen with the following month's update (as I understand it). When it all goes live, we will share instructions.
FYI @Digiover and @Nuklon the fix should be out with the Windows April monthly update, next month. It will require manual update, or will automatically happen with the following month's update (as I understand it). When it all goes live, we will share instructions.
Thanks @nibanks! Any information on how to update, or is it wrapped in a Window Update package for this month?
msquic.sys seems to be updated with latest KB, it's now signed March 8, and version 1.0.4. Some clarification from @nibanks on whether the fix is in place is welcome 👀
Yes @nuklon, that should be correct. I believe the version should be 1.0.4.233914-official
in the Product version of the .sys file.
Thanks @nibanks, 2 days stable now 👍
Not sure if this is related to this fix or not, so if you'd like me to open a new issue, I can do so.
Since running this new version, I've had to restart the server twice due to memory getting to 100%. I've looked into it with poolmon and one tag uses ~32GB of RAM:
Memory:67018228K Avail: 270612K PageFlts: 9355 InRam Krnl:47756K P:390536K
Commit:72832896K Limit:84635012K Peak:80802964K Pool N:37417736K P:635236K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc
Qc1A Nonp 92404489 ( 0) 6827 ( 0) 92397662 34002339616 ( 0) 368
Sorry for bad formatting, I couldn't get this formatted correctly. Here's the screenshot:
Since the server worked correctly before without any problems, I suspected it was due to msquic, and looking at the code it seems this tag is indeed used by msquic: https://github.com/microsoft/msquic/blob/main/src/inc/quic_platform.h#L96
I do not have a memory dump. I tried creating one with notmyfault but the server crashed/rebooted immediately without writing any dump, and I have since disabled quic in IIS again.
My setup is still the same, ASP.NET Core 6.0, IIS, WS2022 latest updates. I have also disabled legacy TLS on most sites.
@nuklon please open a new issue for this. I've never seen it before. This seems to indicate that sends at the UDP layer are getting leaked. Since the sends are unconditionally freed (at least to the pool) in the completion path (CxPlatDataPathSendComplete
to CxPlatSendDataFree
), it must mean the IRP isn't getting completed.
cc @thhous-msft
@Nuklon also please include information about your NIC HW and driver version with the new issue. It could be a driver issue, not returning/completing the UDP send NBLs.
Thanks @nibanks, I opened #2658, let's continue there.
Describe the bug
After enabling HTTP3 and AltSvc response headers in the Windows registry (per instructions), the server occasionally reboots caused by a bugcheck.
Affected OS
Additional OS information
Also available are IIS, ASP.NET 4.8, .NET 5.0 & 6.0, .NET Core 3.1, PHP 8.1.1, 8.0.13 and 7.4.26 (all PHP is FastCgi)
MsQuic version
main
Steps taken to reproduce bug
C:\Windows
.Expected behavior
The server should not reboot
Actual outcome
I'm no WinDbg pro, but this is what
!analyze -v
gave me:Additional details
There is not much HTTP3 web traffic on the web server, yet. According to our
\quic performance diagnostics\quic connections connected
and\quic performance diagnostics\quic streams active
Zabbix monitoring, "quic connections connected" maxed at 20 and 119 "quic streams active".