Open 7MinSec opened 2 years ago
@7MinSec The only time I've seen this happen is with an EDR. Apparently there was a "feature" where if it detected credential dumping the EDR killed the whole DC to stop it. So if you know of any security products they use on the DC I'd start there.
The symptoms you mentioned describe it perfectly - the dump went for a while before the protection triggered.
@jagotu thank you for this insight. I'm looking to get the EDR logs now to see if it looks like that played a part. I did get a peek at the event logs, and I believe this one is super telling:
EventID 1074:
The process wininit.exe has initiated the restart of computer XXX on behalf of user for the following reason: No title for this reason could be found
Reason Code: 0x50006
Shutdown Type: restart
Comment: The system process 'C:\Windows\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart.
In doing some Google-Fu, most of the buzz around some of these search terms point back to a faulty Jan 2022 update (http://www.edugeek.net/forums/windows/225791-looks-like-january-windows-updates-causing-all-sorts-issues.html).
I'll report back when I know about the EDR situation.
Update: no EDR installed on the DC.
Interesting. I've had the exact same issue as you (lsass crash to reboot after ~30 seconds of successful dumping) once and thought it was EDR, but seems maybe it actually wasn't 🤔
I'll try to look up my notes but probably will not be able to add more than my anecdotal "I've seen it happen, too" report.
Just wanted to report, FWIW, that I had this happen again tonight in a different environment with a single DC running a fully patched Server 2019 Standard. The hash dump was humming along and got roughly half way through and then timed out. When I used CrackMapExec to auth to the domain controller, it failed.
When the client remoted in to the DC, they saw this:
They clicked ok, the DC rebooted, and everything came up ok. I could then use CME to auth with my account, and the domain continued functioning as normal (besides giving me a heart attack).
Like I mentioned earlier, I couldn't find any smoking gun to point to except event 1074 in the System log:
The process wininit.exe has initiated the restart of computer XXX on behalf of user for the following reason: No title for this reason could be found
Reason Code: 0x50006
Shutdown Type: restart
Comment: The system process 'C:\Windows\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart.
I have seen the exact same issue, same events in the eventlog, on different 2 occasions. Not directly running secretsdump, but using "crackmapexec smb dc01 -u user -p pass --ntds", from Kali. Both DC's ran Win2012R2.
I'm having the same issue here.
I idid however had different DCs where I could test this on. Note that I tested both "cme --ntds" and secretsdump.py.
The DCs that the full dump was possible on were normal Server 2012 R2 DCs (normal setup). The DCs (tested on 2) that were rebooting with the same error messages as above were both Server 2019 and were hosted on Azure.
I was unable to determine the reason of the reboot. No real smoking gun on the system logs as well.
Having the same issue here. Haven't noticed it until recently but it may have been happening for a while. It happened on both server 2019 and 2016, in two separate environments. Neither were in Azure as far as I know. I suspect EDR but when attempting to lab out the environments, could not replicate...
@0xdeaddood @bugch3ck I'm not convinced this is a bug in secretsdump.py but I will leave that up to you.
Here's some data from the lab. Hope it can help you troubleshoot.
Secretsdump Debug Output
Impacket v0.10.1.dev1+20230203.111903.32178de - Copyright 2022 Fortra
[+] Impacket Library Installation Path: /usr/local/lib/python3.11/dist-packages/impacket
[+] Saving output to dump
[*] Dumping Domain Credentials (domain\uid:rid:lmhash:nthash)
[*] Using the DRSUAPI method to get NTDS.DIT secrets
[+] Session resume file will be sessionresume_RGsKxyFm
[+] Calling DRSCrackNames for S-1-5-21-2468572375-1347652797-2136843564-500
[+] Calling DRSGetNCChanges for {1b66e962-5836-4d1c-aed0-15eb15e93e8a}
[+] Entering NTDSHashes.__decryptHash
[+] Decrypting hash for user: CN=Administrator,CN=Users,DC=crash,DC=lab
Administrator:500:<REDACTED>::: (status=Enabled)
[+] Leaving NTDSHashes.__decryptHash
[+] Calling DRSCrackNames for S-1-5-21-2468572375-1347652797-2136843564-501
[+] Calling DRSGetNCChanges for {8990ecb4-b506-48d5-bec2-229ef4408270}
<-- SNIP -->
[+] Entering NTDSHashes.__decryptHash
[+] Decrypting hash for user: CN=employee16,CN=Users,DC=crash,DC=lab
CRASH.LAB\employee16:1118:<REDACTED>::: (status=Enabled)
[+] Leaving NTDSHashes.__decryptHash
[+] Calling DRSCrackNames for S-1-5-21-2468572375-1347652797-2136843564-1119
[+] Calling DRSGetNCChanges for {962a96f5-eb01-4eff-ae89-50655ca43f3e}
[-] [Errno 104] Connection reset by peer
[*] Something went wrong with the DRSUAPI approach. Try again with -use-vss parameter
[*] Cleaning up...
Windbg Crash Analysis
Microsoft (R) Windows Debugger Version 10.0.22621.755 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\CrashDumps\lsass.exe.600.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: srv*
Executable search path is:
Windows 10 Version 17763 MP (2 procs) Free x64
Product: LanManNt, suite: TerminalServer SingleUserTS
Edition build lab: 17763.1.amd64fre.rs5_release.180914-1434
Machine Name:
Debug session time: Thu Mar 2 18:19:40.000 2023 (UTC - 5:00)
System Uptime: not available
Process Uptime: 0 days 0:08:05.000
................................................................
.......................................................
Loading unloaded module list
..........
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(258.2b4): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForMultipleObjects+0x14:
00007ffa`65ff1344 c3 ret
0:028> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : AV.Fault
Value: Read
Key : Analysis.CPU.mSec
Value: 3077
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 68260
Key : Analysis.Init.CPU.mSec
Value: 1718
Key : Analysis.Init.Elapsed.mSec
Value: 59944
Key : Analysis.Memory.CommitPeak.Mb
Value: 164
Key : Timeline.Process.Start.DeltaSec
Value: 485
Key : WER.OS.Branch
Value: rs5_release
Key : WER.OS.Timestamp
Value: 2018-09-14T14:34:00Z
Key : WER.OS.Version
Value: 10.0.17763.1
Key : WER.Process.Version
Value: 10.0.17763.3532
FILE_IN_CAB: lsass.exe.600.dmp
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
CONTEXT: (.ecxr)
rax=0000022296a4d660 rbx=000000b10007e200 rcx=f22d45cd9728be6e
rdx=0000022296a50210 rsi=0000022296a4a8e0 rdi=0000022296a3d9e0
rip=00007ffa60d0f994 rsp=000000b10007dd30 rbp=000000b10007dd80
r8=000000000000000b r9=0000000000000058 r10=00007ffa60ea4f80
r11=00007ffa60ea4f80 r12=0000000000000000 r13=0000022293ca0700
r14=0000022293838810 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
ntdsai!draXlateNativeReplyToOutboundReply+0x150:
00007ffa`60d0f994 8b01 mov eax,dword ptr [rcx] ds:f22d45cd`9728be6e=????????
Resetting default scope
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffa60d0f994 (ntdsai!draXlateNativeReplyToOutboundReply+0x0000000000000150)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: ffffffffffffffff
Attempt to read from address ffffffffffffffff
PROCESS_NAME: lsass.exe
READ_ADDRESS: ffffffffffffffff
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: ffffffffffffffff
STACK_TEXT:
000000b1`0007dd30 00007ffa`60d007b3 : 00000000`000777be 00000000`00000005 00000000`00000000 00000000`00000a6c : ntdsai!draXlateNativeReplyToOutboundReply+0x150
000000b1`0007de50 00007ffa`65da8e93 : 00000222`93865350 00000000`00000008 00000222`96a3d8d0 00000222`93ca0700 : ntdsai!IDL_DRSGetNCChanges+0x1a83
000000b1`0007ea80 00007ffa`65d4dff3 : 00007ffa`60dec9d2 000000b1`0007eee0 00000000`00000003 00000222`93c99a20 : rpcrt4!Invoke+0x73
000000b1`0007eaf0 00007ffa`65d51a2a : 00000000`80090301 000000b1`0007f370 00000000`00000000 00007ffa`61e19a18 : rpcrt4!NdrStubCall2+0x3e3
000000b1`0007f130 00007ffa`65d80e18 : 000000b1`0007f1f0 00000222`00000001 000000b1`0007f180 00007ffa`61e198f3 : rpcrt4!NdrServerCall2+0x1a
000000b1`0007f160 00007ffa`65d6b510 : 00000000`00002bc3 00007ffa`61e198a9 000000b1`0007f340 000000b1`0007f278 : rpcrt4!DispatchToStubInCNoAvrf+0x18
000000b1`0007f1b0 00007ffa`65d6aebb : 00000222`8d3adf80 00000000`00000000 00000000`00000000 00000222`93c99a20 : rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0x1a0
000000b1`0007f280 00007ffa`65d6ac99 : 00000222`93c99a20 00000222`93c998c0 00000000`00000000 00000222`8d14c480 : rpcrt4!RPC_INTERFACE::DispatchToStub+0xcb
000000b1`0007f2e0 00007ffa`65d6ab3b : 00000222`93c998c0 00000222`93c998c0 00000000`00000000 00000000`00000000 : rpcrt4!OSF_SCALL::DispatchHelper+0x125
000000b1`0007f400 00007ffa`65d692d1 : 00000222`93c998c0 00000222`8d4ac190 00000000`00000000 00000222`93c998c0 : rpcrt4!OSF_SCALL::DispatchRPCCall+0x7b
000000b1`0007f430 00007ffa`65d68f81 : 00000222`00000000 00000000`00000154 00000000`00000124 00000222`93c998c0 : rpcrt4!OSF_SCALL::ProcessReceivedPDU+0x2e1
000000b1`0007f500 00007ffa`65d689d2 : 00000000`00002bc3 000000b1`0007f5c9 00000222`93cb68d0 00007ffa`65e1eca4 : rpcrt4!OSF_SCALL::BeginRpcCall+0xbd
000000b1`0007f530 00007ffa`65d6bedd : 00000222`93cb6ad8 00000000`00000154 00000000`00000000 000000b1`0007f9f0 : rpcrt4!OSF_SCONNECTION::ProcessReceiveComplete+0x16a
000000b1`0007f630 00007ffa`62146cb0 : 000000b1`0007f9f0 00000222`93cb6b30 00000222`8d4ac190 00007ffa`00000154 : rpcrt4!CO_ConnectionThreadPoolCallback+0xbd
000000b1`0007f6b0 00007ffa`65f7eba9 : 00000222`941ecfa0 00000000`00000000 00000000`00000000 00000222`941ed068 : KERNELBASE!BasepTpIoCallback+0x50
000000b1`0007f700 00007ffa`65f666e8 : 00000222`941ed068 00000222`00000000 00000222`93cb6b30 00000000`00000000 : ntdll!TppIopExecuteCallback+0x129
000000b1`0007f780 00007ffa`64237974 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x3c8
000000b1`0007fa70 00007ffa`65faa371 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
000000b1`0007faa0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
SYMBOL_NAME: ntdsai!draXlateNativeReplyToOutboundReply+150
MODULE_NAME: ntdsai
IMAGE_NAME: ntdsai.dll
STACK_COMMAND: ~28s; .ecxr ; kb
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_ntdsai.dll!draXlateNativeReplyToOutboundReply
OS_VERSION: 10.0.17763.1
BUILDLAB_STR: rs5_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 10.0.17763.3650
FAILURE_ID_HASH: {94434ab1-e65f-b768-d896-92adf0d86393}
Followup: MachineOwner
---------
Domain Controller Systeminfo
C:\Users\Administrator>systeminfo
Host Name: DC01
OS Name: Microsoft Windows Server 2019 Standard Evaluation
OS Version: 10.0.17763 N/A Build 17763
OS Manufacturer: Microsoft Corporation
OS Configuration: Primary Domain Controller
OS Build Type: Multiprocessor Free
Registered Owner: Windows User
Registered Organization:
Product ID: 00431-10000-00000-AA873
Original Install Date: 3/2/2023, 4:57:17 PM
System Boot Time: 3/2/2023, 6:56:07 PM
System Manufacturer: VMware, Inc.
System Model: VMware20,1
System Type: x64-based PC
Processor(s): 2 Processor(s) Installed.
[01]: Intel64 Family 6 Model 140 Stepping 1 GenuineIntel ~2803 Mhz
[02]: Intel64 Family 6 Model 140 Stepping 1 GenuineIntel ~2803 Mhz
BIOS Version: VMware, Inc. VMW201.00V.20904234.B64.2212051119, 12/5/2022
Windows Directory: C:\Windows
System Directory: C:\Windows\system32
Boot Device: \Device\HarddiskVolume2
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC-05:00) Eastern Time (US & Canada)
Total Physical Memory: 4,095 MB
Available Physical Memory: 2,539 MB
Virtual Memory: Max Size: 5,503 MB
Virtual Memory: Available: 3,825 MB
Virtual Memory: In Use: 1,678 MB
Page File Location(s): C:\pagefile.sys
Domain: crash.lab
Logon Server: \\DC01
Hotfix(s): 3 Hotfix(s) Installed.
[01]: KB5020627
[02]: KB5019966
[03]: KB5020374
Network Card(s): 1 NIC(s) Installed.
[01]: Intel(R) 82574L Gigabit Network Connection
Connection Name: Ethernet0
DHCP Enabled: Yes
DHCP Server: 192.168.81.254
IP address(es)
[01]: 192.168.81.131
[02]: fe80::e0da:e880:9893:924f
Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.
Packet Capture pcap.zip
No, no Rapid7 EDR running on the DC's, but both DC's were running Symantec AV.
On 2023-03-02 22:55, twosevenzero wrote:
@DeserranoJorden [1] @BusyR [2] @7MinSec [3] In your cases, was the domain controller also running Rapid7's InsightIDR agent ir_agent.exe?
-- Reply to this email directly, view it on GitHub [4], or unsubscribe [5]. You are receiving this because you were mentioned.Message ID: @.***>
Links:
[1] https://github.com/DeserranoJorden [2] https://github.com/BusyR [3] https://github.com/7MinSec [4] https://github.com/fortra/impacket/issues/1436#issuecomment-1452600813 [5] https://github.com/notifications/unsubscribe-auth/AM5DTQHNINU6K6FMGQAWSHLW2EJN7ANCNFSM6AAAAAARZF3LUE
No, no Rapid7 EDR running on the DC's, but both DC's were running Symantec AV.
I have verified that it happens on a fresh install of Windows and DC Promotion.
Confirm here also on a Server 2019.
It happens only after a while so if the directory does not contain enough accounts it may not happen.
In my testing It could go as far as 19K accounts before rebooting.
I have also experienced the reboot with windows server 2012 R2 even though it seems to trigger later compared to 2019.
Can we get this issue tagged as a bug/request instead of a question, please?
After additional testing, I am not as convinced as I was in the beginning. Now, I am pretty sure this IS an issue with the way secretsdump performs the dcsync.
Using other tools like dsinternals and mimikatz to do full syncs do not result in a crash of the domain controller. Examining the logs on the domain controller also show that there is a login attempt for each and every user while using secretsdump. This is obviously going to put a bunch of stress on LSASS. Here is a copy of the Windows Logs/Security Event 4662 log.
An operation was performed on an object.
Subject :
Security ID: CRASH\Administrator
Account Name: Administrator
Account Domain: CRASH
Logon ID: 0xA6753C
Object:
Object Server: DS
Object Type: domainDNS
Object Name: DC=crash,DC=lab
Handle ID: 0x0
Operation:
Operation Type: Object Access
Accesses: Control Access
Access Mask: 0x100
Properties: Control Access
{1131f6ad-9c07-11d1-f79f-00c04fc2dcd2}
domainDNS
Additional Information:
Parameter 1: -
Parameter 2:
These happen about 80-100 times per second while doing a full dcsync as described in the logs I included in my previous post.
Who at Fortra is in charge of this project now? Can I get someone to take a look at the code to see if there is a way to reduce these connections?
For the pentesters that are finding this after getting crushed by a client for bringing down a domain controller, I would switch to using mimikatz or the dsinternals powershell module to get the same result.
To do this you will need a runas session for the account you want to use but it works as expected once you are set up.
Install-Module DSInternals -Force
Once this is installed you can use the Get-ADReplAccount
to perform your DCSync. Obviously, you will need to read into how this works and add flags based on your environment. An example to get started with would be something like:
Get-ADReplAccount -Server dc01.crash.lab -All | Format-Custom -View PwDump
I've tested on a brand new ad server 2022. Added 20k user and try to dump using secretdump.py the NTDS. No crash, no error. (defender was enabled).
The windows server was up to date.
@mpgn Thanks for the update. I will test on 2022 as well and report back.
@mpgn After multiple passes with secretsdump against a fresh 2022 domain controller, I am seeing the same things you are. No crashes. 👍 However, this is still an issue for Windows Server 2012r2, 2016, and 2019 domain controllers which are significantly more prevalent in live corporate environments where many penetration testers are using the tools. Have you been able to test these to see if you are able to replicate the issue?
Thanks a ton for your input here. As a core contributor and developer of security tools, I am sure your voice is regarded much higher than mine and you are likely able to see what the issues may be faster than I will since you are intimately familiar with it all. Looking forward to your test results. ❤️
I was able to reproduce on a windows server 2019, the error is not always the same but the DC crash yep. The domain controller was brand new and up to date.
@snovvcrash @0xdeaddood @gabrielg5 Can we get this marked as a bug instead of a question? I would create a new issue but there is so much valuable data in this ticket.
Just my 2 pence here. If you can crash lsass remotely via DRS, this is a CVE. INVALID_POINTER_READ suggests a dangling pointer, so could potentially be converted to RCE.
LSASS is trying to read from 0xffffffffffffffff, which is clearly an invalid address triggered by impackets implementation of dcsync
Microsoft closed my ticket about LSASS. I tested it today after the 04-2023 patches were applied and it still crashes so I guess they do not care. Feel free to weaponize it @CCob
It seems the issue was known for at least 5 years https://twitter.com/SkelSec/status/1042560510061932546 https://github.com/skelsec/windows_ad_dos_poc. Buggy function "draXlateNativeReplyToOutboundReply" was ported to windows 2019
There are some slight modifications in windows 2022 though:
Can we use secretsdump on assignments safely? or is it still a risk of BSOD on servers.
Not sure unfortunately. I've been using either CrackMapExec or just the Windows ntdsutil to do the AD dumps just to play it super safe.
Revisited this issue today with a few colleagues and after extensive testing confirmed 2016 and 2019 are still crashing. More testing needs to be done on 2012 but presumably it's also still affected.
Can we use secretsdump on assignments safely? or is it still a risk of BSOD on servers.
I am sure you just meant crash, but it's important to know that the DC does not BSOD. The LSASS process crashes and the OS catches that. It then displays an error stating that the system is unstable and will reboot in 30 or 60 seconds. Reboots in most modern environments take like 20 seconds so you will only see this much downtime making it very easy to think there was just some kind of network hiccup. I would bet that teams across the world are just crashing LSASS and rebooting DCs all the time.
Ok let me dump in an organized way, what I've been observing from my testing these days:
I managed to port mimikat'z implementation to impacket. I'm currently testing a PoC in order to eventually submit a PR. So far I encountered, in general, an improvement in performance and stability of the remote lsass process, BUT:
Ok let me dump in an organized way, what I've been observing from my testing these days:
- secretsdump it's prone to crash lsass on environments with thousands of user accounts (tested against an up to date 2019). It never crashed windows 2022 though.
- Microsoft has been reported regarding this issue, but patch has never been released.
- impacket's dcsync implementation potentially, causes stress of the remote lsass process, as 1 rpc call (actually 2 calls) is issued per user account. This implementation seems to be replicated in several open source projects on github, and even ported to other languages (aiosmb, metasploit ,etc).
- Mimikatz's implementation, on the other hand, issues 1 rpc call every 1000 objects, which at first, looks like a more straightforward way to handle thousands of accounts https://github.com/stjordanis/mimikatz/blob/6910c7b930b9f1115d2fe1623c906f51e8e85311/mimikatz/modules/lsadump/kuhl_m_lsadump_dc.c#L69
- I've been able to crash the remote lsass with mimikatz ( consider that mimikatz uses the microsoft stack in order to request/response and serialize rpc )
When I used mimikatz, in a domain with 50,000+ user, it caused lsass to crash and the DC to restart. mimikatz doesn't seem to solve the problem
Configuration
impacket version: Latest Python version: 3.10 Target OS: Kali latest
Debug Output With Command String
I was doing a secretsdump with DA creds as follows:
./secretsdump.py -just-dc-ntlm DOMAIN/USER@IP.OF.DOMAIN.CONTROLLER -outputfile /path/to/some/file/name
The dump started scrolling by as normal for about 30 seconds, then stopped on one particular line and just hung for about 30 seconds, followed by:
I used CME just to quickly revalidate the DC was up and accepting my creds, and I got dozens of lines of an exception/error. I talked to the client and they found the DC had rebooted. I did not run secretsdump again after that since I had a small heart attack.
Looking through the issues here and doing some Googling, I don't see other instances of this happening. Any thoughts on what might've caused it, or what info you might want to know about the target DC? All I know at this point is it is a Win Server 2016 Standard.
Thanks, Brian
PCAP
Sorry, don't have one.
Additional context
None at this time.