microsoft / terminal

The new Windows Terminal and the original Windows console host, all in the same place!
MIT License
95.93k stars 8.35k forks source link

Terminal crashes sometimes when running `clear` #13499

Closed tovine closed 1 year ago

tovine commented 2 years ago

Windows Terminal version

1.13.11431.0

Windows build number

10.0.19044.0

Other Software

bash (inside WSL, Debian)

Steps to reproduce

Do normal work, switch between being connected locally and remotely via RDP (I think the main point here is probably a change in the available displays, e.g. I have 5 monitors at the office and 2 at home - if that for some reason could matter). In some cases, there will be a lot of text missing (some of it perhaps visible when scrolling back up).

Expected Behavior

I'd expect that executing clear should clear up the window viewport and recover the shell to a fresh state so I could keep working.

Actual Behavior

Resizing the window sometimes forces a redraw and (at least partially) fixes the problem, but running clear in the bash shell to "recover" the terminal to a normal working state in most cases just causes the Terminal window to crash and disappear.

tovine commented 2 years ago

The tricky part about this is that it's not 100% reproducible, but it's happened often enough that I'm fairly certain it's the same issue. I have captured Event Log files from 3 separate days when this happened, not sure where I can upload them to you (might contain sensitive info, so not suited for attaching here?)

tovine commented 2 years ago

Event log details from the crash today:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2022-07-13T08:07:44.2630049Z" /> 
  <EventRecordID>157500</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>removed</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>WindowsTerminal.exe</Data> 
  <Data>1.13.2205.23001</Data> 
  <Data>628bd6e7</Data> 
  <Data>Microsoft.Terminal.Control.dll</Data> 
  <Data>1.13.2205.23001</Data> 
  <Data>628bd51b</Data> 
  <Data>c0000005</Data> 
  <Data>00000000000c1622</Data> 
  <Data>6120</Data> 
  <Data>01d895c3d9743c03</Data> 
  <Data>C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe</Data> 
  <Data>C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe\Microsoft.Terminal.Control.dll</Data> 
  <Data>e3bb1d28-9a5b-4cf9-acfc-7621cb1bb9bd</Data> 
  <Data>Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe</Data> 
  <Data>App</Data> 
  </EventData>
  </Event>

Previous report is exactly the same (apart from Process ID, time and report id)

zadjii-msft commented 2 years ago

There's a small chance that we might be able to look this crash up on the backend if you file feedback via the Feedback Hub, and share the link here: /feedback

If not, I'd recommend the following steps, to try and collect an automatic crash dump of the Terminal: Can you try following the steps in this post to set up automatic crash dumps? (For more info, see this link) If that works, then you should be able to automatically get a .dmp of the terminal when it crashes. Then, can you zip that dump up and send it to us, so we can investigate? Thanks!

ghost commented 2 years ago

Hi there!

Can you please send us feedback with the Feedback Hub with this issue? Make sure to click the "Start recording" button, then reproduce the issue before submitting the feedback. Once it's submitted, paste the link here so we can more easily find your crash information on the back end?

Thanks!

image

image

image

tovine commented 2 years ago

I tried recording using the Feedback Hub earlier, but for some reason it wouldn't start (also I guess it'd have to be recording until the issue happened, so that could be running for a while...)

Just enabled WER dumping like the guide in the blog post said: bilde Does it look correct, or should I add/change anything? Also, where do I send the dump once I get it?

tovine commented 2 years ago

It crashed again just now, but no dumps were found in the AppData\Local folder, not even the CrashDumps folder itself - do you have to create that manually first for it to work?

EDIT: could it be related to reaching the end of the scrollback buffer or something? It usually happens when I clear the terminal after having had quite a bit of text scroll by. BTW, my History size setting is at 32.6k lines - not sure if that's relevant...

briceo commented 2 years ago

Mine crashes when I had a large scrollback history. I have my bash terminal set to keep 9999999 lines, and when I get a large number of lines in the scrollback buffer, that's when clear crashes my windows terminal. And by crash, I should say, it actually freezes, and won't even redraw (becomes unresponsive), and the only thing I can do is to force close the terminal and restart the terminal application

DHowett commented 2 years ago

Interesting. We are supposed to ignore history sizes > 32000 . . .

zadjii-msft commented 2 years ago

HMM which distro are you using? I think the clear implementation changed between Ubuntu 18.04 and Ubuntu 22.04. Knowing which one in particular might help ID the root cause.

FWIW I can't repro this with historySize: 9999999 and clear with either Ubuntu, but maybe there's something else I'm missing. I'm testing with Terminal 1.16 and there's quite a few changes here since 1.13. Do you have anything like magnifier, narrator, NVDA, anything that listens for UIA events at all? There were a PILE of crashes related to UIA fixed in the last few months...

fok666 commented 2 years ago

I am currently experiencing this issue. Often with large scrollback history (over 10000 lines), performing "clear", "Ctrl+L" or "reset" freezes the whole Terminals app, with the process hanging with 100% use of a vCPU. Tested with Terminals 1.14.1962.0 and Ubuntu 20.04.

Avi0 commented 2 years ago

I have a reliable way to reproduce this crash. But the combination of events must be very specific.

  1. Install AlmaLinux 8 in WSL. (For some reason I can't reproduce this in Ubuntu, maybe because it has different shell configuration scripts that "arm" this crash during shell startup. I couldn't figure out what setting exactly leads to this crash).
  2. Open a WSL session and add "clear" to your ~/.bash_logout.
  3. Open many tabs using keyboard shortcut for WSL (for example, with Ctrl+Shift+3). You need 15-20 tabs.
  4. Start closing sessions by pressing Ctrl+D (shell logout).

For some reason, it's easier to reproduce this bug in Zsh. Install Zsh and set your default shell to Zsh (for example, with sudo lchsh username). Zsh in AlmaLinux already has "clear" in /etc/zlogout. This is how I encountered this problem in the first place.

briceo commented 2 years ago

@zadjii-msft

$ uname -a
Linux COMPUTER 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

My windows terminal version is:

Windows Terminal
Version: 1.14.1963.0

I do have an NVIDIA GPU in this system (RTX A1000 Laptop GPU):

Operating System:   Windows 10 Pro, 64-bit
DirectX version:    12.0 
GPU processor:      NVIDIA RTX A1000 Laptop GPU
Driver version:     516.59
Driver Type:        DCH
Direct3D feature level: 12_1
CUDA Cores:     2048 
Max-Q Technologies  Yes
Dynamic Boost       No
WhisperMode     No
Advanced Optimus    No
Maximum Graphics Power  40 W
Core clock:     1140 MHz 
Memory data rate:   11.00 Gbps
Memory interface:   128-bit 
Memory bandwidth:   176.03 GB/s
Total available graphics memory:    20322 MB
Dedicated video memory: 4096 MB GDDR6
System video memory:    0 MB
Shared system memory:   16226 MB
Video BIOS version: 94.07.5B.00.86
IRQ:            Not used
Bus:            PCI Express x8 Gen4
Part Number:        4736 0010

[Components]

nvui.dll        8.17.15.1659        NVIDIA User Experience Driver Component
nvxdplcy.dll        8.17.15.1659        NVIDIA User Experience Driver Component
nvxdbat.dll     8.17.15.1659        NVIDIA User Experience Driver Component
nvxdapix.dll        8.17.15.1659        NVIDIA User Experience Driver Component
NVCPL.DLL       8.17.15.1659        NVIDIA User Experience Driver Component
nvCplUIR.dll        8.1.940.0       NVIDIA Control Panel
nvCplUI.exe     8.1.940.0       NVIDIA Control Panel
nvWSSR.dll      31.0.15.1659        NVIDIA Workstation Server
nvWSS.dll       31.0.15.1659        NVIDIA Workstation Server
nvViTvSR.dll        31.0.15.1659        NVIDIA Video Server
nvViTvS.dll     31.0.15.1659        NVIDIA Video Server
nvLicensingS.dll    6.14.15.1659        NVIDIA Licensing Server
nvDevToolSR.dll     31.0.15.1659        NVIDIA Licensing Server
nvDevToolS.dll      31.0.15.1659        NVIDIA 3D Settings Server
nvDispSR.dll        31.0.15.1659        NVIDIA Display Server
nvDispS.dll     31.0.15.1659        NVIDIA Display Server
PhysX           09.20.0221      NVIDIA PhysX
NVCUDA64.DLL        31.0.15.1659        NVIDIA CUDA 11.7.99 driver
nvGameSR.dll        31.0.15.1659        NVIDIA 3D Settings Server
nvGameS.dll     31.0.15.1659        NVIDIA 3D Settings Server
Avi0 commented 2 years ago

On my other laptop, it's much easier to reproduce. AlmaLinux 8 + zsh. Opening a tab and then exiting a session with Ctrl+D is sufficient to trigger a crash. Simply running "clear" does not produce a crash, but "clear" that runs from /etc/zlogout reliably crashes the terminal.

Avi0 commented 2 years ago

I reproduced the crash and uploaded data to the feedback hub: https://aka.ms/AAhmp2s

ghost commented 2 years ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

Avi0 commented 2 years ago

Hey @tovine, can you add a comment saying that the crashdump is uploaded, otherwise this issue will be closed for inactivity and lack of author feedback.

tovine commented 2 years ago

Just to keep the bots happy, I guess I can comment. Glad to see that you guys were able to reproduce it so I'm not the only one, should hopefully give the devs enough to work with.

zadjii-msft commented 2 years ago

I still haven't managed a repro for this, and unfortunately, the feedback hub link only had OpenConsole dumps in it, not the Terminal crash. I did find an STATUS_INVALID_PARAMETER from ntdll thrown by the Terminal, but that error message unfortunately didn't give me a stack.

What's everything in your settings.json? Maybe there's something there I'm missing.

Avi0 commented 2 years ago

I can reproduce this problem even with default Terminal settings. I also have a more reliable way to reproduce this:

  1. Install AlmaLinux 8 into WSL.
  2. Install zsh and set it as your default shell.
  3. Put this into your ~/.zshrc
    fpath=($fpath /usr/share/zsh/vendor-completions/)
    autoload -U compinit && compinit
    exit 0
  4. Open a new tab with AlmaLinux 8. It will immediately close and crash Terminal.

This has something to do with tab being closed immediately after "clear". If you use "exit 1", tab won't close because of non-zero exit code and Terminal won't crash.

I'm willing to help with reproducing this problem. Let me know if I can provide more information or collect crash dumps in another way or anything else.

zadjii-msft commented 2 years ago

ooh we got a live one (finally). I needed to use a debug build. Looks like a race, which might explain why it wasn't hitting in release builds for me.

0:046> k
 # Child-SP          RetAddr               Call Site
00 00000088`af51de60 00007fff`3d3fa0db     Microsoft_Terminal_Control!Microsoft::Console::Render::Renderer::TriggerRedraw+0x8f [D:\dev\private\OpenConsole\src\renderer\base\renderer.cpp @ 224] 
01 00000088`af51df60 00007fff`3d3f8006     Microsoft_Terminal_Control!TextBuffer::TriggerRedraw+0x4b [D:\dev\private\OpenConsole\src\buffer\out\textBuffer.cpp @ 980] 
02 00000088`af51dfa0 00007fff`3d5e00f5     Microsoft_Terminal_Control!TextBuffer::WriteLine+0x1b6 [D:\dev\private\OpenConsole\src\buffer\out\textBuffer.cpp @ 392] 
03 00000088`af51e200 00007fff`3d5e0396     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::AdaptDispatch::_FillRect+0x2e5 [D:\dev\private\OpenConsole\src\terminal\adapter\adaptDispatch.cpp @ 560] 
04 00000088`af51e630 00007fff`3d5da6e6     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::AdaptDispatch::_EraseScrollback+0x206 [D:\dev\private\OpenConsole\src\terminal\adapter\adaptDispatch.cpp @ 1950] 
05 00000088`af51e7c0 00007fff`3d5c3f63     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::AdaptDispatch::EraseInDisplay+0xa6 [D:\dev\private\OpenConsole\src\terminal\adapter\adaptDispatch.cpp @ 614] 
06 00000088`af51e9e0 00007fff`3d5c5139     Microsoft_Terminal_Control!`Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionCsiDispatch'::`4'::<lambda_1>::operator()<Microsoft::Console::VirtualTerminal::VTParameter>+0x73 [D:\dev\private\OpenConsole\src\terminal\parser\OutputStateMachineEngine.cpp @ 494] 
07 00000088`af51ea30 00007fff`3d5be781     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::VTParameters::for_each<`Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionCsiDispatch'::`4'::<lambda_1> >+0x39 [D:\dev\private\OpenConsole\src\terminal\adapter\DispatchTypes.hpp @ 184] 
08 00000088`af51ea90 00007fff`3d5b694a     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionCsiDispatch+0xbf1 [D:\dev\private\OpenConsole\src\terminal\parser\OutputStateMachineEngine.cpp @ 492] 
09 00000088`af51f2b0 00007fff`3d5b8df6     Microsoft_Terminal_Control!`Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch'::`2'::<lambda_1>::operator()+0x16a [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 482] 
0a 00000088`af51f380 00007fff`3d5b9287     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch'::`2'::<lambda_1> >+0x26 [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 2039] 
0b 00000088`af51f3d0 00007fff`3d5b3c67     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecuteWithLog<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch'::`2'::<lambda_1> >+0x37 [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 2054] 
0c 00000088`af51f410 00007fff`3d5b4f36     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch+0x77 [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 480] 
0d 00000088`af51f470 00007fff`3d5b2d9d     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::StateMachine::_EventCsiParam+0x106 [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 1296] 
0e 00000088`af51f4b0 00007fff`3d5b3041     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::StateMachine::ProcessCharacter+0x1ed [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 1749] 
0f 00000088`af51f510 00007fff`3d56a3af     Microsoft_Terminal_Control!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString+0x171 [D:\dev\private\OpenConsole\src\terminal\parser\stateMachine.cpp @ 1853] 
10 00000088`af51f6b0 00007fff`3d052042     Microsoft_Terminal_Control!Microsoft::Terminal::Core::Terminal::Write+0xff [D:\dev\private\OpenConsole\src\cascadia\TerminalCore\Terminal.cpp @ 506] 
11 00000088`af51f790 00007fff`3d06500e     Microsoft_Terminal_Control!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler+0x82 [D:\dev\private\OpenConsole\src\cascadia\TerminalControl\ControlCore.cpp @ 1709] 
12 00000088`af51f810 00007fff`3d119ce9     Microsoft_Terminal_Control!`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::TerminalOutputHandler<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_218_>::operator()<winrt::hstring const &>+0x6e [D:\dev\private\OpenConsole\src\cascadia\TerminalControl\Generated Files\winrt\Microsoft.Terminal.TerminalConnection.h @ 521] 
13 00000088`af51f860 00007fff`55d3ca98     Microsoft_Terminal_Control!winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_218_> >::Invoke+0x39 [D:\dev\private\OpenConsole\src\cascadia\TerminalControl\Generated Files\winrt\Microsoft.Terminal.TerminalConnection.h @ 180] 
14 00000088`af51f8a0 00007fff`55cf943a     TerminalConnection!winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::operator()+0x68 [D:\dev\private\OpenConsole\src\cascadia\TerminalConnection\Generated Files\winrt\Microsoft.Terminal.TerminalConnection.h @ 465] 
15 00000088`af51f900 00007fff`55cca9f8     TerminalConnection!winrt::impl::invoke<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >+0x3a [D:\dev\private\OpenConsole\src\cascadia\TerminalConnection\Generated Files\winrt\base.h @ 5762] 
16 00000088`af51f970 00007fff`55d99ee5     TerminalConnection!winrt::event<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler>::operator()<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > >+0x108 [D:\dev\private\OpenConsole\src\cascadia\TerminalConnection\Generated Files\winrt\base.h @ 5897] 
17 00000088`af51fa10 00007fff`55d9a867     TerminalConnection!winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::_OutputThread+0x535 [D:\dev\private\OpenConsole\src\cascadia\TerminalConnection\ConptyConnection.cpp @ 672] 
18 00000088`af51fbf0 00007fff`55d9a8b5     TerminalConnection!`winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::Start'::`3'::<lambda_1>::operator()+0x37 [D:\dev\private\OpenConsole\src\cascadia\TerminalConnection\ConptyConnection.cpp @ 374] 
19 00000088`af51fc30 00007fff`c074458d     TerminalConnection!`winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::Start'::`3'::<lambda_1>::<lambda_invoker_cdecl>+0x25 [D:\dev\private\OpenConsole\src\cascadia\TerminalConnection\ConptyConnection.cpp @ 377] 
1a 00000088`af51fc70 00007fff`c2197548     KERNEL32!BaseThreadInitThunk+0x1d [clientcore\base\win32\client\thread.c @ 75] 
1b 00000088`af51fca0 00000000`00000000     ntdll!RtlUserThreadStart+0x28 [minkernel\ntdll\rtlstrt.c @ 1192] 

We're in the ControlCore::_connectionOutputHandler. Seemingly, the ControlCore has been destructed here(!), cause looking at that frame, m_inner is 0, _closing=true, and every renderer member is nulled out.

We took the _terminal write lock. We started parsing, and eventually got to a TextBuffer::TriggerRedraw where we try to invalidate the viewport and zoop zoop, there's no render data.

I suppose it's totally possible for the _connectionOutputHandler to start processing the string, then have the CPU context switch to the main thread and do ControlCore::Close to revoke the handler, (but it's too late, the output thread is already handling the string), and then get to the end of the ControlCore::dtor and have it clean up the renderer, before switching back to the output thread that finishes the output processing.

That being said, that should only be possible to trigger if you do something as the tab is closing. Just running clear by itself, it a otherwise working terminal, won't hit that crash. Hmm.

We've had some other deadlocks, esp when RDP is involved, that we've been tracking over the last couple weeks. (#13705 et al). If RDP is a consistent part of this repro, then maybe what we're seeing in this thread is one of those deadlocks, that's triggering a crash with clear instead of getting stuck locked.

AndrewBabbitt97 commented 2 years ago

I am able to repro this with 1.14.1963.0 on Windows 11 Build 22621.382. If any additional info is needed that may be of use please let me know, it has been a source of constant headaches while working the past week or so,

As an additional note, most (if not all) of my crashes have been caused when using clear while ssh'd into a Ubuntu 20.04machine.

zadjii-msft commented 2 years ago

@AndrewBabbitt97 do you have a consistent, minimal repro that doesn't involve closing the tab/pane? I'm thinking the stack I saw here: https://github.com/microsoft/terminal/issues/13499#issuecomment-1220778182 was ultimately a different issue than the issue at the root of this thread

AndrewBabbitt97 commented 2 years ago

@zadjii-msft No, but I have some notes that are consistant:

  1. Crashes are usually on longer terminal sessions (usually one or more hours, I rarely get one under that)
  2. Usually occurs on larger volumes of history in the session

I'll try to get a crash dump, but we may also be able to repro it using some loops in a bash script involving clear and printing volumes of text to the terminal as well. I'll play around and try some things.

AndrewBabbitt97 commented 2 years ago

@zadjii-msft So context:

I have been running some decently large ansible playbooks to deploy openstack using kolla-ansible, this produces very large amounts of output to the console. This issue definitely SEEMS to be related to console history size. The settings UI does not restrict the max size of the history to 32000, and I am unable to repro the issue when my history size is set to this.

However you can set the number really easily to something like 32767 (I dont know why mine was set to this specifically, I think I saw this as the max at some point in the past) I can get a repro using this script inside of WSL using Ubuntu 22.04:

#!/bin/bash

for i in {0..100000}
do
  echo "============================== $i =============================="
done

clear
AndrewBabbitt97 commented 2 years ago

If 32000 is indeed the maximum size, is this number not being floored before use? Another consideration may be to make sure the settings UI is also flooring this number.

AndrewBabbitt97 commented 2 years ago

Also used the feedback app as well: https://aka.ms/AAhrk3o

tovine commented 2 years ago

Great to see that this issue has been reproduced in different ways by several independent reporters, and especially that it's been assigned to a milestone!

I've started to see a new failure mode recently, which may or may not be related: when exiting less (typically after browsing a git diff result) the terminal window hangs with a white title bar and a "(Not responding)" at the end of the window name. It never recovers from this and needs to be closed.

Here's the info in Event Viewer (let me know if you want the detailed files and attachments and I can provide it):

Fault bucket , type 0
Event Name: AppHangTransient
Response: Not available
Cab Id: 0

Problem signature:
P1: WindowsTerminal.exe
P2: 1.14.2207.15002
P3: 62d1f0dc
P4: unknown
P5: unknown
P6: unknown
P7: unknown
P8: 
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2025.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2045.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2044.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2083.tmp.txt

These files may be available here:

Analysis symbol: 
Rechecking for solution: 0
Report Id: 9be1b928-d8d1-4f86-a6f9-54e9439f2b7b
Report Status: 2049
Hashed bucket: 
Cab Guid: 0
tovine commented 2 years ago

Happened again with new version: 1.14.2281.0 (updated after the previous comment I made)

Event log:

Faulting application name: WindowsTerminal.exe, version: 1.14.2208.16001, time stamp: 0x62fc1924
Faulting module name: Microsoft.Terminal.Control.dll, version: 1.14.2208.16001, time stamp: 0x62fc1770
Exception code: 0xc0000005
Fault offset: 0x00000000000c56fa
Faulting process id: 0x6cb0
Faulting application start time: 0x01d8bc48b9b7bbb9
Faulting application path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.14.2281.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
Faulting module path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.14.2281.0_x64__8wekyb3d8bbwe\Microsoft.Terminal.Control.dll
Report Id: 328fc548-b9ee-469d-a349-f48aaa717892
Faulting package full name: Microsoft.WindowsTerminal_1.14.2281.0_x64__8wekyb3d8bbwe
Faulting package-relative application ID: App
tovine commented 2 years ago

Any updates on this? Terminal has crashed 3 times today (in about 3 hours) so it's getting pretty annoying... Happy to test beta versions with a suggested fix or anything similar

EDIT: it seems that the terminal crashes extra often when I'm switching between logging in locally vs remote desktop (something related to switching the desktop resolution and shape - different multi-monitor setups). And on the other hand it seems to (almost?) never crash if I resize the terminal window first, triggering some resize and redraw event I guess?

zadjii-msft commented 2 years ago

@tovine If you try out the Atlas renderer on 1.16, does this issue persist?

image

aursulis commented 2 years ago

I found a WindowsTerminal.exe dump file in the folder I've set up for local dumps. Backtrace looks related to this issue, here's the !analyze -v:

WindowsTerminal.exe.18484.dmp ``` 0:006> !analyze -v ******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* KEY_VALUES_STRING: 1 Key : AV.Fault Value: Read Key : Analysis.CPU.mSec Value: 843 Key : Analysis.DebugAnalysisManager Value: Create Key : Analysis.Elapsed.mSec Value: 2396 Key : Analysis.IO.Other.Mb Value: 0 Key : Analysis.IO.Read.Mb Value: 0 Key : Analysis.IO.Write.Mb Value: 0 Key : Analysis.Init.CPU.mSec Value: 187 Key : Analysis.Init.Elapsed.mSec Value: 2368 Key : Analysis.Memory.CommitPeak.Mb Value: 156 Key : Timeline.OS.Boot.DeltaSec Value: 1109997 Key : Timeline.Process.Start.DeltaSec Value: 497830 Key : WER.OS.Branch Value: vb_release Key : WER.OS.Timestamp Value: 2019-12-06T14:06:00Z Key : WER.OS.Version Value: 10.0.19041.1 Key : WER.Process.Version Value: 1.15.2210.14004 FILE_IN_CAB: WindowsTerminal.exe.18484.dmp NTGLOBALFLAG: 0 PROCESS_BAM_CURRENT_THROTTLED: 0 PROCESS_BAM_PREVIOUS_THROTTLED: 0 APPLICATION_VERIFIER_FLAGS: 0 CONTEXT: (.ecxr) rax=00000058237ff800 rbx=0000005f00000000 rcx=06410397466e0000 rdx=0000020d0baba400 rsi=00000058237ff970 rdi=0000020d0b9dc120 rip=00007ffc879c4bc1 rsp=00000058237ff8d0 rbp=0000020d7f3f4be0 r8=0000000000000000 r9=0000000000000000 r10=00000058237ff800 r11=0000000000000000 r12=0000000000000000 r13=0000020d7ea27770 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl nz na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 Microsoft_Terminal_Control!Microsoft::Terminal::Core::Terminal::Write+0x69: 00007ffc`879c4bc1 488b4730 mov rax,qword ptr [rdi+30h] ds:0000020d`0b9dc150=???????????????? Resetting default scope EXCEPTION_RECORD: (.exr -1) ExceptionAddress: 00007ffc879c4bc1 (Microsoft_Terminal_Control!Microsoft::Terminal::Core::Terminal::Write+0x0000000000000069) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 0000000000000000 Parameter[1]: 0000020d0b9dc150 Attempt to read from address 0000020d0b9dc150 PROCESS_NAME: WindowsTerminal.exe READ_ADDRESS: 0000020d0b9dc150 ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_CODE_STR: c0000005 EXCEPTION_PARAMETER1: 0000000000000000 EXCEPTION_PARAMETER2: 0000020d0b9dc150 STACK_TEXT: 00000058`237ff8d0 00007ffc`879c4b43 : 0000020d`0b032960 00000000`00000000 0000020d`0b032960 00000000`00000000 : Microsoft_Terminal_Control!Microsoft::Terminal::Core::Terminal::Write+0x69 00000058`237ff950 00007ffc`879e5f52 : 0000020d`7ea27760 0000020d`7ea27750 00000000`00000000 00007ffc`d6b25c72 : Microsoft_Terminal_Control!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler+0x33 00000058`237ff9a0 00007ffc`d6b25b05 : 0000712d`00000001 0000020d`7ea27748 00000000`00000000 00000000`00000000 : Microsoft_Terminal_Control!winrt::impl::delegate'::`1':: >::Invoke+0x32 00000058`237ff9f0 00007ffc`d6b259df : 0000020d`00000000 00004000`00000000 00000000`00000000 00000058`237ffac0 : TerminalConnection!winrt::impl::invoke,std::allocator > >+0x55 00000058`237ffa50 00007ffc`d6b2c4ae : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : TerminalConnection!winrt::Microsoft::Terminal::TerminalConnection::implementation::ConptyConnection::_OutputThread+0x1ff 00000058`237ffb20 00007ffc`f51a7034 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : TerminalConnection!_Closure_wrapper_64d1e025_14::+0xe 00000058`237ffb50 00007ffc`f52e26a1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 00000058`237ffb80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 STACK_COMMAND: ~6s; .ecxr ; kb FAULTING_SOURCE_LINE: C:\a\_work\1\s\src\cascadia\TerminalCore\Terminal.cpp FAULTING_SOURCE_FILE: C:\a\_work\1\s\src\cascadia\TerminalCore\Terminal.cpp FAULTING_SOURCE_LINE_NUMBER: 496 FAULTING_SOURCE_CODE: No source found for 'C:\a\_work\1\s\src\cascadia\TerminalCore\Terminal.cpp' SYMBOL_NAME: Microsoft_Terminal_Control!Microsoft::Terminal::Core::Terminal::Write+69 MODULE_NAME: Microsoft_Terminal_Control IMAGE_NAME: Microsoft.Terminal.Control.dll FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_Microsoft.Terminal.Control.dll!Microsoft::Terminal::Core::Terminal::Write OS_VERSION: 10.0.19041.1 BUILDLAB_STR: vb_release OSPLATFORM_TYPE: x64 OSNAME: Windows 10 IMAGE_VERSION: 1.15.2210.14004 FAILURE_ID_HASH: {39be1696-4bbf-0321-1166-d994398e7f26} Followup: MachineOwner --------- ```

I use WSL and switch between RDP and local connection a lot, so is it the same issue or should I create a new one?

tovine commented 2 years ago

@aursulis it seems very similar to my original issue (switching between RDP and local), and the refernced DLL matches my error reports.

@zadjii-msft since updating to 1.15.2874.0 (installed from the Store) I haven't had a single crash in quite some time. Not sure enough to say the issue is permanently gone, but it's at least worked well so far. Haven't been able to test out AtlasEngine since I'm not on that version.

rar222 commented 1 year ago

I can reproduce this problem even with default Terminal settings. I also have a more reliable way to reproduce this:

I think that https://github.com/microsoft/terminal/issues/13499#issuecomment-1220593605 is likely the same crash issue that I see every so often.

Essentially in a linux terminal "Ctrl-D" to exit the terminal sometimes crashes all tabs.

Faulting application name: WindowsTerminal.exe, version: 1.16.2301.26001, time stamp: 0x63d3331b
Faulting module name: ntdll.dll, version: 10.0.19041.2130, time stamp: 0xb5ced1c6
Exception code: 0xc000000d
Fault offset: 0x0000000000112684
Faulting process id: 0x2974
Faulting application start time: 0x01d9675b51a5aa30
Faulting application path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.16.10261.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 49e1b2b2-5312-4fc2-99a8-e38586f3fb91
Faulting package full name: Microsoft.WindowsTerminal_1.16.10261.0_x64__8wekyb3d8bbwe
Faulting package-relative application ID: App

I have tabs set to close on exit.

lhecker commented 1 year ago

This is internally filed as MSFT-41429540 and the failure bucket is 39be1696-4bbf-0321-1166-d994398e7f26.

zadjii-msft commented 1 year ago

You know, crawling through the dumps, there's no dumps of this on 1.17. none.

And a sharp decrease in these hits around April 12th. That's weird cause I don't think we serviced anything then. Maybe there was some compounding bug that was?

Regardless, the few dumps we did have from MSFT:41429540 did look the same as this report, so I'm gonna go out on a limb and say that this was fixed in 1.17. There were some lifetime changes in that build that come to mind that might have fixed this.

I'll tag it up as "Fixed?" - if anyone sees this on 1.17+, we'll probably want a fresh issue to track the new root cause.

Thanks!

zadjii-msft commented 1 year ago

Addenda: MSFT:43308096 looks like the inverse frequency - the crashes started going up around 4/12. Interesting. That one's a ~ControlCore a/v, which is kinda similar. No repros of that one in 1.17 either, so that's good.

microsoft-github-policy-service[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

microsoft-github-policy-service[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

microsoft-github-policy-service[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

tovine commented 1 year ago

Ok, guess I'll have to chime in to prevent the bot from auto closing? Nice to see that you're tracking it internally and making progress! :)

FWIW I haven't had any crashes of this kind recently, so you've at least fixed something 🤗 PS: I'm currently on version 1.16.10261.0, installed from the Store

zadjii-msft commented 1 year ago

Oh I mean, this seems fixed, you say it's fixed, no one else has chimed in - I think it's just fixed 😄