microsoft / vscode

Visual Studio Code
https://code.visualstudio.com
MIT License
164.93k stars 29.52k forks source link

Many Different Extensions Frequently Crash All Others on Startup with Reoccurring "Extension host terminated unexpectedly" – Extension Sandboxing / Isolation, Missing Error Log Fixes and Host Auto-Restart Needed #79782

Open PowerWeb5 opened 5 years ago

PowerWeb5 commented 5 years ago

There have been far too may reports (800+ now) of the VSCode Extension Host crashing with "Extension host terminated unexpectedly" like shown in the below screenshot:

image

When this happens, one extension ends up crashing all other extensions and even crashing core VSCode services as well as apparently preventing logging of the errors, and then requires users to manually restart all extensions.

These issues affect a wide range of popular extensions, including core Microsoft ones like Live Share Audio, as well as frequent third party ones like Live Chat (karigari.chat) and Live Server. And there are 800+ reports of the critical "Extension host terminated unexpectedly" errors, and this issue becoming far more frequent of late.

It's not even possible to determine the problematic extension (so that you can disable it and submit a bug report) – even if you were to startup 170 times with a different extension disabled each time, as, for some reason, the original extension error and even the extension host crash itself, as well as what extension was starting to be activated or updated, often fail to ever get logged when such crashes occur.

This has become more frequent and has been reoccurring for extensions even after fixing it, breaking again even when no extension updates were released. It has been causing VS Code to crash on startup much of the time for many months now without any way to identify and disable the extension causing it. Then, after finally disabling one that fixed for a while, another extension started causing the same thing all over again. And, on each VSCode startup, repeated manual user intervention is often required to resolve what could be auto-repaired.

Therefore, it seems that VSCode changes are needed to minimize these issues (vs many repeated workarounds for each extension) – both short-term and long-term – as outlined further below.

The Case for Extension Sandboxing

I believe these widespread reoccurring critical issues also indicate the need for Extension Sandboxing (like proposed in #52116). Among other benefits, that would prevent extensions from crashing any (or at least not all) other extensions as well as core VSCode services, make it possible to determine which extensions are crashing, and also open the door to security benefits with extension permission management. I've

Repeated Critical Crashes on Startup I've Encountered For Months with Multiple Different Extensions

I have had to manually restart extension host nearly every time I started VSCode for months now. Then, finally, after identifying and disabling the problematic extension (only possible by reading through may bug reports to determine most likely causes), the issue started happening again, with another extension (equally difficult to determine which one).

And that was for an extension where the issue had been fixed several times already, with no update released since the last version fixing it, yet the issue started occurring again (possibly due to VS Code or other extensions changing). Considering that as well as how it seems to almost be randomly occurring, always on startup (eg. as if timing related or while other extensions are being updated), then it may be that something in VSCode needs to change to minimize these issues.

Related Bug Report with Log Files

You can find details regarding my own most experiences with VS Code host crashes for the most recent extension, Live Server, at ritwickdey/vscode-live-server#552.

There, I've provided zips of log files when crashes do and do not occur (as needed for comparison due to lack of detailed logging when crashes occur).

I've also referenced to a number of related issues in that report, as well as further below.

That does not include my earlier months of VS Code crashing on startup with other extensions, such as what I'd eventually identified (despite lack of any indication in log files) as being caused by Live Share Chat, which I'd fixed just a week or so before encountering with Live Server as well.

Suggested VSCode Fixes / Changes (to Minimize Host Crashes, Allow Identifying Problem Extensions, and Prevent Crashing All Extensions)

Identifying and Addressing the Root Cause vs. Repeated Per-Extension Temporary Workarounds

It would be helpful if someone could investigate what, if any, common denominator there may be between the different extension host crash causing extension issue (such as use of binary dependencies or how extensions are updated or activated) or such as the various extensions starting with "Live" in their name (if not a coincide with them being the most frequent offenders).

The root cause should be addressed – with changes made to VSCode (or the extension host) to minimize them instead of requiring extension developers to have to keep (clearly unreliably, temporarily) implementing workarounds for each extension. Whether changes to how extensions are

800+ Outstanding Issue Reports and Many (Even Official) Extensions with Frequent, Reoccurring Extension Host Crashes

Many others are reporting the same, such as my recent report (ritwickdey/vscode-live-server#285) as well as #72476, #75267, MicrosoftDocs/live-share#1510, ritwickdey/vscode-live-server#285, Live Share Chat review, and 250+ other "Extension host terminated" bug reports just in the VSCode repo, 133 open issues under chrmarti/testissues and 850+ "Extension host terminated unexpectedly" bug reports across all GitHub VSCode extension repos

Other recent reports in VSCode repo:

77099, #79177, #78524, #78196, #77917, #77780, #78809, #78331, #78196, #77768, #77547, #77505

Other issues related to extension crashing affecting other extensions:

73041, #24441, #76178

Many Different Extensions Frequently Crash ALL Other Extensions with Issue Reoccurring or Unable to Be Fixed

Just a few of the highly popular, and even Microsoft official VSCode extensions which this occurs for includes:

Live Share Audio, Live Server, Live Share Chat (karigari.chat), Live Share, Live SASS Compiler, Typed Test, Java Language Server, Peacock, and Mocha Sidebar

Not only are these issues widespread across a large number of popular extensions, but they are reoccurring. The issue keeps coming back for the same extensions not long after each fix is released.

Extension Host Crash Reappears After Fixing even when Extension Weren't Updated Since

With Live Server, as well as other extensions, such extension host crashing on VSCode startup has come back every time it gets (temporarily) fixed.

In that issue I'd just reported for Live Server, for example, the crashing started occurring again for the very same extension version (5.6.1) which was released specifically to fix the issue and which had been confirmed (for a little while) as having done so, as seen in vscode-live-server#431 the issue is back not months after the last update which was released specifically to address that issue.

Accordingly, it seems like changes to VSCode itself (or possibly frequently arising conflicts between extensions may be the cause) may be part of the issue preventing reliable fixes or avoidance of such crashes.

And before that was report "Extension host terminated unexpectedly is back" (ritwickdey/vscode-live-server#399) about how the issue had come back after having been fixed a number of times before even that.

Crashes ALL Extensions and Core VSCode Services

Worse yet, such crashes don't just affect the problematic extension, they crash ALL extensions. Even worse, they crash a core VSCode service, the Extension Host, and require the user to manually restart the host to reload all extensions.

Especially considering that, it seems like some kind of core VSCode or Extension Host changes may be needed to reduce, avoid or at least isolate the effect of the errors which are currently causing frequent reoccurring Extension Host crashes.

Some of the Popular Extensions Most Frequently Causing Extension Host Crashes

This issue is well known to often occur with Live Share Chat (karigari.chat) like reported in #77099 and mentioned in dozens of other VSCode issues. And it remains an open known issue, again, which is still unable to be addressed, like seen at vsls-contrib#410.

Even Microsoft's own Live Share Audio extension (as well as Live Share itself) frequently causes Extension Host crashes. And there is no plan to fix it any time soon, according to its developers.

Possible Causes of Frequent Extension Host Crashes (eg. Timing or Dependencies)?

Can anyone clarify what the common causes of such crashes are? I've seen reference to use of binary dependencies as a common case. Is that so? Why would such crashes start occurring again, with the same version where it was fixed previously, despite no new updates to such fixed extensions?

Whatever the cause, this seems widespread enough and critical enough that I would suggest something be done to minimize the frequency of such Extension host crashes, if possible.

Does anyone have any ideas on what might be able to be done to help reduce the frequency of Extension Host crashes?

If the most recent fixes for Live Share Chat (which was causing months of issues prior to Live Server) are any indication (where "Add Live Share as an extension dependency to fix timing issues during activation" is mentioned as the primary fix in v0.21.0 released on 2019-08-19), then interdependency between extensions and timing related issues might be one type of common cause in Extension Host crashes, especially considering how the crashes don't occur on every startup (just seemingly randomly ones).

I'd also seen extension a number of "Deleted from disk" entries in sharedprocess.log (one of the few non-empty log files) as well as "Starting to clean up unused language packs.", which I mention in case the timing of such extension updates/cleanups might be a factor.

Extension Sandboxing / Isolation Needed for Reliability (as well as Security) (+ Link to Detailed Proposal/Ways to Implement with Electron)

In any case, this is a good example of why Extension Sandboxing / Isolation is needed.

I had previously proposed in #52116 several options for implementing Extension Sandboxing / Isolation for reliability and security (as well as Chrome/Firefox-like Permission Management). There are several projects, like I'd summarized there, which could be utilized or which have successfully implemented such extension sandboxing, even with Electron-based apps like VSCode and Electron options/APIs now available, which could be used as a model or incorporated directly.

Need to Auto Restart Host If Occurs Immediately Upon Startup

Also, at the very least, I believe that there should be an option to automatically recover.

Ideally this can occur after logging the crash itself, trying to identify which extension may have caused the crash and logging and reporting that to the user, and then flushing the logs (without waiting for any user interaction).

Then the Extension Host should be automatically restarted when Extension Host crashes on startup.

This is important and reasonable to do because, in each of the many dozens of crashes I've encountered, this crash occurs within seconds of VSCode startup, and restarting always fixes the issue successfully (which my never encountering another host crash afterward during that session).

Therefore, IMO, the extension host should always be automatically restarted if it crashes within a configurable number of seconds (which could default to 5 or 10) such apparently expected. This seems reasonable handling for what is apparently far from exceptional and repeated issues, especially when automated recovery/restarting without any impact on the user can occur and even more so if this might be related to auto extension downloading/updating/install which may be occurring prior to the crash.

The user should be notified still, but there should be an option to hide such future notifications (at least showing such an option if another crash notification occurs within X days or Y number of startups since the last one) to avoid nagging the user. Especially when, at one point, I had to see annoying crash notifications and manually restart as prompted nearly every time I launched VS Code, without any indication of what extension caused or way for me to stop this occurring for months on end.

Need to Report the Extension Causing the Crash & Prompt to Disable

Also, VSCode should try to determine which extension caused (or set of extensions may have likely caused) the crash. If a specific extension can be identified as the cause, that should be shown to the user together with the host crash notification, and with a suggestion (at least for repeat offender extensions) and in-notification button (together with the Show Developer Tools and Restart Extension Host buttons) or 1-click disabling of that extension. Most often so far, the extensions causing such crashes are ones I can live without.

Currently Impossible to Identify Extension Causing Crash (Due to Logging Issues and Lack of Isolation)

However, the most important problem of all, is that it, as it stands now, it is nearly impossible to identify which extension is causing the problem.

It's not feasible for users to test startups by disabling extensions one-by-one to determine which extension or combination of extensions is causing the issue. In fact, that becomes virtually impossible when you consider that such crashes on startups don't happen every time (just every day or two in many cases). And even more so, when it might be a conflict between multiple extensions (as I've seen reference to in other issue reports) or there are multiple extensions causing such crashes at the same time (which I have also encountered).

Extension Error and Even Host Crash Itself Fail to Get Logged or Flushed

Even after finding and reading through (as well as searching) every single log file under Roaming/Code/Logs for the session (including the annoying number of empty folders to search through which are created on each startup, which I would suggest be created as-needed instead) there was absolutely no logged error. Even the activation of the extension which ended up seeming to be the cause of the crashes didn't end up in the logs or flushed to disk. And restarting the extension host or showing Developer Tools ended up obscuring the log entries and can occur 20 minutes after the actual crash (as was the case for me).

However, the only time I saw an actual error logged related to the host crash was after Restarting the Extension Host and showing Developer Tools, in which case it either failed to show anything from before that or it obscured it with many more entries. Even then, it was just a generic extension host crash error, without any indication of the original cause (neither extension or the error that caused it).

When trying to look for the relevant errors to identify which extension may have caused them and therefore which to try to disable to avoid these crashes, it was always the case that almost all log files were empty (extensions.log, tasks.log, essentially main.log for most part, and nearly all files in the few created "output_" subfolders). No errors appeared (except for a few clearly unrelated warnings which occurred even in non-crash cases). Only by comparing to non-crash startup logs could I get an idea of what might have caused the issue, and only by looking at what log entries were missing such as for extension activations that occurred immediately after where the log file was cutoff.

Logging Fixes Needed So Crash or Extension Error Is Actually Logged

As the host crash error itself doesn't end up in the log files in many cases under Roaming/Code/Logs or visibly elsewhere (other than the notification box shown to the user) the following might be helpful:

My Platform and Configuration

vscodebot[bot] commented 5 years ago

(Experimental duplicate detection) Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

Gruntfuggly commented 5 years ago

I know my extension causes the host to crash, but I have no way of find our where or how. :disappointed:

Please address this somehow vscode team!

PowerWeb5 commented 5 years ago

@RMacfarlane and @kieferrm, please see the crash report which I just submitted, #80165 for additional details and logs likely useful for debugging this.

That provides detailed logs that may help in debugging this issue, with logs both for sessions with and without this issue occurring, despite same steps to reproduce (needed since doesn't seem to log any extension-specific errors related just to the crash case), as well as logs for session preceding the crash (in case related to extension updating/cleanup) and log files saved from Developer Tools (which show some errors, which otherwise don't get saved to log folders on disk).

Narvey commented 4 years ago

Why is this not a higher priority?

I recommend the sandboxing solution (#52116) Without it, VS Code looks like a liability to the IT departments that might consider approving it! (think about trying to determine the risks of a wide variety of addons, and whether they will be stealing company secrets like proprietary code)

vintprox commented 2 years ago

On kinda related note, Chat Session from VS Live Share is randomly closing after 2 minutes of inactivity. I don't know how to catch it, but it just happened a lot today, when I tried it out with my student.