search

dotnet / interactive

.NET Interactive combines the power of .NET with many other languages to create notebooks, REPLs, and embedded coding experiences. Share code, explore data, write, and learn across your apps in ways you couldn't before.
MIT License
2.92k stars 389 forks source link

Dotnet-interactive processes continue running even after shutdown/restart kernel commands issued form the VSCode UI and quitting the VSCode app #1005

Closed nasosev closed 1 year ago

nasosev commented 3 years ago

Describe the bug

Dotnet-interactive processes continue running even after shutdown/restart kernel commands issued form the VSCode UI and quitting the VSCode app.

After investigating slow down issues on my computer, I looked in the Activity Manager to find dozens of dotnet-interactive processes running, some consuming up to 100GB of memory and growing, even though the VSCode application is no longer even running. These processes were spawned by VSCode Jupyter notebook extension and seem to fail to be aborted by pressing the shutdown/restart kernel buttons or by quitting VSCode. I reproduced this issue after restarting the computer.

I have since stopped using VSCode and am using Juypter Lab instead which does not appear to be affected.

Please complete the following:

Which version of .NET Interactive are you using? There are a few ways to find this out:

1.0.206501+2cf4523097c44f5b4e0c6babf8022359a5d94e8a

Screenshots

If applicable, add screenshots to help explain your problem.

jonsequitur commented 3 years ago

I'm not able to reproduce this on 1.0.207203+f90ff415badc98a82c7db599658df94ad527328a and macOS 11.1.

If you can still reproduce it, can you tell us a bit more about what you're doing? How are you opening the files, what languages are you using, can you reproduce it without running any code cells, etc.?

nasosev commented 3 years ago

@jonsequitur I can reproduce it as follows.

  1. make a new notebook bug.ipynb in Jupyter lab with the following cell (F#) that does an expensive computation
/// Permutations of a list.
let rec permutations (xs: List<'A>): List<List<'A>> =
    if List.isEmpty xs then
        [ [] ]
    else
        [ for x in xs do
            yield!
                xs
                |> List.filter ((<>) x)
                |> permutations
                |> List.map (fun xs -> x :: xs) ]

let p = [1..20] |> permutations
  1. open the directory from terminal by entering code . then open bug.ipynb from the sidebar; run the cell by press shift+enter on it.
  2. observe by entering top -o men at terminal the dotnet-interactive process consuming large amounts of memory.
  3. try to stop the computation by pressing either the Stop or Restart kernel buttons in the VSCode Jupyter UI; observe that the dotnet-interactive processes continues to run and consume large amounts of memory
  4. exit VSCode; observe dotnet-interactive process continues running and eating memory...

Thanks for looking into it.

jonsequitur commented 3 years ago

Thanks for the repro!

Cancelling cell execution is a known issue that might require a runtime change: https://github.com/dotnet/runtime/issues/41291.

But issuing the commands to stop the kernel entirely, or shutting down VS Code, should definitely be killing these processes, and this looks like it's not working on macOS.

I can't reproduce this bug on Windows.

brettfo commented 3 years ago

Good find. We try to kill the process here by calling this.childProcess.kill();. According to this, the signal sent by default is SIGTERM which according to this "can be blocked, handled, and ignored". Looking further down it appears we should check the return value of the blind .kill(); call and if it was unsuccessful, we should then send SIGKILL which "cannot be handled or ignored, and is therefore always fatal. It is also not possible to block this signal."

tmitchel2 commented 3 years ago

Can confirm this is still an issue on mac

joanimato commented 3 years ago

I'm also experiencing this issue on a Mac.

retifrav commented 3 years ago

I've been experiencing this issue for some months now. But I was using VS Code Insiders, so I reckoned that it might be still unstable/experimental feature, and so i wasn't sure if I should create a bugreport. But today I've tried it out in regular VS Code, and the issue is present there too, so thanks to nasosev for creating this bugreport.

As it was already described, even after VS Code is closed, dotnet processes continue to run and load 100% of a core, wasting CPU and battery charge. Here's the output from htop:

CPU% MEM% Command
99.7  0.2 /usr/local/share/dotnet/dotnet /Users/vasya/.nuget/packages/microsoft.dotnet-interactive/1.0.258004/tools/net6.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser
99.6  0.2 /usr/local/share/dotnet/dotnet /Users/vasya/.nuget/packages/microsoft.dotnet-interactive/1.0.258004/tools/net6.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser
 0.8  1.7 /Users/vasya/.dotnet/tools/dotnet-interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2329WtGsCHtBFDBr.json
 0.0  0.6 /Users/vasya/.dotnet/tools/dotnet-interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2632TfmMI3aa4vM2.json
 0.0  0.2 dotnet interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2329WtGsCHtBFDBr.json
 0.0  0.2 dotnet tool run dotnet-interactive -- notebook-parser
 0.0  0.2 dotnet interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2632TfmMI3aa4vM2.json
 0.0  0.2 dotnet tool run dotnet-interactive -- notebook-parser

At the moment my environment (on Mac OS) is:

$ sw_vers -productVersion
11.6.1

$ ~/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code --version
1.62.3
ccbaa2d27e38e5afa3e5c21c1c7bef4657064247
x64

$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  11.6
 OS Platform: Darwin
 RID:         osx.11.0-x64
 Base Path:   /usr/local/share/dotnet/sdk/6.0.100/

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  2.2.401 [/usr/local/share/dotnet/sdk]
  3.1.415 [/usr/local/share/dotnet/sdk]
  5.0.100 [/usr/local/share/dotnet/sdk]
  5.0.402 [/usr/local/share/dotnet/sdk]
  6.0.100 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.2.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.21 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.11 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.6 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.21 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Also I was surprised that I had to install .NET Core 3.1, otherwise .NET Interactive/Notebook refused to start kernel and execute cells. That wasn't required with VS Code Insiders.

zangruizhe commented 2 years ago

I've been experiencing this issue for some months now. But I was using VS Code Insiders, so I reckoned that it might be still unstable/experimental feature, and so i wasn't sure if I should create a bugreport. But today I've tried it out in regular VS Code, and the issue is present there too, so thanks to nasosev for creating this bugreport.

As it was already described, even after VS Code is closed, dotnet processes continue to run and load 100% of a core, wasting CPU and battery charge. Here's the output from htop:

CPU% MEM% Command
99.7  0.2 /usr/local/share/dotnet/dotnet /Users/vasya/.nuget/packages/microsoft.dotnet-interactive/1.0.258004/tools/net6.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser
99.6  0.2 /usr/local/share/dotnet/dotnet /Users/vasya/.nuget/packages/microsoft.dotnet-interactive/1.0.258004/tools/net6.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser
 0.8  1.7 /Users/vasya/.dotnet/tools/dotnet-interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2329WtGsCHtBFDBr.json
 0.0  0.6 /Users/vasya/.dotnet/tools/dotnet-interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2632TfmMI3aa4vM2.json
 0.0  0.2 dotnet interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2329WtGsCHtBFDBr.json
 0.0  0.2 dotnet tool run dotnet-interactive -- notebook-parser
 0.0  0.2 dotnet interactive jupyter --default-kernel csharp --http-port-range 1000-3000 /var/folders/5x/jyhk_09s1m53tt41l55xtgmw0000gn/T/tmp-2632TfmMI3aa4vM2.json
 0.0  0.2 dotnet tool run dotnet-interactive -- notebook-parser

At the moment my environment (on Mac OS) is:

$ sw_vers -productVersion
11.6.1

$ ~/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code --version
1.62.3
ccbaa2d27e38e5afa3e5c21c1c7bef4657064247
x64

$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  11.6
 OS Platform: Darwin
 RID:         osx.11.0-x64
 Base Path:   /usr/local/share/dotnet/sdk/6.0.100/

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  2.2.401 [/usr/local/share/dotnet/sdk]
  3.1.415 [/usr/local/share/dotnet/sdk]
  5.0.100 [/usr/local/share/dotnet/sdk]
  5.0.402 [/usr/local/share/dotnet/sdk]
  6.0.100 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.2.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.21 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.11 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.6 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.21 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Also I was surprised that I had to install .NET Core 3.1, otherwise .NET Interactive/Notebook refused to start kernel and execute cells. That wasn't required with VS Code Insiders.

the same issue, "dotnet microsoft.dotnet-interactive" suddenly use 100% CPU after vscode close.

abelperezok commented 2 years ago

I have the same issue, I'm using Linux (Debian 11) VS Code 1.63.2 I notice it's consistent with the previous comments, after closing VS Code, that process starts to spike CPU and I never used the notebook so I think it's not necessarily related to having opened the notebook while using VS Code prior to closing it and seeing the CPU 100%.

eduarddejong commented 2 years ago

Hi, I am experiencing this issue on Arch Linux, using Code version 1.69.2-1 (not some version from AUR, but the default version, installed using just the pacman package manager). The same happened in previous versions too (since the first time that I installed the .NET Interactive extension I guess).

It happens basically every time after I simply click to activate the ".NET INTERACTIVE" tab next to the Terminal tab, which starts up .NET Interactive automatically.

Symptoms: 1 CPU core quickly shoots up to 100% usage and continues to do so after exiting Code. To get rid of it I really need to manually kill the still running dotnet process (for example by starting htop and send it a SIGTERM).

So, in other words: there's clearly an infinite loop going on in some program thread.

Eddie-Hartman commented 2 years ago

Still experiencing this issue. Any word on a fix? M2 Air on latest VsCode and polyglot interactive notebook. VsCode 1.73.1 Polyglot v1.0.3553070

Let me know if you need more info that could help.

acesnik commented 1 year ago

I'm experiencing this infinite-loop-like behavior when using Polyglot Notebooks in VS Code, as well. I had to uninstall Polyglot to get it to stop.

macOS 13.3 (Intel i9) VS Code 1.77.3 Polyglot v1.0.4212010

jonsequitur commented 1 year ago

I'm not able to repro this on my M1 MacBook Pro. If you have a repro, could you also let use know:

acesnik commented 1 year ago

Steps:

I'm using .ipynb notebooks in VS Code with python 3.9 kernels using conda environments. Even without any notebooks having .NET Interactive selected for the kernel, i.e. only python kernels selected, this spawns a dotnet process that runs infinitely, even if VS Code is quit.

Other possibly related extensions:

.NET Extension Pack
v1.0.12
C#
v1.25.4
Dev Containers
v0.288.0
Docker
v1.24.0
Ionide for F#
v7.5.2
Jupyter
v2023.3.1201040234
Jupyter Cell Tags
v0.1.8
Jupyter Keymap
v1.1.0
Jupyter Notebook Renderers
v1.0.15
Jupyter Slide Show
v0.1.5
Polyglot Notebooks
v1.0.4212010
reStructuredText
v189.3.0
Vim
v1.25.2
jonsequitur commented 1 year ago

Thanks for the details, @acesnik!

This repros for me with Python Jupyter .ipynb files, not with Polyglot Notebooks .ipynb files.

One guess is that the runaway process is the notebook parser and because Polyglot Notebooks isn't handling the file, this never gets cleaned up when it's closed. @brettfo, thoughts on that?

brettfo commented 1 year ago

It shouldn't be the notebook parser; I think it's only invoked for .dib files. My best guess is that the backing process is coming from this line, which would indicate a problem with the isDotNetNotebook() check a few lines up.

Is there a way to see what arguments the ever-running process was started with? That should tell us which notebook it was (or if it was, indeed, the notebook parser).

jonsequitur commented 1 year ago

This happens with only a single, new notebook document (targeting Python), so it's obvious which notebook it is. Two processes are created. One of them ramps up to 100% CPU after closing VS Code.

jonsequitur commented 1 year ago

It shouldn't be the notebook parser; I think it's only invoked for .dib files.

Then why are two processes created for only one file?

frr149 commented 1 year ago

Same thing happening on Fedora 37 with python-only notebooks. Any news on a fix?

MatisseHack commented 1 year ago

I can reproduce this issue with both Jupyter .ipynb files (e.g. using a Python kernel) and Polyglot .dib/.ipynb files. However, I was surprised to find that it's easier to reproduce with Jupyter notebooks than it is with Polyglot notebooks. The issue arises almost every time I close VS Code after editing a Python Jupyter notebook, but I only see the issue with Polyglot notebooks if I have several extensions enabled.

I spent some time today enabling and disabling different extensions to see if I could find any particular ones that cause the issue with Polyglot notebooks, but was not able to narrow it down. I can reproduce the issue with different sets of extensions enabled so it's not specific to a single one. Instead, I think it has something to do with the number of extensions enabled and potentially how much time they add to startup/shutdown? I can't reproduce the issue 100% consistently with any particular set of extensions installed, which also leads me to think it's a timing related issue potentially caused by a race condition somewhere?

However, the number of enabled extensions doesn't seem to matter when editing a Python Jupyter notebook. Even with only the Polyglot extension and its dependencies installed, these two dotnet processes get kicked off:

$ ps -o pid,%cpu,time,command -p $(pgrep dotnet)
  PID  %CPU      TIME COMMAND
48324   0.0   0:00.88 dotnet tool run dotnet-interactive -- notebook-parser
48327   0.0   0:00.24 /usr/local/share/dotnet/dotnet /Users/mhac/.nuget/packages/microsoft.dotnet-interactive/1.0.420401/tools/net7.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser

And after quitting VS Code, one process hits 100% CPU usage:

$ ps -o pid,%cpu,time,command -p $(pgrep dotnet)
  PID  %CPU      TIME COMMAND
48401   0.0   0:00.75 dotnet tool run dotnet-interactive -- notebook-parser
48402 100.0   0:08.90 /usr/local/share/dotnet/dotnet /Users/mhac/.nuget/packages/microsoft.dotnet-interactive/1.0.420401/tools/net7.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser

Compare that to editing a Polyglot (.dib) notebook, which launches four dotnet processes:

$ ps -o pid,%cpu,time,command -p $(pgrep dotnet)
  PID  %CPU      TIME COMMAND
46545   0.0   0:00.95 dotnet tool run dotnet-interactive -- notebook-parser
46548   0.0   0:00.29 /usr/local/share/dotnet/dotnet /Users/mhac/.nuget/packages/microsoft.dotnet-interactive/1.0.420401/tools/net7.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser
46551   0.0   0:00.95 dotnet tool run dotnet-interactive -- [vscode] stdio --working-dir /Users/mhac/Downloads
46553   0.0   0:02.08 /usr/local/share/dotnet/dotnet /Users/mhac/.nuget/packages/microsoft.dotnet-interactive/1.0.420401/tools/net7.0/any/Microsoft.DotNet.Interactive.App.dll [vscode] stdio --working-dir /Users/mhac/Downloads

And after quitting VS Code, one process hits 100% CPU usage (as long as there are a couple of other extensions installed):

$ ps -o pid,%cpu,time,command -p $(pgrep dotnet)
  PID  %CPU      TIME COMMAND
46630   0.0   0:00.75 dotnet tool run dotnet-interactive -- notebook-parser
46634 100.0   0:07.43 /usr/local/share/dotnet/dotnet /Users/mhac/.nuget/packages/microsoft.dotnet-interactive/1.0.420401/tools/net7.0/any/Microsoft.DotNet.Interactive.App.dll notebook-parser

One potentially interesting detail I just noticed: the process that hits 100% CPU usage has a different PID from the original process that was running while VS Code was running. In both cases (Jupyter notebooks and Polyglot notebooks) the new process that hits 100% CPU usage gets started right as VS Code is shutting down.

Please let me know if there is anything else I can do to help debug!

Meowtimer commented 1 year ago

This issue is with me everyday I open this dib file I use to query commits I made across some repos. As soon as I open it and then close vscode two dotnet processes appear/get stuck one of which consumes 100% of a core. I don't even have to run the notebook. QueryCommits.md (actually dib)

(Edit: Not exactly everytime, but very often. I tried with all extensions disabled except Juypter und Polyglot Notebooks)

Edit Edit: Actually, it really starts happening everytime when https://marketplace.visualstudio.com/items?itemName=Oracle.oracledevtools are enabled alongside Polyglot Notebooks 🫢

g0t4 commented 1 year ago

Me too, on macOS.

Reproduce:

  1. Open vscode
  2. Open an *.ipynb
  3. Close vscode

Observe:

Furthermore,