Excessive R sessions being spawned and not removed

[gowerc]

Describe the bug

It appears whilst using the extension an excessive number of R sessions are still being spawned and not released :(

To Reproduce

Use R + vscode extension + language server in a rocker/verse docker container (via attach vscode to running container) for an extended period of time.

Do you want to fix by self? (We hope your help!)

Yes, but realistically no as I don't think this is within my skillset to fix :(

(If yes,) what kind of help do you want? (e.g. Which file should I fix, Survey (related documents)

major guidance on where to even start :)

Config file

Not sure all this is needed but can provide if you think otherwise, main thing is I am using a standard R terminal not radian.

Expected behavior

R sessions to clean themselves up and not hog memory

Screenshots

Environment (please complete the following information):

• OS: Linux, Fedora 35
• VSCode Version: 1.67.2
• R Version: 4.1.2
• vscode-R version: v2.5.0

However I am running the extension in a docker container namely: rocker/r-ver:4.1.2

EDIT:

A screenshot of my R language server log:

[renkun-ken]

Would you like to provide some steps to reproduce the problem?

[gowerc]

If I'm honest this is a bit of a weird one, the problem is super reproducible in that it happens to me pretty much every day but I struggle to isolate a specific action to the problem, it just kinda occors over time if I spend an extended period of time developing in docker + vscode. i.e. after an hour of general development I check my activity monitor and see the above screenshot :(

General workflow though is:

• Spawn R docker container based on rocker/verse
• attach vscode to the running container
• Proceed to work in active container with vscode
• Notice tons of lingering processes

Will test a few more things like parallel processing and other commands to see if I can reproduce in more detail.

For reference though how many language server processes should there be if everything was running as expected ?

[eitsupi]

Note: Consider using the R definition included in the Remote - Containers extension. This is generally the preferred method over attaching vscode to an existing container.

[gowerc]

I had forgotten about this feature, thanks for pointing it out. Will give it a try and see if I still get the same problem

[gowerc]

Nope tried using the devcontainer approach but am still seeing the same issue :(

I wondered it it might be connecting to running parallel code but that didn't see to force the issue to occur

[gowerc]

Ok, I managed to get it to be quite reliably reproducible. If I simply open a file run some code then switch to another file and run some more code it seems to quite often trigger it. It feels like its more often triggered by running devtools::load_all() but I've had cases where its done it despire never running devtools. A short clip to demo this:

https://user-images.githubusercontent.com/8447209/169708873-21735df5-795c-4bda-ab63-fc7a1533e31a.mp4

EDIT:

The code repo I'm using if you want to experiment with my exact setup is: https://github.com/gowerc/age-of-statistics . The container is launched from the "backend" folder

[albertosantini]

Hey. Giving a quick look at the code, it seems the stack trace is the following one:

In util.ts

• executeRCommand
• spawnAsync
• spawn
• adding the process to asDisposable

It seems to me the process is disposed only when the extension is deactivated or vscode quits, if I am not wrong.

I am afraid it is missing, in this case, the trigger to dispose the process during a session.

[renkun-ken]

From your screencast, it is quite abnormal that multiple language servers are spawned to handles these files in the same workspace folder in the first place. If everything works properly, it is expected that only one language server will be spawned and should not exit until the workspace folder is closed or vscode exits.

Let me try with an environment in docker and see if I could reproduce it somehow.

[renkun-ken]

@gowerc Would you like to "toggle developer tools" and take a look at the logging in the console.

If each file triggers a new language server process, I guess it is something wrong with the file URI and the way we check if a file belongs to the current workspace folder does not work properly in your workspace for some reason.

[renkun-ken]

I tried with both a fresh container and your age-of-statistics dev container environment, but cannot reproduce the problem. Only one language server is spawned to handle the files in the same workspace folder, as expected.

[gowerc]

Apologies I've headed out for a week but will update this with the details you asked for when I get back, apologies for the delay. EDIT: I will try on a separate machine as well to see if I can reproduce the problem on different computers.

[gowerc]

Ported over my project from Fedora to Windows + wsl and am seeing the same problem, inspecting the console didn't reveal anything obvious:

Will retry after disabling all extensions to see if it from a potential conflict :S

[gowerc]

Potentially something here ?

[gowerc]

i disabled all extensions (except R and remotes) and removed all settings / keyboard shortcuts and am still seeing the same issue :(

[renkun-ken]

From your screenshot, it looks like you are always trying to open a file from temp dir. Are you using go-to-definition to inspect the function body in packages? If so, does it still happen if you don't go-to-definition and only edit files in your workspace?

Currently, each file outside workspace uses a language server started from the parent folder of that file. I guess it could be improved by letting those files share one language server started from the same parent folder.

[adalisan]

Ok, I was not sure if I should open a new issue, but I have nearly the same problem: Reditorsupport R extension creates new R processes that don't die even after I exit the remote sessions. So my setup is that I use VS Code in a remote SSH setting connecting to a server. I have installed vscode-R extension in that vscode-server.

Even long after I exited that session Reditorsupport still is running the R processes (in the screenshot above) and using a lot of CPU for some reason.

[D3SL]

Looks like the same problem I have from #728, which it turns out wasn't actually fixed. Have you tried disabling the language server? That fixed it for me.

[albertosantini]

I think there are three scenarios:

1. R sessions being spawned and not removed with a local setup (fixed via https://github.com/REditorSupport/vscode-R/pull/941)
2. R sessions being spawned and not removed with a remote setup
3. A lot of R sessions being spawned due to the language server

I suggest to open two new issues for points 2) and 3), closing the old ones, because suggestions and working progress are different and actually the three scenarios are mixed in the issues.

• 1st point was related to the async nature of closing sessions in quitting handler of VSCode
• 2nd point is related to (guessing) how to intercept the disconnection from the remote host to close the processes
• 3rd point is related to (guessing) how to language server pool is implemented

@renkun-ken @Yunuuuu What do you think?

[renkun-ken]

@albertosantini Thanks for summarizing the issue. I tested with the following setups:

• Local: macOS 12.6, R 4.2.0
• Local: Ubuntu 22.04, R 4.2.0
• Local: Windows 10, R 4.2.1
• Local: macOS 12.6, Remote: Ubuntu 18.04, R 4.1.3
• Local: Ubuntu 22.04, Remote: Ubuntu 18.04, R 4.1.3
• Local: Windows 10, Remote: Ubuntu 18.04, R 4.1.3

Under all above setups, I cannot reproduce the problem where some R processes (help server, language server and child processes, etc.) are left unkilled when vscode exits.

Would anyone experiencing such problem share your setup in the following aspects:

• Local or Remote?
• Local OS, R version and Remote OS, R version (if applicable)
• What R processes are left open after vscode exits? On Windows, you can see it task manager details when command arguments are shown. On macOS and Linux, run ps aux | grep "exec/R" in a terminal.

Do you have a minimal sequence of steps to reproduce the problem? For example, just open a project which contains R files, and vscode-R extension will be activated, and a help server will be spawned in the background. Opening any R file in the workspace will start the language server. The language server, by default, will start 6 child processes to handle requests.

$ ps aux | grep "exec/R" | grep -v grep
ken              29163   0.7  1.4 34018604 239260   ??  S     3:42PM   0:02.80 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --silent --slave --no-save --no-restore -e base::source(base::commandArgs(TRUE)) --args /Users/ken/.vscode/extensions/reditorsupport.r-2.5.3/R/languageServer.R
ken              29155   0.3  0.5 33821464  82792   ??  S     3:42PM   0:00.90 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --silent --slave --no-save --no-restore -e base::source(base::commandArgs(TRUE)) --args /Users/ken/.vscode/extensions/reditorsupport.r-2.5.3/R/help/helpServer.R
ken              29270   0.0  0.3 33792952  55048   ??  Ss    3:42PM   0:00.22 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --no-readline --slave --no-save --no-restore
ken              29195   0.0  0.3 33792952  54040   ??  Ss    3:42PM   0:00.30 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --no-readline --slave --no-save --no-restore
ken              29194   0.0  0.3 33792952  55284   ??  Ss    3:42PM   0:00.30 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --no-readline --slave --no-save --no-restore
ken              29193   0.0  0.5 33809320  79336   ??  Ss    3:42PM   0:00.72 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --no-readline --slave --no-save --no-restore
ken              29189   0.0  0.5 33833616  76400   ??  Ss    3:42PM   0:00.67 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --no-readline --slave --no-save --no-restore
ken              29187   0.0  0.4 33793976  67028   ??  Ss    3:42PM   0:00.38 /Library/Frameworks/R.framework/Versions/4.2/Resources/bin/exec/R --no-readline --slave --no-save --no-restore

Above are all R processes vscode-R and langaugeserver spawn in the background with default settings.

[albertosantini]

@renkun-ken Thanks for the quick reply.

My point was to highlight the closing part of the processes in a local scenario is fixed, intending the extension follows correctly the lifecycle of vscode and there is no bugs there, as explained in #941. I think the reports about not closing processes are only related to remote setup.

In the remote setup, with remote extension and vscode-sever running behind the scenes, I suspect the lifecycle of an extension, spawning processes as vscode-R, it is not the same as local one. When the user closes the remote workspace, quitting vscode or changing project, the processes are not closed because vscode-server (the parent process) continues to run.

Indeed, reading @adalisan list of processes above, you can see vscode-server instance opened and other processes of other extensions (not only vscode-R) running. From the docs https://code.visualstudio.com/docs/remote/faq#_how-do-the-remote-development-extensions-work See "Extn Child Processes". Really here I would open an issue about this behaviour in https://github.com/microsoft/vscode/issues, as we (you) did asking for the async nature in the quitting handler and then we resolved it..

At last, the other issue is not about closing the processes, but how many processes are started for language server, help server, etc. In my configuration I have been using R_LANGSVR_POOL_SIZE=0 (in .Renviron). Anyway here (local setup) the point is number of the processes not if they are closed or not.

Of course in a remote setup, you will see a lot of processes opened and thery are not closed, when vscode quits, but they are two different issues.

Recap.

Firstly I would attack the issue about remote setup, opening an issue in vscode repo, highlighting the local setup is ok (about closing the processes, maybe remind them that old issue describing the problem, then resolved, just to give them a bit of context). Only later I would analyze the pool management.

What do you think?

[renkun-ken]

I tested with remote development under the following setups as I mentioned above, but could not reproduce the problem.

• Local: macOS 12.6, Remote: Ubuntu 18.04, R 4.1.3
• Local: Ubuntu 22.04, Remote: Ubuntu 18.04, R 4.1.3
• Local: Windows 10, Remote: Ubuntu 18.04, R 4.1.3

I'm curious about the setup that has the problem. Maybe some settings (remote.*) regarding remote development extensions cause the problem.

[albertosantini]

@renkun-ken Very likely.

Also vscode-server is started with the option --enable-remote-auto-shutdown, but some users need to kill manually the processes: https://www.jibsheet.net/linux/index.php/2022/05/02/how-to-kill-all-vscode-process-on-remote-linux-server/

[albertosantini]

Sorry for my compulsive quest... a few closed issues:

• https://github.com/microsoft/vscode/issues/138310
• https://github.com/microsoft/vscode-remote-release/issues/1244
• https://github.com/Microsoft/vscode/issues/50763

[renkun-ken]

Under my setups, when no vscode instances are open, the vscode-server processes on the remote linux servers are persistent unless I manually kill them.

$  ps aux | grep vscode-server | grep -v grep
ken      1690130  0.0  0.0   2608   596 ?        S    09:54   0:00 sh /home/ken/.vscode-server/bin/64bbfbf67ada9953918d72e1df2f4d8e537d340e/bin/code-server --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level all --connection-token-file /home/ken/.vscode-server/.64bbfbf67ada9953918d72e1df2f4d8e537d340e.token
ken      1690140  0.1  0.0 1037644 200876 ?      Sl   09:54   1:05 /home/ken/.vscode-server/bin/64bbfbf67ada9953918d72e1df2f4d8e537d340e/node /home/ken/.vscode-server/bin/64bbfbf67ada9953918d72e1df2f4d8e537d340e/out/server-main.js --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level all --connection-token-file /home/ken/.vscode-server/.64bbfbf67ada9953918d72e1df2f4d8e537d340e.token
ken      1690180  0.3  0.0 732984 71164 ?        Sl   09:54   1:58 /home/ken/.vscode-server/bin/64bbfbf67ada9953918d72e1df2f4d8e537d340e/node /home/ken/.vscode-server/bin/64bbfbf67ada9953918d72e1df2f4d8e537d340e/out/bootstrap-fork --type=ptyHost --logsPath /home/ken/.vscode-server/data/logs/20221009T095454

[albertosantini]

This issue is very close to our scenario: https://github.com/microsoft/vscode-jupyter/issues/1626

Really I didn't understand the solution or the cause. Maybe temporary files locking the shutdown of the spawned process. Main comment: https://github.com/microsoft/vscode-jupyter/issues/1626#issuecomment-1102628338. It contains also a test scenario for their issue.

Any light bulb after reading that? ^ About file watcher or files (guessing)

About "files", another hypothesis is remote workspaces based on NFS filesystem, but this is unlikely, I didn't read any comment about this.

[albertosantini]

@gowerc @adalisan

This is not directly related, but it may give some hint: https://earlruby.org/2021/06/fixing-vscode-when-it-keeps-dropping-ssh-connections/

It seems that VSCode keeps a cache of data and code on the remote machine, and something in a VSCode update was trying to do something using the bits in the old cache data that was no longer supported.

• Exit completely out of VSCode so that no VSCode processes are running. Force quit if you have to.
• ssh to the remote machine(s) and delete the ~/.vscode-server directory with rm -Rf ~/.vscode-server/
• If you get any “cannot remove [file]: Device or resource busy” errors then look for stuck processes:
• lsof | grep $HOME/.vscode-server | awk '{ print $2 }' | sort -u

I am interested to that print.

• then kill those processes, then trying removing the directory again.
• Restart VSCode.

After closing VSCode is there any process running or is it ok (no processes)?

[gowerc]

Will try and post my setup shortly when I get some time. I was able to re-produce the issue on my machine pretty reliably a couple of months ago but never got around to documenting it sorry.

My rough steps were:

• Launch Rocker container with a folder from the local file system mounted as a volume which contains my source code
• Use ssh remotes in VScode to open the container
• Open multiple R files and run small snipets of code from each of them

Closing vscode would kill all the processes, my issue was more that the language server would keep spawning spurious proccess until my machine eventually ran out of RAM. At least that was my setup in the video I posted here: https://github.com/REditorSupport/vscode-R/issues/1107#issuecomment-1133943856 (that was being run on a fedora machine I think on a ubuntu docker container)

The htop screenshot in the OP shows the excessive number of processes being spawned alongside their resource consumption. They just slowly accrue over time, after an hour of working I usually have to either do a restart of vscode or run a script to find and kill all R processes.

[D3SL]

• Local or Remote?
• Local OS, R version and Remote OS, R version (if applicable)
• What R processes are left open after vscode exits? On Windows, you can see it task manager details when command arguments are shown. On macOS and Linux, run ps aux | grep "exec/R" in a terminal.

As I mentioned in my issue #728 I get a large number of "R for windows frontend" and "R for windows terminal frontend" processes when local, on windows 10, with both the most recent and last several R versions.

It's extremely strange that something I can and multiple others reproduce at will doesn't occur for you at all and makes me think there's an environment interaction occurring. Perhaps something unique about your personal installation that the rest of us don't have.

[renkun-ken]

It's extremely strange that something I can and multiple others reproduce at will doesn't occur for you at all and makes me think there's an environment interaction occurring. Perhaps something unique about your personal installation that the rest of us don't have.

Do you have access to any clean Windows VM on e.g. Azure to reproduce the problem? I was able to reproduce the previous problem on a Windows VM on Azure, and only then I was able to fix it. Now, with a Windows VM on Azure, I cannot reproduce the problem stated above. It would be helpful if it could be reproduced in a clean environment.

[github-actions[bot]]

This issue is stale because it has been open for 365 days with no activity.

[gowerc]

Not-stale