Remote VSCode over SSH crashes EC2 instance

Zenexer commented 4 years ago

Issue Type: Bug

I've been attempting to use the new remote VSCode feature to work with a project stored on an AWS EC2 instance. Each time I use it, it works fine for a few hours. Eventually, the whole instance stops responding. AWS indicates that the instance is unresponsive in the control panel, and I have to force-stop it. The screenshot/log feature on AWS doesn't show anything. Once I boot the instance back up, there's nothing in the logs--they just cut off at the time the instance stopped responding. I wish I had more information to give you, but I'm at a loss of how to troubleshoot this.

Other notes:

If I leave htop or top open, when the instance finally crashes, there's no indication of anything unusual. Plenty of memory, etc.
VSCode complained about fs.inotify.max_user_watches being too low when I first started using it remotely. I increased it per VSCode's instructions and confirmed that it took effect. The warning went away, but the crashes still happen.
Even if I disconnect from the remote session, the instance will still crash.
Instance type: t3a.micro
Region: us-east-1

lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:        19.10
Codename:       eoan

cat /proc/cpuinfo

% cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7571
stepping        : 2
microcode       : 0x8001250
cpu MHz         : 2199.958
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni
pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4399.91
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management:
...

VS Code version: Code 1.43.2 (0ba0ca52957102ca3527cf479571617f0de6ed50, 2020-03-24T07:38:38.248Z) OS version: Windows_NT x64 10.0.18363

System Info

|Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-8086K CPU @ 4.00GHz (12 x 4008)| |GPU Status|2d_canvas: enabled
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
protected_video_decode: unavailable_off
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
viz_display_compositor: enabled_on
viz_hit_test_surface_layer: disabled_off_ok
webgl: enabled
webgl2: enabled| |Load (avg)|undefined| |Memory (System)|31.95GB (18.31GB free)| |Process Argv|| |Screen Reader|no| |VM|0%|

Extensions (4)

Extension|Author (truncated)|Version ---|---|--- remote-ssh|ms-|0.51.0 remote-ssh-edit|ms-|0.51.0 remote-wsl|ms-|0.42.4 cpptools|ms-|0.27.0

Zenexer commented 3 years ago

Add me to the list, can reliably crash any spec EC2 instance (tried lightsail too which is basically an EC2 instance anyway) in a matter of minutes by just using vs code remote-ssh. Interestingly though, I can happily code php based projects for hours with no problem using remote-ssh, its when I'm doing react based projects that the server will crash. Cannot do anything other than forcefully shut it down on AWS and restart. Could be an issue with node.js?

It's been a long time since I've attempted to reproduce this (I gave up on the whole concept of remote VSC), but when I first reported the issue, I had no trouble reproducing it with a PHP project. The project language didn't seem to play a role, and I was never able to narrow down the cause further than what I initially reported.

sholtomaud commented 3 years ago

Your report is concerning but I can't reproduce it. I work with vscode remote frequently on an ubuntu cloud VM with an uptime of almost a year. I would need more info but I'm not even sure what to ask for. The easy possibility is that you have some remote extension installed which is causing issues.

@roblourens can you please share a repo/gist which has your VSCode settings.json & ~/.ssh/config and the specification of your EC2 instance (in AWS CDK would be useful) so we can reproduce your working setup.

wd40bomber7 commented 3 years ago

I'm reliably hitting this on Azure VMs when working on javascript projects remotely. The VM completely locks up and becomes unusable. Very frustrating!

Foochapp commented 3 years ago

Hey everyone. Thanks for all the comments and help. I finally could fix the problem, at least from my side.

I had the very same problem with VSCode connecting to my EC2 instance. However, this wasn't the first time that I had VSCode running with an EC2, and the other server that I had, was running smoothly. What was the difference between my old server (that is actually still running) and the new one? That in the old one, I was running an Apache server with only PHP code, so basically, I was only using VSCode for setting files and PHP.

I created a new server, started writing PHP again and everything was going fine. The problem came when I started writing JS code in a Node.js, and suddenly became very slow and began crashing, just with the same symptoms described above.

So, it looks like VSCode has an extension built-in in a release called "TypeScript and JavaScript Language Features" but you can find it easily by looking at the extensions box and writing: "@builtin typescript". Find it and disable it and you'll be just fine.

JakeTompkins commented 3 years ago

Hey everyone. Thanks for all the comments and help. I finally could fix the problem, at least from my side.

I had the very same problem with VSCode connecting to my EC2 instance. However, this wasn't the first time that I had VSCode running with an EC2, and the other server that I had, was running smoothly. What was the difference between my old server (that is actually still running) and the new one? That in the old one, I was running an Apache server with only PHP code, so basically, I was only using VSCode for setting files and PHP.

I created a new server, started writing PHP again and everything was going fine. The problem came when I started writing JS code in a Node.js, and suddenly became very slow and began crashing, just with the same symptoms described above.

So, it looks like VSCode has an extension built-in in a release called "TypeScript and JavaScript Language Features" but you can find it easily by looking at the extensions box and writing: "@Builtin typescript". Find it and disable it and you'll be just fine.

Yeah, I think we'd pretty much established that the problem was from the TS and JS features hogging too much memory, but unfortunately, disabling that makes for a pretty lousy development experience. It would be much more ideal if the problem itself could be remedied instead of sacrificing the quality of the experience.

keidikapllani commented 3 years ago

Hey everyone. Thanks for all the comments and help. I finally could fix the problem, at least from my side. I had the very same problem with VSCode connecting to my EC2 instance. However, this wasn't the first time that I had VSCode running with an EC2, and the other server that I had, was running smoothly. What was the difference between my old server (that is actually still running) and the new one? That in the old one, I was running an Apache server with only PHP code, so basically, I was only using VSCode for setting files and PHP. I created a new server, started writing PHP again and everything was going fine. The problem came when I started writing JS code in a Node.js, and suddenly became very slow and began crashing, just with the same symptoms described above. So, it looks like VSCode has an extension built-in in a release called "TypeScript and JavaScript Language Features" but you can find it easily by looking at the extensions box and writing: "@Builtin typescript". Find it and disable it and you'll be just fine.

Yeah, I think we'd pretty much established that the problem was from the TS and JS features hogging too much memory, but unfortunately, disabling that makes for a pretty lousy development experience. It would be much more ideal if the problem itself could be remedied instead of sacrificing the quality of the experience.

Even though the TS/JS builtin extensions seems to be highlighted the as the problem here, after disabling it on my environment I still experiences the EC2 instance crashing within minutes.

franp9am commented 3 years ago

I'm also facing this issue, but it's not AWS, but just a CentOS server, unrelated to AWS... Whenever I connect with VSCode, it most likely restarts. But I'm not sure if it's really a problem of VSCode, or a bug in Linux. Because VSCode has no sudo access and it shouldn't be that easy to kill a remote machine...

steelkorbin commented 3 years ago

I have been suffering from this issue for over a year. And I have done enough tests to know that the VS CODE Remote Plugin does not crash our systems. You can SSH into yourself on your own local machine, use VS CODE as you find yourself naturally doing on any instance/droplet/VM, just as if you were remote, and VS CODE will crash your local box. What I can say is that using SSH native or launched with this Remote plugin will illuminate some critical architectural flaws in VS CODE and how it watches files and the background processes that are trying to use node to do the job. This can also be observed without SSH if you capture the VS CODE launch locally because it will spike very quickly and resolve quickly, locally so we never see it until our projects get to normal sizes, then we complain about VS CODE being slow or using up all of our CPU, etc... We are only seeing a more exaggerated set of these symptoms of this issue when we add SSH into the mix, making it something we can not directly fix or repro with consistency because the Remote Plugin is not a problem, it actually works amazing. The real issue that we are looking at is much deeper in VS CODE itself and that ripples out to many other bug tickets listed here on Github that are directly related to the file watcher and background processes. Now MS must be fully aware of the technical details of this issue and does not see a way to correct it, meaning the technology has them locked in, or they are waiting for us to fix it for them through open source. Both are reasonable at this point. I for one am voting that we close this ticket because the Remote Plugin is working fine, if it was the problem and fixable @roblourens would have fixed it already, instead we have been chasing our tails. Close this ticket, the VS CODE Remote SSH plugin is not the problem, VS CODE itself is the problem here and a new ticket is in order plus a large pile of work to sort it out.

robertkruk commented 3 years ago

I've experienced server issues only when I've attempted to load a very large json file (e.g. 30mb) from the file view. It hasn't hard crashed the server but drops out for a few minutes (possibly bandwidth/session or memory limits?)

giltee commented 3 years ago

My whole team has started to get these issues over the past week. We added a significant amount of packages to our repo lately which lead me to believe that it could be related to inode file watchers (we have ours maxed out). What logs should I be checking to see the root of this issue?

glins97 commented 3 years ago

Same here... It has come to the point where my team and I need to move away from vscode.

SSun97 commented 3 years ago

I had the same issue and seems that the server crashed frequently when I was trying to push the commits to Github. It never happened After I stoping using VScode to push the commits. I just open a terminal, connect to the server and manually add commits and push to Github.

Kinda thinking if AWS has a protection mechanism that to maximum avoid "writing" to another public server

nortakales commented 3 years ago

I ran face first into this recently, and the first Google result brings you straight here. It seems like this is a common problem, and the end result is a completely unusable product. Can we get link(s) to issue(s) filed elsewhere if this issue is no longer being tracked here?

For what it's worth, I followed the above suggestion of deleting the *.log files in ~/.vscode-server and have not experienced a crash since.

Edit a few months later: actually, that solution did not work... it still crashes.

4cdc commented 3 years ago

https://code.visualstudio.com/insiders/ solved definitely the issue for me.

giddyhup commented 2 years ago

I'm late to the party but in the last two days I crashed an Oracle Cloud Ubuntu 20.04 instance twice by simply opening a freshly cloned bare minimum repository in a separate VSCode window.

levi-wj commented 2 years ago

I had this same issue. The way to solve it is to disable the built-in javascript/typescript extensions which max out the CPU and cause AWS to stall the server.

In the extensions tab, I searched for '@builtin' and disabled everything to do with typescript or javascript. After reconnecting, my CPU usage stayed at around 2-3% instead of spiking to 99% after a couple minutes.

keidikapllani commented 2 years ago

I have the same issue. I am trying to work on a Python Server (Flask) project in a t1.micro instance in AWS via VS Code Remote SSH and it crashes every few hours without notice or warning. I have to restart the server completely for it to come back to life.

I have tried all the above suggestion but the only thing that fixed it for me was disabling the Python extension completely which is less than ideal for a Python project

chriswatt commented 2 years ago

I ran into the same problem of the server locking up and needing a hard reboot. It would only happen after using using php xdebug functionality. I deleted the .vscode directory in my project, closed and reopened vscode. This seems to have fixed the issue for now at least.

clshortfuse commented 2 years ago

Hi again. I think the issue is the extremely large memory usage of the JS/TS Intellisense. I used to have an 8GB laptop and while debugging, the memory would just balloon. This was actually on WSL, not EC2 but the consequence was the same. The moment I hovered on something in JS, I was already telling myself "uh oh" because I knew Intellisense was kicking in. Then I'd get the same popup when I'm using SSH with EC2 that the connection died and I'd have to reload the window.

There probably need to be more work done on the Intellisense features because it seems to not work well when constrained. Probably more care needs to be done to be able to work with a limited set of memory because right now it doesn't look like any limit whatsoever works. It consumes as much as it can causing a crash on the host OS.

Replicating could be pretty easy with WSL, I'd assume. Create %USERPROFILE%\.wslconfig and set an instance's memory to something like 1GB. Connect to it with VSCode and try to load a library with a large Intellisense library (Anything with DOM references would probably do). And it should just die after a number of times triggering Intellisense.

The default maxTsServerMemory is 3GB, which is a bit silly. That essentially makes 4GB a bare minimum requirement on any instance. It should probably proportional to the host machine. 50% would make more sense.

"typescript.tsserver.maxTsServerMemory": 3072

clshortfuse commented 2 years ago

Okay, last comment. It's not this plugin. It's TypeScript Language Feature's TS Server.

https://github.com/microsoft/vscode/blob/e802791cf1df891aa81c7f57dc4ca57d54c0282a/extensions/typescript-language-features/src/utils/configuration.ts#L195-L201

https://github.com/microsoft/vscode/blob/53350bc6661449bc097496b04a1d8712474bd301/extensions/typescript-language-features/src/tsServer/serverProcess.electron.ts#L184-L185

https://nodejs.org/api/cli.html#--max-old-space-sizesize-in-megabytes

By default it will have a 3GB limit on it's Node instance. Once it hits that, Node crashes. There's no real memory leak prevention. It'll continue caching items until the instance runs out of memory and crashes. It's not designed in any way to alleviate the memory heap when pressured. The limit is only there to be raised, not lowered. On a machine where you don't have 3GB of RAM free, it'll keep consuming memory until it fills up the system memory, causing crazy CPU spikes and memory usage. The Node instance of TSServer doesn't get to crash. It already has locked up the system before that happens.

The quick, dirty fix is disable TSServer, or try to clamp the memory so it can crash "as intended". The long-term solution is to fix TSServer to release items from memory when pressured. I've looked and there's no code to do this. It should probably analyze ..getHeapStatistics() and total_available_size and start releasing CachedResponse items from RAM when it starts getting low. Right now, as I said, it's just going to consume until Node decides it should crash.

For the time being I'm going to file a bug with https://github.com/microsoft/vscode/issues and tag this.

Zenexer commented 2 years ago

It's TypeScript Language Feature's TS Server.

I wasn't using TS when I initially opened this bug report. I suspect that the issue you're encountering isn't the same as the one that initially prompted me to create this issue.

Additionally, the crashes were happening long after all VSCode-related processes terminated.

glins97 commented 2 years ago

This is easily reproducible for me when trying to view a large file. Maybe consuming all ram is cause, after all.

On Fri, Dec 24, 2021, 13:01 Paul Buonopane @.***> wrote:

It's TypeScript Language Feature's TS Server.

I wasn't using TS when I initially opened this bug report. I suspect that the issue you're encountering isn't the same as the one that initially prompted me to create this issue.

Additionally, the crashes were happening long after all VSCode-related processes terminated.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/2692#issuecomment-1000886591, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVCJAOTAMWKV2TUPRTXCZDUSSKNVANCNFSM4L4LP2OA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

clshortfuse commented 2 years ago

@steelkorbin

Close this ticket, the VS CODE Remote SSH plugin is not the problem, VS CODE itself is the problem here and a new ticket is in order plus a large pile of work to sort it out.

Well, I opened the ticket addressing specifically that the JS/TS plugin will crash a system because it defaults to consume upto 3GB of RAM and will inevitably crash a machine that doesn't have that available.

https://github.com/microsoft/vscode/issues/140090

It was closed and saying it's not really happening.

If people are seeing issues in real world use cases, we will take a look (but again, our telemetry does not suggest this is happening)

Right now however you have not provided evidence of an actual problem, just concerns. Sorry to be blunt but that's not helpful

I provided a laundry list of evidence and it was ignored. This isn't going to be fixed with any help from Microsoft looks like.

glins97 commented 2 years ago

Man... this is really frustrating.

On Thu, Jan 6, 2022, 16:55 Carlos Lopez @.***> wrote:

@steelkorbin https://github.com/steelkorbin

Close this ticket, the VS CODE Remote SSH plugin is not the problem, VS CODE itself is the problem here and a new ticket is in order plus a large pile of work to sort it out.

Well, I opened the ticket addressing specifically that the JS/TS plugin will crash a system because it defaults to consume upto 3GB of RAM and will inevitably crash a machine that doesn't have that available.

microsoft/vscode#140090 https://github.com/microsoft/vscode/issues/140090

It was closed and saying it's not really happening.

If people are seeing issues in real world use cases, we will take a look (but again, our telemetry does not suggest this is happening)

Right now however you have not provided evidence of an actual problem, just concerns. Sorry to be blunt but that's not helpful

I provided a laundry list of evidence and it was ignored. This isn't going to be fixed with any help from Microsoft looks like.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/2692#issuecomment-1006891299, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVCJAIWPASRE6KPBUCGSCLUUXXRRANCNFSM4L4LP2OA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Foochapp commented 2 years ago

@steelkorbin

Close this ticket, the VS CODE Remote SSH plugin is not the problem, VS CODE itself is the problem here and a new ticket is in order plus a large pile of work to sort it out.

Well, I opened the ticket addressing specifically that the JS/TS plugin will crash a system because it defaults to consume upto 3GB of RAM and will inevitably crash a machine that doesn't have that available.

microsoft/vscode#140090

It was closed and saying it's not really happening.

If people are seeing issues in real world use cases, we will take a look (but again, our telemetry does not suggest this is happening)

Right now however you have not provided evidence of an actual problem, just concerns. Sorry to be blunt but that's not helpful

I provided a laundry list of evidence and it was ignored. This isn't going to be fixed with any help from Microsoft looks like.

I gave access to 4 devs to the same environment on EC2 that I have. None of them had the same problem and we were all happy, until... I reinstalled VSCode and the problem started again. I disabled the module that was built in the VSCode and everything started working again.

The only difference between my 4 devs and I? -> All of them are working from Windows, I'm working from Mac.

Could that be something to analyze?

Greetings

bentontripp commented 2 years ago

I also am having the same problem, and it is driving me nuts. I'm using Python in a t1.micro instance and whenever I use the VS Code Remote SSH extension, it crashes whenever I attempt to edit files/run code. The only way to fix it is to restart the server completely. So far, NONE of the above suggestions have worked for me.

clshortfuse commented 2 years ago

I think we should take a step back and consider the reality that we are using an application meant for development on a production machine. I can't work on a Windows laptop with 8GB without WSL crashing. Expecting VSCode to run on a 1GB server instance is probably not a great idea. I tried to reach out to the VSCode team for reducing the possibility of it crashes in general in the JS/TS context, but I got gaslighted and ignored.

That said, I'll try to explore @Foochapp 's observation about this not happening on Mac, but I'm not too hopeful. I still have my 1GB RAM EC2 where I can reproduce crashes. Maybe there is a Windows client to Linux issue, but I think considering the fact VSCode doesn't care to use up all the memory on the system and request even more than the system even has, we're already working with a poor foundation.

Zenexer commented 2 years ago

Again, the issue many of you are observing is unrelated to the issue I originally reported. I was reporting kernel panics several hours after completely exiting VSC on the remote machine and ruled out a memory-related issue.

I don't think it makes sense to keep conflating these two issues, as they're unlikely to be related.

clshortfuse commented 2 years ago

@Zenexer I'm well aware, but you're still running this on a 1GB t3a.micro. You were getting fs.inotify.max_user_watches issues which is directly tied to memory usage. VSCode will cripple any 1GB machine, causing any processes to stall and die. That includes system processes, not just itself. Our fastest indication that VSCode killed a machine is that it own process. It might killing something else and you're experiencing the timebomb because it killed some sort of watchdog.

Note that @roblourens said:

Hm, I would guess that memory is somehow the issue here even though you say it doesn't seem to be using a lot.

The reality is VSCode will create an unstable environment. Once a system hit full memory even for second, there's no telling what's going to die as a result. The next process to ask for more memory will die. Also, there's the point of not releasing things like handles or watches can cause an issue later when resources you don't see in just htop are at their max and never properly released. In order words, the kernel still thinks they are at their max.

That's why I hammer the issue of preventing VSCode from being allowed to pressure the system to 100%. I don't mind if VSCode crashes because I underspec'd its working environment. I just don't want the system to crash. The easiest reproducible is starting up TS/JS with something as simple as the ES5 DOM typings and watching the entire system, not just VSCode, die. This shouldn't be allowed at all.

Zenexer commented 2 years ago

@clshortfuse I later ruled that out by testing with more memory.

I can confirm what @steelkorbin has observed; it doesn't appear to be related to resource usage. No matter the instance type, using VSCode Remote arms a kernel time bomb. Even if VSCode Remote is exited, eventually the instance will crash--hard. No logs, no way to debug the issue. It could happen hours after VSCode Remote has been closed.

Furthermore, there was no indication in any logs of the Linux OOM killer kicking in. I wasn't using TS/JS or any other language-specific extensions.

giddyhup commented 2 years ago

I agree with @clshortfuse, VSCode should not bring down the whole machine. Let it die gracefully or ungracefully but don't put the whole VM into a state where pulling the plug is the only solution.

Edit: I gave up using a free-tier Oracle instance and now used a paid EC2 with more memory. I understand now that the first machine was underpowered memory-wise but it supported VSCode until hard-locked up.

nortakales commented 2 years ago

For what it's worth, all I have to do is connect VSCode to my t2.micro instance, do absolute nothing in the editor after connecting, and the entire instance crashes after some time. This has been the case with both Windows 10 and 11 for the client and AL2 on the EC2 host.

Foochapp commented 2 years ago

For what it's worth, all I have to do is connect VSCode to my t2.micro instance, do absolute nothing in the editor after connecting, and the entire instance crashes after some time. This has been the case with both Windows 10 and 11 for the client and AL2 on the EC2 host.

Have you tried disabling Typescript module on your VSCode? Or, are you sure you have it disabled?

Also, I've been able to reproduce the same problem with another PHP plugin that was consuming around 20% of memory as soon as I connected to the server, and it just got worst over the following minutes until it crashed:

Of course, I uninstalled the module and everything went back to normal. So my guess here is that the problem could be that, there is a lot of modules that are "intellisense" types that runs when you edit your files directly into the server, and that could be causing the memory consumption until crash. That's why probably everyone has had the same problem but with different situations.

My advice is that whenever you realize your server is leaking memory, check your modules/extensions on VSCode first because you, or someone else, could have something there without even knowing.

prashant9912 commented 2 years ago

Search for ‘@builtin TypeScript’ in plugin and disable it restart ec2 instance and connect again it will work.

dudeful commented 2 years ago

Any updates on this issue? I have been able to fix it by disabling the '@Builtin TypeScript' extension, but it takes away very nice (and even important) features for my dev scenario. I know that the current robust solution is to get a bigger machine with 2gb+ of ram, but it is really the only way to go in order to use vscode with all the built-in features? Btw, thanks to the community for providing a temporary solution.

s6fcda commented 2 years ago

I also had my small AWS instance crashed quite a lot of times when I used this extension.

System info:

Instance type: t3.small
OS: CentOS 7.7

I first noticed that there was something wrong with the server when the auto-complete didn't work anymore (for example, putting a dot after an object doesn't give me any options). Then if I tried to do something in my SSH window, it didn't respond at all. Looking at the AWS CloudWatch monitoring, I didn't see abnormal spikes on my CPU usage, BUT I saw a crazy spike for "disk read".

This is a Sum-5-minute graph for disk usage. As you can see that it went over 9GB per 5 minutes. During that period of high disk usage, since SSH didn't respond, I couldn't do anything to stop any process or get any information for troubleshooting, and just hoped that it would stop by itself - it sometimes would. At the end, after waiting for 30 minutes I still couldn't gain control, so I restarted the instance. On the next day I tried again, and it spiked up to 5GB disk read, and I had to restart the server. It happened twice on that day and I gave up using this extension.

That is an AWS instance for my personal project, and it runs without any issues if I don't this extension, so I'm pretty sure that this is problem is caused by this extension. I would love to get more information when this happens, but as I mentioned even SSH didn't respond (or super slow like 30-second response time), I can't get anything real time. If I leave "top" open until it crashes, I don't see any abnormal numbers for CPU and memory (well yeah it uses a lot of memory, but it is not exhausted). If there is any logs that I can grab after it crashes, I'm very happy to share.

xiandong79 commented 2 years ago

seem that in 2022,the latest VS code still have this issue on Linux 18.04 AWS t3.m server.

clshortfuse commented 2 years ago

@s6fcda Is your issue repeatable? I never really tracked disk usage. I know when memory is pressured that watchers will fail and VSCode may keep trying to load them. I have also heard of logs getting big. Can you see if there is something in ~/.vscode-server that reports what it was doing to cause that spike?

s6fcda commented 2 years ago

@s6fcda Is your issue repeatable? I never really tracked disk usage. I know when memory is pressured that watchers will fail and VSCode may keep trying to load them. I have also heard of logs getting big. Can you see if there is something in ~/.vscode-server that reports what it was doing to cause that spike?

@clshortfuse Regarding "repeatable", my server froze a few times when I used this plugin, but I can't tell when exactly it happened, so I'm unable to really reproduce it on purpose.

I had a look at ~/.vscode-server, but I don't see any abnormal logs there. I even tried to look for the records around the time with the disk usage spike, but no error was found at around that time. Log file size also seems normal. So looking at that folder, it seems like nothing bad has happened before...

alexanderthenotsobad-git commented 2 years ago

I'm sorry, I was not able to read the totality of this thread, but yeah.

I'm having the same issue with a server on a DreamHost shared service. Same symptoms as have been reported on here by many:

My intellisense stopped working all of a sudden
My JS/TS language feature started crashing (.... TS/JS language feature died 5 times in 5 minutes...)
My remote VS Code server has an extrodinarily large node folder (not node_modules) NODE folder in the .vscode-server/bin/7898720381740387143.../ dir
The remote server just drops my connection randomly after usage
I kill the .vscode-server processes before logging back in with VS Code, and it will work, sometimes for a while, sometimes for a few seconds, then drops my connection.

Again, I'm on a shared server, not a Virtual Private Server (VPS), which is akin to an AWS EC2 instance, but the symptoms and the outcome are the same. Dreamhost does not have all the fancy dashboard features AWS does, but everything I've read on here seems to match my predicament. Good luck to us....

Cyclotron555 commented 2 years ago

An alternate Solution maybe it helps someone: I'm a student working on a project and came across this issue, I don't think there is a solution at least for me, I disabled the typescript/js add on and after about 4 hours the cpu was spiking at 100% again. I'm sure that most of these guys are pros and they already know but for a novice like me you should know that you can use an add-on from the VS code marketplace called ftp-simple and it works exactly like ssh, the only difference is that you won't have access to a terminal but I mean you can use putty or use EC2 instance connect in aws. Plus I think using this add-on gives you the advantage that your files are always backed up on your local machine. The changes are instant and even better than ssh you can just hit ctrl-save instead of saving as root since ctrl-save is set in the add-on to automatically upload your file to the server. Hope it helps someone and maybe Microsoft will give a damn about us someday.

trilambda122 commented 2 years ago

I can report, the same issue on t2.mrico. VScode will consistently crash the instance, I don't believe it to be an issue with memory. It constantly has a CPU spike to 90-100% and requires a hard reboot to restore. I switched back to using emacs and never an issue. This is a real bummer.

TheMaverickProgrammer commented 2 years ago

This happens on all of my Digital Ocean VM's too. Absolutely crashes the server in lightning fast time.

sholtomaud commented 2 years ago

Interestingly there is something going on with the VSCode "server" which seems to run on a remote machine.

I run a local network at home, and x2 pcs. VSCode runs fine on each PC separately, however when I use the remote connection the remote PC becomes unresponsive with maximum memory usage.

Mr Sholto Maud m: +61 421 192 453

On Thu, 31 Mar 2022 at 07:10, TheMaverickProgrammer < @.***> wrote:

This happens on all of my Digital Ocean VM's too.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/2692#issuecomment-1083576800, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADBIQQ4YE7SLBSZRJAAUT3VCSYJ5ANCNFSM4L4LP2OA . You are receiving this because you commented.Message ID: @.***>

abuzarshabab commented 2 years ago

All the above scenarios I'm facing 😐

clshortfuse commented 2 years ago

Maybe with the new VS Code changes with heap profiling built-in, we can find the leak in the typescript extension (or wherever it is) and plug it.

https://code.visualstudio.com/updates/v1_66#_javascript-debugging

mrmckeb commented 2 years ago

I've been trying to use this on my Raspberry Pi and it's unusable within minutes. I assume it's the same issue, as I'd definitely done this previously and not had any problems.

ghost commented 2 years ago

Same issue here using AWS MEAN stack.

sholtomaud commented 2 years ago

Yep - still going. The ability to bring down a server with this code is high hence probably a critical fail.

All3xJ commented 2 years ago

https://medium.com/good-robot/use-visual-studio-code-remote-ssh-sftp-without-crashing-your-server-a1dc2ef0936d

microsoft / vscode-remote-release

Remote VSCode over SSH crashes EC2 instance #2692