search

coder / code-server

VS Code in the browser
https://coder.com
MIT License
68.8k stars 5.65k forks source link

Unexpected random crashes #1009

Closed EnKrypt closed 4 years ago

EnKrypt commented 5 years ago

Description

code-server

I run code-server with the command above. Everything is fine for a while. I open my browser and use it fine, but then it randomly crashes. I say random because I cannot find any pattern between the crash and possible stimuli. It can crash when I'm saving a file, or opening a project folder, or changing a setting, etc.

When it crashes, it brings me back to the terminal.

The logs don't seem to have anything useful. One time, I ran code-server as above, and had it immediately crash when I opened the URL with param ?folder=/home/arvind/Blueshift

[2019-09-20 18:41:46.724] [remoteagent] [info] update#setState idle
[2019-09-20 18:41:46.724] [remoteagent] [trace] telemetry/optInStatus {"properties":{"common.machineId":"4efb74722d175601cc7>
[2019-09-20 18:41:46.724] [remoteagent] [trace] telemetry/optInStatus {"optIn":true}
[2019-09-20 18:41:46.724] [remoteagent] [trace] telemetry/machineIdFallback {"properties":{"common.machineId":"4efb74722d175>
[2019-09-20 18:41:46.724] [remoteagent] [trace] telemetry/machineIdFallback {"usingFallbackGuid":false}
[2019-09-20 18:41:49.856] [remoteagent] [info] request.url "/?folder=/home/arvind/Blueshift"
[2019-09-20 18:41:51.623] [remoteagent] [trace] Started scanning system extensions

I'm not sure how this can even be reproduced. But if anyone has any ideas to explore this, I'm all ears.

EnKrypt commented 5 years ago

Skipped adding the [v2] flair to the title, and I can't seem to add the v2 label to the issue myself.

sr229 commented 5 years ago

if all else fails, try running it in with a debugger attached and see if what happens.

EnKrypt commented 5 years ago

Attach a debugger to the binary? What's the best way to do that?

sr229 commented 5 years ago

You can use gdb for this purpose.

nhooyr commented 5 years ago

This is very bizarre that there are no logs. @code-asher Any ideas what could be causing this?

sr229 commented 5 years ago

Apparently this only happens in Arch as far as testing is concerned. Any sane distro that isn't Arch should work.

nhooyr commented 5 years ago

@sr229 Almost all of us use arch at the coder office :P

None of us have ran into this.

sr229 commented 5 years ago

I think it depends on what library version they're using @nhooyr

@EnKrypt can you give additional info about your system?

nhooyr commented 5 years ago

Not sure what you mean, what library version?

EnKrypt commented 5 years ago

If you tell me what libs code-server uses, I can post what versions of those libs I have.

I'm running a headless Arch server.

I considered that this could be an out of memory crash, given that the server has 4 gigs of RAM, but it also has 4 gigs of Swap, so it should ideally go to swap instead of getting killed by the OS.

NN708 commented 5 years ago

I have the same problem with version 2.1523-vsc1.38.1. My server is running Ubuntu 18.04.3 LTS, and have 2GB of RAM.

EnKrypt commented 5 years ago

I haven't had a chance to play around with this yet, but my current theory is that it could simply be related to memory management.

@NN708 Can you monitor RAM usage when the crash happens? If the crash happens at high memory usage, and the RAM usage jumps down post-crash, then this will be definitive.

EnKrypt commented 5 years ago

On a side note, do you have swap enabled?

NN708 commented 5 years ago

Hi, @EnKrypt , I monitored the RAM usage, and found that it was only using about 29% of RAM when it crashed. I think perhaps the problem is not related to RAM usage.

wpitallo commented 5 years ago

I am also getting this on Ubuntu. version 2.1665-vsc1.39.2. It seems to crash every time I run: npm install. Are there any crash logs that we can view? Also seems to just crash at random. I am running it on an EC2 Instance and I tried upgrading to a larger instance but it still crashes all the time. Have not been able to run for longer than a few min without a crash.

code-asher commented 5 years ago

I've also seen crashes when I use code-server on Chromium for a prolonged amount of time and then load it up in Firefox. As soon as Firefox touches code-server it crashes silently. Not every time though. I have no idea what the cause is.

Other crash reports: https://github.com/cdr/code-server/issues/559 https://github.com/cdr/code-server/issues/820 https://github.com/cdr/code-server/issues/919 (this looks very similar to what I'm seeing except I'm not restarting code-server)

wpitallo commented 5 years ago

Hi @code-asher what system setup are you using that you don't get crashes that frequently? Is there a stable version? I want to try a few workarounds if possible? Right now its really un-usable for me, but I am not sure why this is not the case for more people then?

code-asher commented 5 years ago

We have some people running it on Ubuntu and some on Arch and other than the issue I mentioned when switching browsers it appears stable (running 2.1665).

I played around with npm install to see if I could replicate your crash but I haven't had any luck (or the opposite, rather) yet. Do you experience crashes inside the Docker image or only when running the binary directly?

You can try running code-server with --verbose and see if there is any useful information in ~/.local/share/code-server/logs but so far the crashes have all been silent. I wonder if there are errors being swallowed by something...I will probably attaching a debugger soon if I can't figure out some replication steps.

wpitallo commented 5 years ago

Hi, thanks @code-asher, mmm I am getting it on 2.1665-vsc1.39, I am only running the binary directly. Will give it a try with the --verbose flag and see if I see anything. Thx again!

code-asher commented 5 years ago

Small update: from my testing V8 has a default limit of around 1.4GB for memory usage (on my machine at least) so I upped that to 2048MB and enabled the max-memory flag that can be used to set your own limit (you can also manually set the max_old_space_size flag via the NODE_OPTIONS environment variable---that's possible even with current releases).

I don't know if that'll fix anyone's crashes but I've seen at least one recently caused by an OOM so hopefully this helps.

It remains to be seen whether the OOM was a result of normal memory use or a leak.

wpitallo commented 5 years ago

It is crashing for me on firefox as well. Not sure if they have similar constraints or not?

hgw77 commented 5 years ago

I run into the problem that sometimes code server stops responding. Mostly when I load initally in the browser. When I check the server I can see that the docker container was restarted and I can find in in the containber logs the following message

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

my server has 8GB Memory and only half of that was used when the code-server container restarted

I will use the max-memory flag and will see how it works :-)

hgw77 commented 5 years ago

from my side, I used --max-memory 4096 but on this morning first try and the container restarted. This time I did not see anything usefull in the log. Will try to start code-server with --verbose option next.

Short question what is the difference between verbose and the log option with debug level? should I use it together?

code-asher commented 5 years ago

--verbose is equivalent to --log trace (and trace is more verbose than debug). The verbose flag takes priority over the log flag. So you only need one or the other.

hgw77 commented 5 years ago

Today my code-server crashes multiple times but sadly there was no usefull information in the logs. I was using --verbose and --log debug

code-asher commented 5 years ago

I'm still not sure as to the origin of the crashes but we're making some changes that will recover from the crashes so that you can at least refresh the page and have VS Code start up again.

EnKrypt commented 4 years ago

Sounds good. Seeing a "Connection Lost" popup might be a bit jarring, but as a temp solution I think that's better than what exists now. Might have to put some thought into crash recovery so that in case of an insta-crash, it doesn't devolve into an infinite crash cycle.

That said, I'd still like to explore the real cause behind these crashes because upstream updates really won't make a difference. My two current theories are that this is either related to memory management, or there is something off with how we are creating binaries since this issue exists only on Linux as far as I can see, in which case this issue might belong with the internal build tool within the org (nbin).

Edit: Wrongly assumed we're depending on pkg for building.

hgw77 commented 4 years ago

for me, code server crashes when I try to load it inititally or when I reload the browser.

davidchen1 commented 4 years ago

I am also experience random crashes. Sometimes it happens at start up, sometime a few hours in. There is no error message even in verbose mode.

After code-server crashes, if I try to start code-server on the same container, it would crash again. I had to restart the container to be able to successfully run code-server again.

OS: Debian version 10.

hgw77 commented 4 years ago

after switching from Firefox Developer Edition to Chromium it is much better. I do not see any crashes. It is possible that the browser is the root cause?

sr229 commented 4 years ago

I'm no longer registering random crashes so I'll close this off for now.

nhooyr commented 4 years ago

This still occurs to both me and Asher.

kylecarbs commented 4 years ago

Nothing has been reported on this in a while, and v3 introduces new architecture to combat a lot of these unexpected nuances.

Closing for now! 🎊

EnKrypt commented 4 years ago

I see that v3 releases are now out. How stable can we expect them to be? Would it be advisable to fall back to older versions for production?

nhooyr commented 4 years ago

v3 is the stable release now so yes, should be good in production.

HALOCORE commented 1 year ago

Still experiencing random crashes in v4. --verbose is not printing out useful things to locate the issue.

kdelong commented 1 year ago

It appears to be an OOM issue for me.

Available Mem showed 5 in top just before the crash. This is on a Nanode (Linode) 1GB instance having (other than code-server) only Node to handle compilation / requests for a React/Next.js project.

$ uname -a
Linux 5.15.0-71-generic #78-Ubuntu SMP Tue Apr 18 09:00:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

To make it crash, I tried to simulate typical single-user usage by saving TypeScript files and refreshing the index page of a project in close succession, such that the request would come in while compilation was still running, etc. It doesn't take much of that for it to crash with my setup.

I see that code-server recommends 1GB RAM and 2 CPUs. I was seeing if I could stretch things with a Nanode, but it seems to be insufficient. On the other hand, it is the one recommended by coder here, so I figure something may be wrong.

$ free -b
               total        used        free      shared  buff/cache   available
Mem:      1016528896   163680256   569794560      966656   283054080   713232384
Swap:      536866816           0   536866816