search

microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.66k stars 286 forks source link

code-server - multiple core dumps #735

Closed ayspunde closed 5 years ago

ayspunde commented 5 years ago

Steps to Reproduce:

  1. Install VS Code on MacOS (Mojave 10.14.5) (+ VS Code-Insiders)
    • no issues
  2. Install VS Code's Remote Development extension (0.15.0)
    • no issues
  3. ssh from macOS to the Linux login node (Cori cluster at NERSC)
    • MFA authentication
    • bash on login node functioning well (bash commands + code editing works just fine)
    • but every ~10 minutes creates new core.nnnnn dump. `ayspunde@cori02:~$ gdb -c core* ... Core was generated by "/global/u1/a/ayspunde/.vscode-server-insiders/bin/7f53e50cbf316dff94fbbe5406d8f" Program terminated with signal SIGABRT, Aborted.

      0 0x00002aaaabbcbf67 in ?? ()

      [Current thread is 1 (LWP 31433)] (gdb)`

Does this issue occur when all extensions are disabled?: Yes/No

roblourens commented 5 years ago

That's strange, is that all it says? /global/u1/a/ayspunde/.vscode-server-insiders/bin/7f53e50cbf316dff94fbbe5406d8f is just a folder, I would expect a pointer to some program.

Are you using any extensions on the remote?

ayspunde commented 5 years ago

Hi

according to 'show running extensions' i have:

Remote-SSH
Remote SSH: editing configuration files
Remote SSH: explorer

that is my 'vscode-server-insiders' on the remote node:

ayspunde@dint10:~$ cd .vscode-server-insiders
ayspunde@dint10:.vscode-server-insiders$ ll
total 10K
drwxrwxrwx 11 ayspunde genome 4.0K Jun 19 13:27 bin
drwxrwxrwx  5 ayspunde genome  512 Jun  5 15:30 data
drwxrwxrwx  2 ayspunde genome  512 Jun  5 15:30 extensions
ayspunde@dint10:.vscode-server-insiders$ cd extensions/
ayspunde@dint10:extensions$ ll
total 0
ayspunde@dint10:extensions$ cd ../bin
ayspunde@dint10:bin$ ll
total 72K
drwxrwxrwx 6 ayspunde genome   4.0K Jun 17 15:38
11b88a3323878a98b418e4efca77e7afabd50d01
drwxrwxrwx 6 ayspunde genome   4.0K Jun 14 19:16
15b7c86d86319b187a5ef255c170184a8519c5af
drwxrwxrwx 6 ayspunde genome   4.0K Jun 10 15:32
5c65d9bfa4c56538150d7f3066318e0db2c6151f
drwxrwxr-x 6 ayspunde ayspunde 4.0K Jun 19 13:27
6520c22b83c0b6f168ba7de3309ffab42281e905
drwxrwxr-x 6 ayspunde ayspunde 4.0K Jun 18 15:51
68a7e5bc437b38d0281df0756997a25da2a2900c
drwxrwxrwx 6 ayspunde genome   4.0K Jun 11 13:38
7bc29bf67c17608b11e23a8497c8f1de5a0922cf
drwxrwxrwx 6 ayspunde genome   4.0K Jun 18 15:49
7f53e50cbf316dff94fbbe5406d8f352dac99a6b
drwxrwxrwx 6 ayspunde genome   4.0K Jun 11 10:05
c089daa858f34631f5f827ff8a0313bf1e2ded6d
drwxrwxrwx 6 ayspunde genome   4.0K Jun 14 18:18
d04b5d2eb16c156e24dbf44010c315d968694bd6

stepping into latest (by time stamp) dir:
cd 6520c22b83c0b6f168ba7de3309ffab42281e905
ayspunde@dint10:6520c22b83c0b6f168ba7de3309ffab42281e905$ ll
total 38M
-rw-r--r--   1 ayspunde ayspunde  14K Jun 18 22:03 LICENSE
drwxr-xr-x   2 ayspunde ayspunde  512 Jun 18 22:21 bin
drwxr-xr-x  27 ayspunde ayspunde 4.0K Jun 18 22:23 extensions
-rwxr-xr-x   1 ayspunde ayspunde  38M Jun 18 22:22 node
drwxr-xr-x 158 ayspunde ayspunde 4.0K Jun 18 22:22 node_modules
drwxr-xr-x   3 ayspunde ayspunde  512 Jun 18 22:21 out
-rw-r--r--   1 ayspunde ayspunde 1.1K Jun 18 22:09 package.json
-rw-r--r--   1 ayspunde ayspunde  18K Jun 18 22:03 product.json
-rwxr-xr-x   1 ayspunde ayspunde  222 Jun 18 22:03 server.sh
ayspunde@dint10:6520c22b83c0b6f168ba7de3309ffab42281e905$ cd extensions/
ayspunde@dint10:extensions$ ll
total 25K
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:21 configuration-editing
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:21 css-language-features
drwxr-xr-x 3 ayspunde ayspunde 512 Jun 18 22:21 debug-auto-launch
drwxr-xr-x 3 ayspunde ayspunde 512 Jun 18 22:21 debug-server-ready
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:21 emmet
drwxr-xr-x 3 ayspunde ayspunde 512 Jun 18 22:22 extension-editing
drwxr-xr-x 6 ayspunde ayspunde 512 Jun 18 22:22 git
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:22 grunt
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:22 gulp
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:22 html-language-features
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:22 jake
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:22 javascript
drwxr-xr-x 3 ayspunde ayspunde 512 Jun 18 22:22 json
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:22 json-language-features
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:22 markdown-language-features
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:23 merge-conflict
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:23 ms-vscode.node-debug
drwxr-xr-x 7 ayspunde ayspunde 512 Jun 18 22:23 ms-vscode.node-debug2
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:23 ms-vscode.references-view
drwxr-xr-x 3 ayspunde ayspunde 512 Jun 18 22:23 node_modules
drwxr-xr-x 5 ayspunde ayspunde 512 Jun 18 22:23 npm
drwxr-xr-x 4 ayspunde ayspunde 512 Jun 18 22:23 php-language-features

Please let me know what else will be helpful Thank you A

On Wed, Jun 19, 2019 at 4:17 PM Rob Lourens notifications@github.com wrote:

That's strange, is that all it says? /global/u1/a/ayspunde/.vscode-server-insiders/bin/7f53e50cbf316dff94fbbe5406d8f is just a folder, I would expect a pointer to some program.

Are you using any extensions on the remote?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/735?email_source=notifications&email_token=ABJ5ZDGMBDULGQVZFRBDXPLP3K5ANA5CNFSM4HZKQU72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYDRMAA#issuecomment-503780864, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJ5ZDHCZCCSGF4IGHKIJL3P3K5ANANCNFSM4HZKQU7Q .

roblourens commented 5 years ago

Thanks, you have no other remote extensions.

Can you share the core dump? Or can you get any other info from it? For example, the output of file <path> might be useful. http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html

ayspunde commented 5 years ago

$>file core.27556 core.27556: ELF 64-bit LSB core file x86-64, version 1 (SYSV), too many program header sections (2971)

I posted core file at https://portal.nersc.gov/dna/microbial/assembly/ayspunde/vscode/ Pls let me know if it's not working, I'll set a Globust FTP node (or a shared google drive folder)

On Wed, Jun 19, 2019 at 4:42 PM Rob Lourens notifications@github.com wrote:

Thanks, you have no other remote extensions.

Can you share the core dump? Or can you get any other info from it? For example, the output of file might be useful. http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/735?email_source=notifications&email_token=ABJ5ZDETUQZ7LSDUK7AOIOLP3K743A5CNFSM4HZKQU72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYDSQSQ#issuecomment-503785546, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJ5ZDB7BG6HLRE263VBHWTP3K743ANCNFSM4HZKQU7Q .

ayspunde commented 5 years ago

Looking at the log file, JavaScript memory management could be an issue (full log file attached)

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x33412cddc01d]

Security context: 0x372844e1e681 1: / anonymous / [0xaa8f84234a9] [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node_modules/readdirp/node_modules/readable-stream/lib/_stream_readable.js:~733] [pc=0x33412d05ea07](this=0x23d11bdc57c1 ,ev=0x109eb60202a9 <String[6]: resume>,fn=0x23d11bdc5bc1 <JSFunction ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x8d20d0 node::Abort() [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 2: 0x8d211c [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 3: 0xb02b6e v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 4: 0xb02da4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 5: 0xef02e2 [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 6: 0xef03e8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 7: 0xefc512 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 8: 0xefce44 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 9: 0xefe359 v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 10: 0xeffb1e v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 11: 0xec8fd4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 12: 0x11685bb v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object*, v8::internal::Isolate) [/global/u1/a/ayspunde/.vscode-server-insiders/bin/6a7020dd4a11e0f33b157b3df4baeec90d85701b/node] 13: 0x33412cddc01d IPC "File Watcher (chokidar)" crashed with exit code null and signal SIGABRT

.6a7020dd4a11e0f33b157b3df4baeec90d85701b.log

roblourens commented 5 years ago

Thanks. Is it possible that your machine is low on memory?

ayspunde commented 5 years ago

Not at the moment: free total used free shared buffers cached Mem: 527508176 227509848 299998328 998244 1240252 182669224 -/+ buffers/cache: 43600372 483907804 Swap: 33554428 10367748 23186680 ayspunde@cori05:seqqc-13763$ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 8 0 10367064 295890912 1240272 186131372 1 2 1 8 0 1 9 3 87 1 0

but I don't know about times when coredumps happened, it's a login node of the cluster, things can happen.

I have Telemetry/Crash Reporter enabled, is it something that can be checked from there? Alternatively, I can start a cron job to check for dump, and run free when detected.

roblourens commented 5 years ago

Are you opening a really large folder? I see https://github.com/microsoft/vscode/issues/75004 although it doesn't have any details. Maybe you can check how much memory the file watcher processes uses as it is running, because I don't know whether this is due to the host getting low on memory, or the process hitting a heap size limit.

ayspunde commented 5 years ago
  1. Large folder: Depends on 'large'; the number of files is in tens, but some are in the 100Gb range (binary). du -sh: 905G (scattered thru few levels)
  2. file watcher process: top on the remote: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    64573 ayspunde 20 0 35140 5896 4024 R 1.316 0.001 0:00.56 top
    9942 ayspunde 20 0 972308 93828 18276 S 0.658 0.018 0:23.97 node
    4564 ayspunde 20 0 164304 5264 3212 S 0.000 0.001 0:03.38 sshd
    5272 ayspunde 20 0 11728 2632 2208 S 0.000 0.000 0:00.00 sh
    5280 ayspunde 20 0 834720 41836 17956 S 0.000 0.008 0:00.94 node
    9447 ayspunde 20 0 11728 2632 2208 S 0.000 0.000 0:00.00 sh
    9456 ayspunde 20 0 848992 42568 18148 S 0.000 0.008 0:05.31 node
    9961 ayspunde 20 0 808924 28840 15344 S 0.000 0.005 0:00.85 node
    9974 ayspunde 20 0 34600 6456 3320 S 0.000 0.001 0:00.09 bash
    9992 ayspunde 20 0 552120 35308 15688 S 0.000 0.007 0:21.82 node
    17559 ayspunde 20 0 11728 484 484 S 0.000 0.000 0:00.00 sh
    17567 ayspunde 20 0 861880 2416 424 S 0.000 0.000 0:17.74 node
    33002 ayspunde 20 0 55204 10036 6016 S 0.000 0.002 0:00.04 mom_local.py
    33004 ayspunde 20 0 55768 6156 5192 S 0.000 0.001 0:00.09 ssh
    33209 ayspunde 20 0 33084 6176 2948 S 0.000 0.001 0:00.08 bash
    33779 ayspunde 20 0 12284 3536 2420 S 0.000 0.001 0:00.05 bash
    34387 ayspunde 20 0 4232 708 616 S 0.000 0.000 0:00.00 sleep
    39292 ayspunde 20 0 12128 3352 2664 S 0.000 0.001 0:00.10 sh
    55716 ayspunde 20 0 34596 6416 3320 S 0.000 0.001 0:00.06 bash
ayspunde commented 5 years ago

Please let me know if additional info will be helpful; issue still persists: ayspunde@cori05:seqqc-13976$ ll ~/core* -rw------- 1 ayspunde ayspunde 2.0G Jul 10 11:57 /global/homes/a/ayspunde/core.18718 -rw------- 1 ayspunde ayspunde 2.0G Jul 10 12:18 /global/homes/a/ayspunde/core.23759 -rw------- 1 ayspunde ayspunde 2.0G Jul 10 11:35 /global/homes/a/ayspunde/core.25895 -rw------- 1 ayspunde ayspunde 2.0G Jul 10 12:08 /global/homes/a/ayspunde/core.45053 -rw------- 1 ayspunde ayspunde 2.0G Jul 10 12:48 /global/homes/a/ayspunde/core.45857 -rw------- 1 ayspunde ayspunde 2.0G Jul 10 11:47 /global/homes/a/ayspunde/core.53475 -rw------- 1 ayspunde ayspunde 2.0G Jul 10 12:29 /global/homes/a/ayspunde/core.61536

roblourens commented 5 years ago

Most likely same as https://github.com/microsoft/vscode-remote-release/issues/1257