Closed cjdelisle closed 9 years ago
Often enough nodejs -e 'console.log("hi");'
is enough to crash it
Linux ci20-3006 3.0.8-12439-gf697891 #98 SMP PREEMPT Thu Sep 11 15:52:29 BST 2014 mips GNU/Linu
root@ci20-3110:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.0 (jessie) Release: 8.0 Codename: jessie root@ci20-3110:~# while (true) ; do nodejs -e 'console.log("hi");' ; done;
path.js:326 resolvedPath = normalizeArray(resolvedPath.split('/').filter(function(p) { ^ <error: TypeError: Cannot call method '499138649' of undefined> at Object.exports.resolve (path.js:326:48) at Function.Module._nodeModulePaths (module.js:209:15) at evalScript (node.js:523:27) at startup (node.js:80:7) at node.js:906:3
path.js:326 resolvedPath = normalizeArray(resolvedPath.split('/').filter(function(p) { ^
The plot thickens :+1:
root@ci20:~# uname -a Linux ci20 4.0.0-rc1+ #15 SMP PREEMPT Mon Feb 23 17:44:38 GMT 2015 mips GNU/Linux root@ci20:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.0 (jessie) Release: 8.0 Codename: jessie root@ci20:~# root@ci20:~# while (true) ; do nodejs -e 'console.log("hi");' ; done; hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi ^Chi ^C^C^C^C root@ci20:~# ^C root@ci20:~# ^C
root@ci20:~# lsb_release -a uname -aNo LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.0 (jessie) Release: 8.0 Codename: jessie root@ci20:~# uname -a Linux ci20 3.16.0-00055-g30327b7-dirty #45 SMP Tue Feb 24 10:18:13 GMT 2015 mips GNU/Linux root@ci20:~# while (true) ; do nodejs -e 'console.log("hi");' ; done; hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi
This could possibly be linked
https://groups.google.com/d/msg/mips-creator-ci20/XbUsPwqYNq0/IK_SxdQUd68J
(gdb) run -e 'console.log("hi")' Starting program: /usr/bin/nodejs -e 'console.log("hi")' [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1". [New Thread 0x2bd04490 (LWP 1966)] hi [Thread 0x2bd04490 (LWP 1966) exited] Inferior 1 (process 1965) exited normally run -e 'console.log("hi")' Starting program: /usr/bin/nodejs -e 'console.log("hi")' [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1". [New Thread 0x2bd04490 (LWP 1968)] hi [Thread 0x2bd04490 (LWP 1968) exited] Inferior 1 (process 1967) exited normally run -e 'console.log("hi")' Starting program: /usr/bin/nodejs -e 'console.log("hi")' [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1". [New Thread 0x2bd04490 (LWP 1970)] warning: GDB can't find the start of the function at 0x55b1c830.
GDB is unable to find the start of the function at 0x55b1c830
and thus can't determine the size of that function's stack frame. This means that GDB may be unable to access that stack frame, or the frames below it. This problem is most likely caused by an invalid program counter or stack pointer. However, if you think GDB should simply search farther back from 0x55b1c830 for code which looks like the beginning of a function, you can increase the range of the search using the `set heuristic-fence-post' command.
Program received signal SIGSEGV, Segmentation fault. warning: GDB can't find the start of the function at 0x55b1c830. 0x55b1c830 in ?? () (gdb) info shared From To Syms Read Shared Object Library 0x2aaa89c0 0x2aac51f0 Yes /lib/ld.so.1 0x2aaef720 0x2ab02720 Yes () /lib/mipsel-linux-gnu/libz.so.1 0x2ab19530 0x2ab26dd0 Yes () /usr/lib/mipsel-linux-gnu/libcares.so.2 0x2ab60430 0x2aea91a0 Yes /usr/lib/libv8.so.3.14.5 0x2af4fad0 0x2af8a720 Yes () /usr/lib/mipsel-linux-gnu/libssl.so.1.0.0 0x2afddef0 0x2b0fdc40 Yes () /usr/lib/mipsel-linux-gnu/libcrypto.so.1.0.0 0x2b155bc0 0x2b156c80 Yes /lib/mipsel-linux-gnu/libdl.so.2 0x2b169980 0x2b16e150 Yes /lib/mipsel-linux-gnu/librt.so.1 0x2b1c7fd0 0x2b2524d4 Yes () /usr/lib/mipsel-linux-gnu/libstdc++.so.6 0x2b295d50 0x2b2dabf0 Yes /lib/mipsel-linux-gnu/libm.so.6 0x2b32d9e0 0x2b3477c0 Yes () /lib/mipsel-linux-gnu/libgcc_s.so.1 0x2b35e6d0 0x2b36fd60 Yes /lib/mipsel-linux-gnu/libpthread.so.0 0x2b39d7e0 0x2b4c3440 Yes /lib/mipsel-linux-gnu/libc.so.6 (*): Shared library is missing debugging information. (gdb)
with debug symbols
root@ci20-3006:~# gdb nodejs -ex 'set confirm no' -ex 'r -e "console.log(\"hi\")"' -ex 'backtrace' -ex 'info registers' -ex 'quit' | tee node-debug
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "mipsel-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from nodejs...Reading symbols from /usr/lib/debug//usr/bin/nodejs...done.
done.
Starting program: /usr/bin/nodejs -e "console.log(\"hi\")"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1".
[New Thread 0x2bcf4490 (LWP 733)]
Program received signal SIGSEGV, Segmentation fault.
v8::internal::ExternalTwoByteString::ExternalTwoByteStringReadBlockIntoBuffer (this=<optimized out>, rbb=0x7fff61a8, offset_ptr=0x7fff61f0,
max_chars=646463582) at ../src/objects.cc:6482
6482 ../src/objects.cc: No such file or directory.
#0 v8::internal::ExternalTwoByteString::ExternalTwoByteStringReadBlockIntoBuffer (this=<optimized out>, rbb=0x7fff61a8,
offset_ptr=0x7fff61f0, max_chars=646463582) at ../src/objects.cc:6482
#1 0x2acf22e0 in v8::internal::String::ReadBlock (input=<optimized out>, rbb=0x7fff61a8, offset_ptr=0x7fff61f0, max_chars=646463582)
at ../src/objects.cc:6576
#2 0x2acf2454 in v8::internal::String::ReadBlock (input=<optimized out>, util_buffer=<optimized out>, capacity=<optimized out>,
remaining=0x7fff61e4, offset_ptr=0x7fff61f0) at ../src/objects.cc:6748
#3 0x2acf60dc in Reset (input=0x24c08091, position=0, this=0x7fff61e0) at ../src/unicode-inl.h:229
#4 Reset (input=0x24c08091, this=0x7fff61e0) at ../src/unicode-inl.h:234
#5 InputBuffer (input=0x24c08091, this=0x7fff61e0) at ../src/unicode.h:230
#6 StringInputBuffer (backing=0x24c08091, this=0x7fff61e0) at ../src/objects.h:7909
#7 v8::internal::String::ComputeAndSetHash (this=0x24c08091) at ../src/objects.cc:7230
#8 0x2aba2304 in Hash (this=0x24c08091) at ../src/objects-inl.h:4890
#9 LinearSearch<(v8::internal::SearchMode)1, v8::internal::DescriptorArray> (len=<optimized out>, valid_entries=13, name=0x4d10bf05,
array=<optimized out>) at ../src/objects-inl.h:2007
#10 v8::internal::Search<(v8::internal::SearchMode)1, v8::internal::DescriptorArray> (array=<optimized out>, name=0x4d10bf05, valid_entries=13)
at ../src/objects-inl.h:2032
#11 0x2ad13798 in Search (valid_descriptors=<optimized out>, name=0x4d10bf05, this=0x30714389) at ../src/objects-inl.h:2041
#12 SearchWithCache (map=0x5440937d, name=0x4d10bf05, this=0x30714389) at ../src/objects-inl.h:2053
#13 LookupDescriptor (result=0x7fff6730, name=0x4d10bf05, holder=0x3074c075, this=0x5440937d) at ../src/objects-inl.h:2065
#14 v8::internal::JSObject::LocalLookupRealNamedProperty (this=<optimized out>, name=0x4d10bf05, result=0x7fff6730) at ../src/objects.cc:2386
#15 0x2ad14574 in v8::internal::JSReceiver::Lookup (this=<optimized out>, name=0x4d10bf05, result=0x7fff6730) at ../src/objects.cc:4315
#16 0x2ac83794 in v8::internal::LookupForRead (object=..., name=..., lookup=0x7fff6730) at ../src/ic.cc:438
#17 0x2ac88000 in v8::internal::LoadIC::Load (this=0x7fff67a0, state=v8::internal::UNINITIALIZED, object=..., name=...) at ../src/ic.cc:904
#18 0x2ac88b18 in v8::internal::LoadIC_Miss (args=..., isolate=0x55f038) at ../src/ic.cc:2088
warning: GDB can't find the start of the function at 0x20c0a572.
GDB is unable to find the start of the function at 0x20c0a572
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
This problem is most likely caused by an invalid program counter or
stack pointer.
However, if you think GDB should simply search farther back
from 0x20c0a572 for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
#19 0x20c0a574 in ?? ()
zero at v0 v1 a0 a1 a2 a3
R0 00000000 7fff67ec 00000002 00000000 0000000a 7fff61a8 7fff61f0 2688405e
t0 t1 t2 t3 t4 t5 t6 t7
R8 24c08091 00000002 00000082 00000003 00000087 00000091 ffffffe0 7fff67fc
s0 s1 s2 s3 s4 s5 s6 s7
R16 2688405e 7fff61a8 24c08091 7fff61f0 25c00498 4d10bf05 fff00000 00000082
t8 t9 k0 k1 gp sp s8 ra
R24 00560734 2acf1774 2af36750 00000000 2af36750 7fff6128 7fff67fc 2acf22e0
status lo hi badvaddr cause pc
00001c13 00008000 00000000 0000000a 00800010 2acf177c
fcsr fir restart
00000064 00330100 00000000
root@ci20-3006:~#
root@ci20-3006:~# gdb nodejs -ex 'set confirm no' -ex 'r -e "console.log(\"hi\")"' -ex 'backtrace' -ex 'info registers' -ex 'quit'
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "mipsel-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from nodejs...Reading symbols from /usr/lib/debug//usr/bin/nodejs...done.
done.
Starting program: /usr/bin/nodejs -e "console.log(\"hi\")"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/mipsel-linux-gnu/libthread_db.so.1".
[New Thread 0x2bcf4490 (LWP 719)]
warning: GDB can't find the start of the function at 0x38269204.
GDB is unable to find the start of the function at 0x38269204
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
This problem is most likely caused by an invalid program counter or
stack pointer.
However, if you think GDB should simply search farther back
from 0x38269204 for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
Program received signal SIGILL, Illegal instruction.
warning: GDB can't find the start of the function at 0x38269204.
0x38269204 in ?? ()
#0 0x38269204 in ?? ()
zero at v0 v1 a0 a1 a2 a3
R0 00000000 0000029c 0000029c 0057f0c8 0000029c 4d74c979 00000000 4d74c1c4
t0 t1 t2 t3 t4 t5 t6 t7
R8 00005512 37b080c1 80028350 0000053e 4d714371 7fff67d8 2ac8ad34 7fff67d8
s0 s1 s2 s3 s4 s5 s6 s7
R16 ffffffff 7fff67e4 2ac8b920 37b08091 37b08091 37b08091 0055f044 4d714371
t8 t9 k0 k1 gp sp s8 ra
R24 00000008 38269200 2af36750 00000000 2af36750 7fff67e8 7fff6804 4cc1862c
status lo hi badvaddr cause pc
00001f13 00008000 00000000 2ac82d48 00800028 38269204
---Type <return> to continue, or q <return> to quit---
fcsr fir restart
00000064 00330000 00000000
root@ci20-3006:~#
If I had a guess, I'd say some system call is returning garbage
Sample of running with strace:
while :; do strace -f nodejs -e 'console.log("hi")' 2>&1 | tail; done | tee debug.dump2
(successful entries removed)
[pid 4974] syscall(0x1033, 0x5541efa8, 0x10001, 0x1, 0x2b924750, 0x5541ee41, 0x5541ee80) = 0
[pid 4974] syscall(0x1033, 0x5541efc0, 0x10001, 0x1, 0x2b924750, 0x5541ee41, 0x5541ee80) = 0
[pid 4974] syscall(0x1033, 0x5541ee80, 0x10001, 0x1, 0x2b924750, 0x5541ee41, 0x5541ee80) = 0
[pid 4974] futex(0x2c70046c, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 4975] <... futex resumed> ) = 0
[pid 4975] nanosleep({0, 900000}, <unfinished ...>
[pid 4974] syscall(0x1033, 0x5541eebc, 0x10001, 0x1, 0x2b924750, 0x1, 0x7fba9f1c) = 0
[pid 4974] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
[pid 4975] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
[pid 5054] syscall(0x1033, 0x2d71efa8, 0x10001, 0x1, 0x2b8ff750, 0x2d71ee41, 0x2d71ee80) = 0
[pid 5054] syscall(0x1033, 0x2d71efc0, 0x10001, 0x1, 0x2b8ff750, 0x2d71ee41, 0x2d71ee80) = 0
[pid 5054] syscall(0x1033, 0x2d71ee80, 0x10001, 0x1, 0x2b8ff750, 0x2d71ee41, 0x2d71ee80) = 0
[pid 5054] futex(0x2c70046c, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 5055] <... futex resumed> ) = 0
[pid 5055] nanosleep({0, 900000}, <unfinished ...>
[pid 5054] syscall(0x1033, 0x2d71eebc, 0x10001, 0x1, 0x2b8ff750, 0x1, 0x7f8b8e0c) = 0
[pid 5054] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
[pid 5055] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
[pid 5110] futex(0x2c70046c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 5111] <... futex resumed> ) = 0
[pid 5111] nanosleep({0, 900000}, <unfinished ...>
[pid 5110] <... futex resumed> ) = 1
[pid 5110] syscall(0x1033, 0x5ab1eebc, 0x10001, 0x1, 0x2b8ef750, 0x1, 0x7fbdf47c) = 0
[pid 5111] <... nanosleep resumed> NULL) = 0
[pid 5111] nanosleep({0, 900000}, <unfinished ...>
[pid 5110] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
[pid 5111] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
[pid 5182] syscall(0x1033, 0x4551efc0, 0x10001, 0x1, 0x2badd750, 0x4551ee41, 0x4551ee80) = 0
[pid 5182] syscall(0x1033, 0x4551ee80, 0x10001, 0x1, 0x2badd750, 0x4551ee41, 0x4551ee80) = 0
[pid 5182] futex(0x2c90046c, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 5183] <... futex resumed> ) = 0
[pid 5182] syscall(0x1033, 0x4551eebc, 0x10001, 0x1, 0x2badd750, 0x1, 0x7f9ad61c <unfinished ...>
[pid 5183] nanosleep({0, 900000}, <unfinished ...>
[pid 5182] <... syscall resumed> ) = 0
[pid 5182] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
[pid 5183] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
[pid 5198] syscall(0x1033, 0x4e91efc0, 0x10001, 0x1, 0x2af8d750, 0x4e91ee41, 0x4e91ee80) = 0
[pid 5198] syscall(0x1033, 0x4e91ee80, 0x10001, 0x1, 0x2af8d750, 0x4e91ee41, 0x4e91ee80) = 0
[pid 5198] futex(0x2be0046c, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 5199] <... futex resumed> ) = 0
[pid 5199] nanosleep({0, 900000}, <unfinished ...>
[pid 5198] syscall(0x1033, 0x4e91eebc, 0x10001, 0x1, 0x2af8d750, 0x1, 0x7fe7183c) = 0
[pid 5198] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
[pid 5199] <... nanosleep resumed> ) = ? <unavailable>
[pid 5199] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
[pid 5278] syscall(0x1033, 0x4d020008, 0x10001, 0x1, 0x2b5f8750, 0x4d01fea1, 0x4d01fee0) = 0
[pid 5278] syscall(0x1033, 0x4d020020, 0x10001, 0x1, 0x2b5f8750, 0x4d01fea1, 0x4d01fee0) = 0
[pid 5278] syscall(0x1033, 0x4d01fee0, 0x10001, 0x1, 0x2b5f8750, 0x4d01fea1, 0x4d01fee0) = 0
[pid 5278] futex(0x2c40046c, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 5279] <... futex resumed> ) = 0
[pid 5279] nanosleep({0, 900000}, <unfinished ...>
[pid 5278] syscall(0x1033, 0x4d01ff1c, 0x10001, 0x1, 0x2b5f8750, 0x1, 0x7fced9fc) = 0
[pid 5278] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
[pid 5279] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
syscall 0x1033 = 4000+147 = __NR_cacheflush
Does not repeat on @ZubairLK 's v3.18 but no SMP so that's not at all surprising.
You could take a look at the dev-mips-kitkat-v3.0.8 branch which has a number of cacheflush fixes for SMP. As far as I remember these fixes only affected a Java JIT and not V8, but it might be worth trying them as it looks as if there may be SMP cache flushing problems...
Good news! Repeats on 3.0.8 even with upper memory section dropped and with nosmp on kernel command line, this tells me it might actually be fixed... testing 3.18 next
More good news (for me), with kernel v3.0.8 and SMP but no high memory section in boot args, the problem does not repeat.
root@ci20-3006:~# uname -a
Linux ci20-3006 3.18.3+ #4 SMP PREEMPT Sat Mar 7 10:25:33 EST 2015 mips GNU/Linux
root@ci20-3006:~# cat /proc/cpuinfo
system type : JZ4780
machine : imgtec,ci20
processor : 0
cpu model : Ingenic JZRISC V4.15 FPU V0.0
BogoMIPS : 1196.85
wait instruction : yes
microsecond timers : no
tlb_entries : 32
extra interrupt vector : yes
hardware watchpoint : yes, count: 1, address/irw mask: [0x0fff]
isa : mips1 mips2 mips32r1 mips32r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 0
package : 0
core : 0
VCED exceptions : not available
VCEI exceptions : not available
processor : 1
cpu model : Ingenic JZRISC V4.15 FPU V0.0
BogoMIPS : 4.50
wait instruction : yes
microsecond timers : no
tlb_entries : 32
extra interrupt vector : yes
hardware watchpoint : yes, count: 1, address/irw mask: [0x0fff]
isa : mips1 mips2 mips32r1 mips32r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 0
package : 0
core : 1
VCED exceptions : not available
VCEI exceptions : not available
root@ci20-3006:~# free -h
total used free shared buffers cached
Mem: 244M 80M 164M 4.6M 0B 30M
-/+ buffers/cache: 49M 195M
Swap: 0B 0B 0B
root@ci20-3006:~# for x in {0..100}; do nodejs -e 'console.log("hello")'; done
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
hello
root@ci20-3006:~#
I cannot test with the high memory section because am unable to get my new kernel to boot with it, I get the following:
[ 15.390000] VFS: Mounted root (ubifs filesystem) on device 0:14.
[ 15.390000] Freeing unused kernel memory: 244K (80813000 - 80850000)
[ 15.480000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[ 15.480000]
[ 15.490000] drm_kms_helper: panic occurred, switching back to text console
[ 15.490000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[ 15.490000]
My .config is here: http://sprunge.us/VYAB if anyone has a suggestion how I should change it to be able to boot with the high memory section.
4c51e3b1 fixes this in 3.0.8
To repeat:
This is not fixed by reducing the memory to 256M as per https://github.com/MIPS/CI20_linux/issues/18