couchjs segfault with SpiderMonkey 60

wohali commented 4 years ago

Starting to get random failures on Jenkins, only for JS tests, and sporadically (but failing an entire PR attempt in the process):

[2020-01-08T02:30:35.272Z] test/javascript/tests/replicator_db_bad_rep_id.js              fail

[2020-01-08T02:58:01.029Z] test/javascript/tests/view_errors.js           fail

These were both in the history of PR-2411:

https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/detail/PR-2411/11/pipeline https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/detail/PR-2411/12/pipeline

We also are having trouble contacting a worker:

[2020-01-08T02:54:54.891Z] couchdb-ci-worker-dal-1-8 was marked offline:
                Connection was broken: java.util.concurrent.TimeoutException: Ping started
                at 1578451854888 hasn't completed by 1578452094889

Don't know if it's related.

/cc @davisp

wohali commented 4 years ago

ALso seeing:

Remote call on JNLP4-connect connection from 76.9a.30a9.ip4.static.sl-reverse.com/169.48.154.118:23453 failed

on occasion, which is causing nodes to be marked offline temporarily. Something's amiss....

davisp commented 4 years ago

For the JNLP4-connect thing, I saw that on couchdb-ci-worker-dal-1-5 yesterday and it cleared itself up after awhile. No idea what's causing that.

For the failed without a stack trace I've also seen that happen to design_docs.js as well. And I have seen it once or twice locally. No idea what's causing it either.

davisp commented 4 years ago

That is to say, for the JS tests I think there's something awkward going on that's not test specific necessarily. Will be good to try and keep track of the failed runs so we can gather logs and so on.

wohali commented 4 years ago

I've updated the title to be more reflective of what we're seeing (the JNLP issue is being worked separately).

For reference, we're seeing this issue in:

Pull Request Jenkins builds
Full Platform Jenkins builds, but only on Debian Buster
This build which is a PR to add arm64v8 support to the Full Platform Jenkins builds (Debian Buster)

That very last one shows an error code of -11, and some very interesting output:

[2020-01-10T04:57:43.943Z] test/javascript/tests/changes.js                               fail: -11
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''
[2020-01-10T04:57:43.943Z] WHUT? ''

Taken together it's clear we have a segmentation violation error (-11) in couchjs, but so far only when we build with SpiderMonkey 60.

/cc @davisp @jiangphcn who are both actively researching this

wohali commented 4 years ago

The above failure is now 100% reproducible on arm64v8 via make javascript suites="changes":

jenkins@443828ef67a2:~/couchdb$ make javascript suites="changes"
test/javascript/tests/changes.js   fail: -11
WHUT? ''
=======================================================
JavaScript tests complete.
  Failed: 1.  Skipped or passed: 0.
make: *** [Makefile:270: javascript] Error 1

Running against SM 1.8.5, the test passes entirely, BUT we still get the weird output:

$ make javascript suites="changes"
test/javascript/tests/changes.js   pass
WHUT? ''
=======================================================
JavaScript tests complete.
  Failed: 0.  Skipped or passed: 1.

Poke me if you need a login to the box to experiment. Once you do login, enter the Docker environment (that has the build env setup) using docker run --rm -it couchdbdev/debian-buster-erlang-20.3.8.24-1, then cd to set your working directory to something sane (/home/jenkins).

davisp commented 4 years ago

The weird output is me forgetting to remove debug logging when I changed the JUnit generation bit. Am opening a PR to remove that bit right now.

wohali commented 4 years ago

coredump acquired, backtrace info:

Reading symbols from bin/couchjs...done.
[New LWP 25199]
[New LWP 25201]
[New LWP 25203]
[New LWP 25204]
[New LWP 25207]
[New LWP 25208]
[New LWP 25206]
[New LWP 25205]
[New LWP 25202]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `src/couch/priv/couchjs --eval -H -T -u test/javascript/couchdb.uri test/javascr'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  GetPropertyOperation (vp=..., lval=..., pc=<optimized out>, script=..., fp=<optimized out>, cx=<optimized out>)
    at ./js/src/vm/JSContext.h:161
161     ./js/src/vm/JSContext.h: No such file or directory.
[Current thread is 1 (Thread 0xffffaa602d80 (LWP 25199))]

(gdb) t a a bt

Thread 9 (Thread 0xffffa9dfdf70 (LWP 25202)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>)
    at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>)
    at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafdf60) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeacf350)
    at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeacf350) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 8 (Thread 0xffffa97faf70 (LWP 25205)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>)
    at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>)
    at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafe0b0) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeae1e90)
    at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeae1e90) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 7 (Thread 0xffffa95f9f70 (LWP 25206)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>)
    at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>)
    at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafe120) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeacb450)
    at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeacb450) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 6 (Thread 0xffffa91f7f70 (LWP 25208)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>)
    at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>)
    at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafe200) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeae2250)
    at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeae2250) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 5 (Thread 0xffffa93f8f70 (LWP 25207)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...)
    at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>)
    at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>)
    at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafe190) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeae2310)
    at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeae2310) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffffa99fbf70 (LWP 25204)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
--Type <RET> for more, q to quit, c to continue without paging--c
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...) at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...) at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>) at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>) at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafe040) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeacf1d0) at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeacf1d0) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffffa9bfcf70 (LWP 25203)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...) at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...) at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>) at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>) at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafdfd0) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeacf290) at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeacf290) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffffaa5fef70 (LWP 25201)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaaeeaf82d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaaeeaf8248, cond=0xaaaaeeaf82a8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xaaaaeeaf82a8, mutex=0xaaaaeeaf8248) at pthread_cond_wait.c:655
#3  0x0000ffffadda528c in mozilla::detail::ConditionVariableImpl::wait (this=0xaaaaeeaf82a8, lock=...) at ./mozglue/misc/ConditionVariable_posix.cpp:118
#4  0x0000ffffadda54b4 in mozilla::detail::ConditionVariableImpl::wait_for (this=<optimized out>, lock=..., a_rel_time=...) at ./mozglue/misc/ConditionVariable_posix.cpp:127
#5  0x0000ffffadb66184 in js::ConditionVariable::wait_for (lock=..., rel_time=..., this=<optimized out>) at ./js/src/threading/ConditionVariable.h:101
#6  js::GlobalHelperThreadState::wait (timeout=..., which=js::GlobalHelperThreadState::PRODUCER, locked=..., this=<optimized out>) at ./js/src/vm/HelperThreads.cpp:994
#7  js::HelperThread::threadLoop (this=0xaaaaeeafdef0) at ./js/src/vm/HelperThreads.cpp:2238
#8  0x0000ffffadb63990 in js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::callMain<0ul> (this=0xaaaaeeacf410) at ./js/src/threading/Thread.h:241
#9  js::detail::ThreadTrampoline<void (&)(void*), js::HelperThread*>::Start (aPack=0xaaaaeeacf410) at ./js/src/threading/Thread.h:235
#10 0x0000ffffad2bb7e4 in start_thread (arg=0xfffff63f1a8f) at pthread_create.c:486
#11 0x0000ffffad3b2adc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffffaa602d80 (LWP 25199)):
#0  GetPropertyOperation (vp=..., lval=..., pc=<optimized out>, script=..., fp=<optimized out>, cx=<optimized out>) at ./js/src/vm/JSContext.h:161
#1  Interpret (cx=0xfffff63f0230, state=...) at ./js/src/vm/Interpreter.cpp:2834
#2  0x0000ffffad858b48 in js::RunScript (cx=cx@entry=0xaaaaeeaf8530, state=...) at ./js/src/vm/Interpreter.cpp:418
#3  0x0000ffffad85909c in js::InternalCallOrConstruct (cx=cx@entry=0xaaaaeeaf8530, args=..., construct=construct@entry=js::NO_CONSTRUCT) at ./js/src/vm/Interpreter.cpp:490
#4  0x0000ffffad8592ec in InternalCall (cx=cx@entry=0xaaaaeeaf8530, args=...) at ./js/src/vm/Interpreter.cpp:517
#5  0x0000ffffad85932c in js::CallFromStack (cx=cx@entry=0xaaaaeeaf8530, args=...) at ./js/src/vm/Interpreter.cpp:523
#6  0x0000ffffad927f14 in js::jit::DoCallFallback (cx=0xaaaaeeaf8530, frame=0xfffff63f13a8, stub_=0xaaaaeeb5b780, argc=0, vp=0xfffff63f1348, res=...) at ./js/src/jit/BaselineIC.cpp:2380
#7  0x00001a13cbce3a64 in ?? ()
#8  0x0000ffffad858794 in Interpret (cx=0xfffff63f12f0, state=...) at ./js/src/vm/Stack.h:1594
#9  0x0000ffffad858794 in Interpret (cx=0xaaaaeeaf8530, state=...) at ./js/src/vm/Stack.h:1594
#10 0x0000fffff63f18c8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

wohali commented 4 years ago

(gdb) print *cx
$3 = {<JS::RootingContext> = {stackRoots_ = {static kSize = 14, mArray = {mArr = {0x7000007c4c0, 0xaaaaeedce9c0, 0x3, 0xaaaaeebf2970,
          0xfffff63f0c10, 0xffffadbcd99c
     <js::ObjectGroup::newPlainObject(JSContext*, js::IdValuePair*, unsigned long, js::NewObjectKind)+1548>,
          0xffffadf29000 <vtable for JS::ubi::Concrete<JSString>+48>, 0xaaaaeeaf8530, 0xfffff63f0c88,
          0xffffadbcd978 <js::ObjectGroup::newPlainObject(JSContext*, js::IdValuePair*, unsigned long, js::NewObjectKind)+1512>,
          0xffffadf29000 <vtable for JS::ubi::Concrete<JSString>+48>, 0x0, 0xaaaaeeb2cf10, 0x637ae96ce7e7ac00}}},
    autoGCRooters_ = 0xfffff63f03a0, geckoProfiler_ = {pseudoStack_ = 0xffffadb93e48
     <js::JSONParserBase::finishObject(JS::MutableHandle<JS::Value>, mozilla::Vector<js::IdValuePair, 10ul, js::TempAllocPolicy>&)+48>},
    compartment_ = 0xfffff63f0648, zone_ = 0xffffadf29000 <vtable for JS::ubi::Concrete<JSString>+48>, nativeStackLimit = {187651128617328,
      187651128190640, 281474813068368}}, <js::MallocProvider<JSContext>> = {<No data fields>},
  runtime_ = {<js::ProtectedData<js::CheckUnprotected, JSRuntime*>> = {value = 0xfff9000000000001}, <No data fields>}, kind_ = {
    value = (unknown: 20)}, threadNative_ = {value = 187651128617328},
  helperThread_ = {<js::ProtectedData<js::CheckThreadLocal, js::HelperThread*>> = {value = 0xfffff63f0c88}, <No data fields>},
  nurserySuppressions_ = {<js::ProtectedData<js::CheckThreadLocal, unsigned long>> = {value = 281474813067824}, <No data fields>},
  options_ = {<js::ProtectedData<js::CheckThreadLocal, JS::ContextOptions>> = {value = {baseline_ = true, ion_ = false, asmJS_ = false,
        wasm_ = true, wasmBaseline_ = false, wasmIon_ = false, testWasmAwaitTier2_ = false, throwOnAsmJSValidationFailure_ = false,
        nativeRegExp_ = false, asyncStack_ = false, throwOnDebuggeeWouldRun_ = false, dumpStackOnDebuggeeWouldRun_ = false, werror_ = false,
        strictMode_ = false, extraWarnings_ = false, streams_ = false, expressionClosures_ = false,
        arrayProtoValues_ = false}}, <No data fields>}, arenas_ = {<js::ProtectedData<js::CheckThreadLocal, js::gc::ArenaLists*>> = {
      value = 0x0}, <No data fields>}, enterCompartmentDepth_ = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {
      value = 4131324992}, <No data fields>}, static reportedError = {dummy = 0}, static reportedOOM = {<JS::Error> = {
      dummy = 0}, <No data fields>}, yieldCallback_ = {<js::ProtectedData<js::CheckThreadLocal, void (*)(JSContext*)>> = {
      value = 0x2}, <No data fields>}, jitActivation = {<js::ProtectedData<js::CheckThreadLocal, js::jit::JitActivation*>> = {
      value = 0xfffff63f0348}, <No data fields>}, regexpStack = {<js::ProtectedData<js::CheckThreadLocal, js::irregexp::RegExpStack>> = {
      value = {static kStackLimitSlack = 32, static kMemoryTop = 18446744073709551615, static kMinimumStackSize = 512,
        static kMaximumStackSize = 67108864, base_ = 0xffffadddbf50 <JSID_VOID>, size = 187651125642544,
        limit_ = 0xaaaaeedce990}}, <No data fields>}, activation_ = {<js::ProtectedData<js::CheckThreadLocal, js::Activation*>> = {
      value = 0xaaaa00000003}, <No data fields>}, profilingActivation_ = 0xaaaaeeaf8558,
  interpreterStack_ = {<js::ProtectedData<js::CheckThreadLocal, js::InterpreterStack>> = {value = {static DEFAULT_CHUNK_SIZE = 4096,
        allocator_ = {chunks_ = {head_ = {
              mTuple = {<mozilla::detail::PairHelper<js::detail::BumpChunk*, JS::DeletePolicy<js::detail::BumpChunk>, (mozilla::detail::StorageType)1, (mozilla::detail::StorageType)0>> = {<JS::DeletePolicy<js::detail::BumpChunk>> = {<No data fields>}, mFirstA = 0x0}, <No data fields>}},
            last_ = 0x7000007c4c0}, unused_ = {head_ = {
              mTuple = {<mozilla::detail::PairHelper<js::detail::BumpChunk*, JS::DeletePolicy<js::detail::BumpChunk>, (mozilla::detail::StorageType)1, (mozilla::detail::StorageType)0>> = {<JS::DeletePolicy<js::detail::BumpChunk>> = {<No data fields>},
                  mFirstA = 0xaaaaeeaf8568}, <No data fields>}}, last_ = 0xfffff63f0bc8}, markCount = 7696581989928,
          defaultChunkSize_ = 187651125642576, curSize_ = 281474813069536, peakSize_ = 7696595273344, static HUGE_ALLOCATION = 52428800},
        static MAX_FRAMES = 50000, static MAX_FRAMES_TRUSTED = 51000, frameCount_ = 281474813068256}}, <No data fields>},
  nativeStackBase = 7168298410870025216, nativeStackQuota = {{<js::ProtectedData<js::CheckThreadLocal, unsigned long>> = {
        value = 281474813068336}, <No data fields>}, {<js::ProtectedData<js::CheckThreadLocal, unsigned long>> = {
        value = 281473596265724}, <No data fields>}, {<js::ProtectedData<js::CheckThreadLocal, unsigned long>> = {
        value = 281474813068872}, <No data fields>}}, entryMonitor = {<js::ProtectedData<js::CheckThreadLocal, JS::dbg::AutoEntryMonitor*>> = {
      value = 0x3}, <No data fields>}, noExecuteDebuggerTop = {<js::ProtectedData<js::CheckThreadLocal, js::EnterDebuggeeNoExecute*>> = {
      value = 0xaaaaeedce970}, <No data fields>}, activityCallback = {<js::ProtectedData<js::CheckThreadLocal, void (*)(void*, bool)>> = {
      value = 0xfffff63f0630}, <No data fields>}, activityCallbackArg = {<js::ProtectedData<js::CheckThreadLocal, void*>> = {
      value = 0xfffff63f0648}, <No data fields>}, requestDepth = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {
      value = 4007028400}, <No data fields>}, autoFlushICache_ = {<js::ProtectedData<js::CheckThreadLocal, js::jit::AutoFlushICache*>> = {
      value = 0xfffff63f0460}, <No data fields>}, dtoaState = {<js::ProtectedData<js::CheckThreadLocal, DtoaState*>> = {value = 0xffffad870108
     <js::ParseJSONWithReviver<unsigned char>(JSContext*, mozilla::Range<unsigned char const>, JS::Handle<JS::Value>, JS::MutableHandle<JS::Value>)+224>}, <No data fields>}, heapState = {<js::ProtectedData<js::CheckThreadLocal, JS::HeapState>> = {value = -290486992}, <No data fields>},
  suppressGC = {<js::ProtectedData<js::CheckThreadLocal, int>> = {value = 43690}, <No data fields>},
  enableAccessValidation = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {value = 4131325104}, <No data fields>},
  inUnsafeRegion = {<js::ProtectedData<js::CheckThreadLocal, int>> = {value = 65535}, <No data fields>},
  generationalDisabled = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {value = 4131324976}, <No data fields>},
  compactingDisabledCount = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {value = 65535}, <No data fields>},
  keepAtoms = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {value = 2914522364}, <No data fields>},
  frontendCollectionPool_ = {<js::ProtectedData<js::CheckThreadLocal, js::frontend::NameCollectionPool>> = {value = {
        mapPool_ = {<js::frontend::CollectionPool<js::InlineMap<JSAtom*, js::frontend::RecyclableAtomMapValueWrapper<unsigned int>, 24, js::DefaultHasher<JSAtom*>, js::SystemAllocPolicy>, js::frontend::InlineTablePool<js::InlineMap<JSAtom*, js::frontend::RecyclableAtomMapValueWrapper<unsigned int>, 24, js::DefaultHasher<JSAtom*>, js::SystemAllocPolicy> > >> = {all_ = {<js::SystemAllocPolicy> = {<No data fields>},
              static kElemIsPod = true, static kMaxInlineBytes = 999, static kInlineCapacity = 32, mBegin = 0xfffff63f0648,
              mLength = 187651128190640, mTail = {<mozilla::Vector<void*, 32, js::SystemAllocPolicy>::CapacityAndReserved> = {
                  mCapacity = 281473598844648},
                mBytes = "\000\254\347\347l\351zc`\004?\366\377\377\000\000 \001\207\255\377\377\000\000\060\205\257\356\252\252\000\000\001\000\000\000\000\000\000\000\000\220\362\255\377\377\000\000\000\254\347\347l\351zc\320\a?\366\377\377\000\000H\004\207\255\377\377\000\000\000\220\362\255\377\377\000\000\060\205\257\356\252\252\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\260f\326\356\252\252\000\000\020\b?\366\377\377", '\000' <repeats 23 times>, "\200\371\377\060\205\257\356\252\252\000\000\000\000\000\000\252\252\000\000\060\205\257\356\252\252\000\000\350\004?\366\377\377\000\000\000\000\000\000\000\000\000\000\n\000\000\000\000\000\000\000\264\271\274\255\377\377\000\000"...},
              static sMaxInlineStorage = <optimized out>}, recyclable_ = {<js::SystemAllocPolicy> = {<No data fields>},
              static kElemIsPod = true, static kMaxInlineBytes = 999, static kInlineCapacity = 32, mBegin = 0xaaaaeeaf8530,
              mLength = 281474813070400, mTail = {<mozilla::Vector<void*, 32, js::SystemAllocPolicy>::CapacityAndReserved> = {
                  mCapacity = 281474813070464},
                mBytes = " \a?\366\377\377\000\000\270f\326\356\252\252\000\000\000\000\000\000\000\000\000\000`\360\a\000\000\a\000\000P\205\257\356\252\252\000\000`\022\004\000\000\a\000\000\270f\326\356\252\252\000\000\230\f?\366\377\377\000\000\000\220\362\255\377\377\000\000\060\205\257\356\252\252\000\000\250\005?\366\377\377\000\000\000\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\324\026\273\255\377\377\000\000\060\205\257\356\252\252\000\000\200\f?\366\377\377\000\000\000\220\362\255\377\377\000\000\060\205\257\356\252\252\000\000\060\205\257\356\252\252\000\000\360\005?\366\377\377\000\000\000\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\230\205\257\356\252\252\000\000\210\006?\366\377\377\000\000"...}, static sMaxInlineStorage = <optimized out>}}, <No data fields>},
        vectorPool_ = {<js::frontend::CollectionPool<mozilla::Vector<JSAtom*, 24, js::SystemAllocPolicy>, js::frontend::VectorPool<mozilla::Vector<JSAtom*, 24, js::SystemAllocPolicy> > >> = {all_ = {<js::SystemAllocPolicy> = {<No data fields>}, static kElemIsPod = true,
              static kMaxInlineBytes = 999, static kInlineCapacity = 32,
              mBegin = 0xffffad867058 <JS::StructGCPolicy<js::JSONParser<unsigned char> >::trace(JSTracer*, js::JSONParser<unsigned char>*, char const*)>, mLength = 18444633011384221696, mTail = {<mozilla::Vector<void*, 32, js::SystemAllocPolicy>::CapacityAndReserved> = {
                  mCapacity = 187651125642544},
                mBytes = "\000\000\000\000l\351zc0\205\257\356\252\252\000\000\200\006?\366\377\377\000\000\000\000\000\000\000\000\000\000\n\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000p\351\334\356\252\252\000\000\000\000\000\000\377\377\000\000\360\353\270\356\252\252\000\000\001\000\000\000\000\000\000\000p\315\277\356\252\252\000\000\000\000\000\000\000\000\000\000\360\363\270\356\252\252\000\000\001\000\000\000\000\a\376\377\000\330\263\356\252\252\000\000\200\a?\366\377\377\000\000\240N\273\255\377\377\000\000\360\a?\366\377\377\000\000\000\220\362\255\377\377\000\000\000\000\000\000\000\000\000\000\230\f?\366\377\377\000\000\060\205\257\356\252\252\000\000\200\f?\366\377\377\000\000\270f\326\356\252\252\000\000@\f?\366\377\377\000\000"...}, static sMaxInlineStorage = <optimized out>},
            recyclable_ = {<js::SystemAllocPolicy> = {<No data fields>}, static kElemIsPod = true, static kMaxInlineBytes = 999,
              static kInlineCapacity = 32, mBegin = 0xffff00000000, mLength = 0,
              mTail = {<mozilla::Vector<void*, 32, js::SystemAllocPolicy>::CapacityAndReserved> = {mCapacity = 187651125642544},
                mBytes = "\210\a?\366\377\377\000\000\003\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\000\330\263\356\252\252\000\000p\315\277\356\252\252\000\000p\351\334\356\252\252\000\000\001\000\000\000\000\000\000\000h\n?\366\377\377\000\000\241M\302\356\252\252\000\000@K\302\356\252\252\000\000\241M\302\356\252\252\000\000\000\254\347\347l\351zc\200\b?\366\377\377\000\000\204\217\205\255\377\377\000\000\370\b?\366\377\377\000\000\000\220\362\255\377\377\000\000\060\205\257\356\252\252\000\000h\n?\366\377\377\000\000\330\002\207\255\377\377\000\000\220\205\257\356\252\252\000\000p\205\257\356\252\252\000\000\230\v?\366\377\377\000\000p\254\323\000\000\a\000\000@K\302\356\252\252\000\000`Q\b\000\000\a\000\000"...}, static sMaxInlineStorage = <optimized out>}}, <No data fields>}, activeCompilations_ = 4007317000}}, <No data fields>},
  suppressProfilerSampling = {<mozilla::detail::AtomicBase<unsigned int, (mozilla::MemoryOrdering)2>> = {
--Type <RET> for more, q to quit, c to continue without paging--c
      mValue = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 3890719744}, <No data fields>}}, <No data fields>}, static TEMP_LIFO_ALLOC_PRIMARY_CHUNK_SIZE = 4096, tempLifoAlloc_ = {<js::ProtectedData<js::CheckThreadLocal, js::LifoAlloc>> = {value = {chunks_ = {head_ = {mTuple = {<mozilla::detail::PairHelper<js::detail::BumpChunk*, JS::DeletePolicy<js::detail::BumpChunk>, (mozilla::detail::StorageType)1, (mozilla::detail::StorageType)0>> = {<JS::DeletePolicy<js::detail::BumpChunk>> = {<No data fields>}, mFirstA = 0xfffff63f0930}, <No data fields>}}, last_ = 0xffffad84f264 <Interpret(JSContext*, js::RunState&)+6804>}, unused_ = {head_ = {mTuple = {<mozilla::detail::PairHelper<js::detail::BumpChunk*, JS::DeletePolicy<js::detail::BumpChunk>, (mozilla::detail::StorageType)1, (mozilla::detail::StorageType)0>> = {<JS::DeletePolicy<js::detail::BumpChunk>> = {<No data fields>}, mFirstA = 0xffffadefd000 <array_methods+1216>}, <No data fields>}}, last_ = 0xfffff63f0d00}, markCount = 281473600098304, defaultChunkSize_ = 187651125642576, curSize_ = 281474813071016, peakSize_ = 187651125642640, static HUGE_ALLOCATION = 52428800}}, <No data fields>}, debuggerMutations = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {value = 4131327032}, <No data fields>}, ionPcScriptCache = {<js::ProtectedData<js::CheckThreadLocal, js::jit::PcScriptCache*>> = {value = 0xfffff63f0cc0}, <No data fields>}, throwing = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 136}, <No data fields>}, unwrappedException_ = {<js::ProtectedData<js::CheckThreadLocal, JS::PersistentRooted<JS::Value> >> = {value = {<js::RootedBase<JS::Value, JS::PersistentRooted<JS::Value> >> = {<js::MutableWrappedPtrOperations<JS::Value, JS::PersistentRooted<JS::Value> >> = {<js::WrappedPtrOperations<JS::Value, JS::PersistentRooted<JS::Value> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <mozilla::LinkedListElement<JS::PersistentRooted<JS::Value> >> = {mNext = 0xfffff63f0230, mPrev = 0x13, mIsSentinel = 80}, ptr = {data = {asBits = 281474813071016, debugView = {payload47 = 140737324715688, tag = 1}, s = {payload = {i32 = -163639640, u32 = 4131327656, why = 4131327656}}, asDouble = 1.3906703530797566e-309, asPtr = 0xfffff63f0ea8, asWord = 281474813071016, asUIntPtr = 281474813071016}}}}, <No data fields>}, overRecursed_ = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 128}, <No data fields>}, propagatingForcedReturn_ = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 201}, <No data fields>}, liveVolatileJitFrameIter_ = {<js::ProtectedData<js::CheckThreadLocal, js::jit::DebugModeOSRVolatileJitFrameIter*>> = {value = 0xfffff63f0920}, <No data fields>}, reportGranularity = {<js::ProtectedData<js::CheckThreadLocal, int>> = {value = -163640128}, <No data fields>}, resolvingList = {<js::ProtectedData<js::CheckThreadLocal, js::AutoResolving*>> = {value = 0xfffff63f0cc0}, <No data fields>}, generatingError = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 136}, <No data fields>}, cycleDetectorVector_ = {<js::ProtectedData<js::CheckThreadLocal, JS::GCVector<JSObject*, 8, js::TempAllocPolicy> >> = {value = {vector = {<js::TempAllocPolicy> = {cx_ = 0xfffff63f0930}, static kElemIsPod = true, static kMaxInlineBytes = 992, static kInlineCapacity = 8, mBegin = 0xffffad853bd8 <Interpret(JSContext*, js::RunState&)+25608>, mLength = 281474813070768, mTail = {<mozilla::Vector<JSObject*, 8, js::TempAllocPolicy>::CapacityAndReserved> = {mCapacity = 281473592953672}, mBytes = "0\205\257\356\252\252\000\000\000\220\362\255\377\377\000\000\250\016?\366\377\377\000\000\270\017?\366\377\377\000\000\300d]\003\000\a\000\000\000\000\000\000\000\000\000\000\060\205\257\356\252\252\000\000\000\000\000\000\000\000\000"}, static sMaxInlineStorage = <optimized out>}}}, <No data fields>}, data = {<js::ProtectedData<js::CheckUnprotected, void*>> = {value = 0xfffff63f1108}, <No data fields>}, outstandingRequests = {<js::ProtectedData<js::CheckThreadLocal, unsigned int>> = {value = 4004886400}, <No data fields>}, jitIsBroken = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 170}, <No data fields>}, asyncStackForNewActivations_ = {<js::ProtectedData<js::CheckThreadLocal, JS::PersistentRooted<js::SavedFrame*> >> = {value = {<js::RootedBase<js::SavedFrame*, JS::PersistentRooted<js::SavedFrame*> >> = {<js::MutableWrappedPtrOperations<js::SavedFrame*, JS::PersistentRooted<js::SavedFrame*> >> = {<js::WrappedPtrOperations<js::SavedFrame*, JS::PersistentRooted<js::SavedFrame*> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <mozilla::LinkedListElement<JS::PersistentRooted<js::SavedFrame*> >> = {mNext = 0x0, mPrev = 0x1222000011ab, mIsSentinel = 19}, ptr = 0xaaaaeeaf8560}}, <No data fields>}, asyncCauseForNewActivations = {<js::ProtectedData<js::CheckThreadLocal, char const*>> = {value = 0xfffff63f0cd0 "@\221\323"}, <No data fields>}, asyncCallIsExplicit = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 168}, <No data fields>}, interruptCallbacks_ = {<js::ProtectedData<js::CheckThreadLocal, mozilla::Vector<bool (*)(JSContext*), 2, js::SystemAllocPolicy> >> = {value = {<js::SystemAllocPolicy> = {<No data fields>}, static kElemIsPod = true, static kMaxInlineBytes = 999, static kInlineCapacity = 2, mBegin = 0xaaaaeeaf8598, mLength = 281474813070112, mTail = {<mozilla::Vector<bool (*)(JSContext*), 2, js::SystemAllocPolicy>::CapacityAndReserved> = {mCapacity = 281474813070488}, mBytes = "p\f?\366\377\377\000\000\300\n?\366\377\377\000"}, static sMaxInlineStorage = <optimized out>}}, <No data fields>}, interruptCallbackDisabled = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 32}, <No data fields>}, interrupt_ = {<mozilla::detail::AtomicBaseIncDec<unsigned int, (mozilla::MemoryOrdering)0>> = {<mozilla::detail::AtomicBase<unsigned int, (mozilla::MemoryOrdering)0>> = {mValue = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 65535}, <No data fields>}}, <No data fields>}, <No data fields>}, interruptRegExpJit_ = {<mozilla::detail::AtomicBaseIncDec<unsigned int, (mozilla::MemoryOrdering)0>> = {<mozilla::detail::AtomicBase<unsigned int, (mozilla::MemoryOrdering)0>> = {mValue = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 4131326776}, <No data fields>}}, <No data fields>}, <No data fields>}, handlingJitInterrupt_ = {<mozilla::detail::AtomicBase<unsigned int, (mozilla::MemoryOrdering)2>> = {mValue = {<std::__atomic_base<unsigned int>> = {static _S_alignment = 4, _M_i = 65535}, <No data fields>}}, <No data fields>}, fx = {cond_ = 0xaaaaeeaf8588, state_ = 4004480368, static lock_ = {<mozilla::detail::AtomicBaseIncDec<js::Mutex*, (mozilla::MemoryOrdering)2>> = {<mozilla::detail::AtomicBase<js::Mutex*, (mozilla::MemoryOrdering)2>> = {mValue = {_M_b = {_M_p = 0xaaaaeeaf82e0}}}, <No data fields>}, <No data fields>}, canWait_ = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 170}, <No data fields>}}, osrTempData_ = {<js::ProtectedData<js::CheckThreadLocal, unsigned char*>> = {value = 0xfffff63f0ad8 "\220\205\257\356\252\252"}, <No data fields>}, ionReturnOverride_ = {<js::ProtectedData<js::CheckThreadLocal, JS::Value>> = {value = {data = {asBits = 281474813070232, debugView = {payload47 = 140737324714904, tag = 1}, s = {payload = {i32 = -163640424, u32 = 4131326872, why = 4131326872}}, asDouble = 1.3906703530758832e-309, asPtr = 0xfffff63f0b98, asWord = 281474813070232, asUIntPtr = 281474813070232}}}, <No data fields>}, jitStackLimit = {<mozilla::detail::AtomicBaseIncDec<unsigned long, (mozilla::MemoryOrdering)0>> = {<mozilla::detail::AtomicBase<unsigned long, (mozilla::MemoryOrdering)0>> = {mValue = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 281474813070256}, <No data fields>}}, <No data fields>}, <No data fields>}, jitStackLimitNoInterrupt = {<js::ProtectedData<js::CheckThreadLocal, unsigned long>> = {value = 281474813070208}, <No data fields>}, getIncumbentGlobalCallback = {<js::ProtectedData<js::CheckThreadLocal, JSObject* (*)(JSContext*)>> = {value = 0xfffff63f0bf8}, <No data fields>}, enqueuePromiseJobCallback = {<js::ProtectedData<js::CheckThreadLocal, bool (*)(JSContext*, JS::Handle<JSObject*>, JS::Handle<JSObject*>, JS::Handle<JSObject*>, void*)>> = {value = 0xaaaaeeaf8568}, <No data fields>}, enqueuePromiseJobCallbackData = {<js::ProtectedData<js::CheckThreadLocal, void*>> = {value = 0xfffff63f0be0}, <No data fields>}, jobQueue = {<js::ProtectedData<js::CheckThreadLocal, JS::PersistentRooted<JS::GCVector<JSObject*, 0, js::SystemAllocPolicy> >*>> = {value = 0xaaaaeed666d0}, <No data fields>}, drainingJobQueue = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 200}, <No data fields>}, stopDrainingJobQueue = {<js::ProtectedData<js::CheckThreadLocal, bool>> = {value = 102}, <No data fields>}, promiseRejectionTrackerCallback = {<js::ProtectedData<js::CheckThreadLocal, void (*)(JSContext*, JS::Handle<JSObject*>, JS::PromiseRejectionHandlingState, void*)>> = {value = 0x3}, <No data fields>}, promiseRejectionTrackerCallbackData = {<js::ProtectedData<js::CheckThreadLocal, void*>> = {value = 0xaaaaeeaf8530}, <No data fields>}}

wohali commented 4 years ago

Consistent reproduction of this issue with the following JS tests in the arm64v8 environment:

changes.js erlang_views.js list_views.js recreate_doc.js reduce_builtin.js rewrite.js rewrite_js.js users_db_security.js view_include_docs.js view_multi_key_all_docs.js view_multi_key_design.js view_multi_key_temp.js view_offsets.js view_pagination.js view_update_seq.js

Test results from a make javascript with the -a flag manually added to ignore failures below.

test/javascript/tests/all_docs.js                              ported to elixir
test/javascript/tests/attachment_names.js                      ported to elixir
test/javascript/tests/attachment_paths.js                      ported to elixir
test/javascript/tests/attachment_ranges.js                     ported to elixir
test/javascript/tests/attachment_views.js                      ported to elixir
test/javascript/tests/attachments.js                           ported to elixir
test/javascript/tests/attachments_multipart.js                 ported to elixir
test/javascript/tests/auth_cache.js                            ported to elixir
test/javascript/tests/basics.js                                ported to elixir
test/javascript/tests/batch_save.js                            ported to elixir
test/javascript/tests/bulk_docs.js                             ported to elixir
test/javascript/tests/changes.js                               fail: -11
test/javascript/tests/coffee.js                                ported to elixir
test/javascript/tests/compact.js                               ported to elixir
test/javascript/tests/config.js                                ported to elixir
test/javascript/tests/conflicts.js                             ported to elixir
test/javascript/tests/cookie_auth.js                           ported to elixir
test/javascript/tests/copy_doc.js                              ported to elixir
test/javascript/tests/design_docs.js                           pass
test/javascript/tests/design_docs_query.js                     pass
test/javascript/tests/design_options.js                        pass
test/javascript/tests/design_paths.js                          pass
test/javascript/tests/erlang_views.js                          fail: -11
test/javascript/tests/etags_head.js                            pass
test/javascript/tests/etags_views.js                           skipped
test/javascript/tests/form_submit.js                           pass
test/javascript/tests/http.js                                  pass
test/javascript/tests/invalid_docids.js                        ported to elixir
test/javascript/tests/jsonp.js                                 pass
test/javascript/tests/large_docs.js                            ported to elixir
test/javascript/tests/list_views.js                            fail: -11
test/javascript/tests/lots_of_docs.js                          ported to elixir
test/javascript/tests/method_override.js                       pass
test/javascript/tests/multiple_rows.js                         ported to elixir
test/javascript/tests/proxyauth.js                             pass
test/javascript/tests/purge.js                                 pass
test/javascript/tests/reader_acl.js                            pass
test/javascript/tests/recreate_doc.js                          fail: -11
test/javascript/tests/reduce.js                                ported to elixir
test/javascript/tests/reduce_builtin.js                        fail: -11
test/javascript/tests/reduce_false.js                          pass
test/javascript/tests/reduce_false_temp.js                     pass
test/javascript/tests/replicator_db_bad_rep_id.js              pass
test/javascript/tests/replicator_db_by_doc_id.js               pass
test/javascript/tests/replicator_db_compact_rep_db.js          skipped
test/javascript/tests/replicator_db_continuous.js              skipped
test/javascript/tests/replicator_db_credential_delegation.js   skipped
test/javascript/tests/replicator_db_field_validation.js        skipped
test/javascript/tests/replicator_db_filtered.js                skipped
test/javascript/tests/replicator_db_identical.js               skipped
test/javascript/tests/replicator_db_identical_continuous.js    skipped
test/javascript/tests/replicator_db_invalid_filter.js          skipped
test/javascript/tests/replicator_db_security.js                skipped
test/javascript/tests/replicator_db_simple.js                  skipped
test/javascript/tests/replicator_db_successive.js              skipped
test/javascript/tests/replicator_db_survives.js                skipped
test/javascript/tests/replicator_db_swap_rep_db.js             skipped
test/javascript/tests/replicator_db_update_security.js         skipped
test/javascript/tests/replicator_db_user_ctx.js                skipped
test/javascript/tests/replicator_db_write_auth.js              skipped
test/javascript/tests/rev_stemming.js                          pass
test/javascript/tests/rewrite.js                               fail: -11
test/javascript/tests/rewrite_js.js                            fail: -11
test/javascript/tests/security_validation.js                   pass
test/javascript/tests/show_documents.js                        pass
test/javascript/tests/stats.js                                 skipped
test/javascript/tests/update_documents.js                      pass
test/javascript/tests/users_db.js                              ported to elixir
test/javascript/tests/users_db_security.js                     fail: -11
test/javascript/tests/utf8.js                                  ported to elixir
test/javascript/tests/uuids.js                                 ported to elixir
test/javascript/tests/view_collation.js                        ported to elixir
test/javascript/tests/view_collation_raw.js        pass
test/javascript/tests/view_compaction.js           pass
test/javascript/tests/view_conflicts.js            pass
test/javascript/tests/view_errors.js               pass
test/javascript/tests/view_include_docs.js         fail: -11
test/javascript/tests/view_multi_key_all_docs.js   fail: -11
test/javascript/tests/view_multi_key_design.js     fail: -11
test/javascript/tests/view_multi_key_temp.js       fail: -11
test/javascript/tests/view_offsets.js              fail: -11
test/javascript/tests/view_pagination.js           fail: -11
test/javascript/tests/view_sandboxing.js           pass
test/javascript/tests/view_update_seq.js           fail: -11

wohali commented 4 years ago

Recompiling SM60 with Ion disabled (--disable-ion) fixed all the above errors on arm64v8. Hrm.

This isn't a good long-term solution, because Debian ships with this enabled - unless we go back to shipping our own compiled version and linking against that instead.

Options include:

build and ship SM60 for platforms, with Ion disabled (lots of heavy lifting)
fall back to SM 1.8.5 on all binary builds for 3.0 release and declare SM60 support beta quality (disappointing for end users who install our binaries)
work harder to find out what's really going on in the JIT code. There's a lot of tips at 1 and 2 on this.

I'm out of time for the week, here's hoping someone else can pick this up.

jiangphcn commented 4 years ago

thanks @wohali for above.

build and ship SM60 for platforms, with Ion disabled (lots of heavy lifting)

does it mean that we need more build time to generate SM library for shipment? Is it once-for-all? Or other efforts to take?

fall back to SM 1.8.5 on all binary builds for 3.0 release and declare SM60 support beta quality (disappointing for end users who install our binaries)

" all binary builds" means that it can't be installed using apt-get -y install libmozjs-60-dev or something like this?

wohali commented 4 years ago

build and ship SM60 for platforms, with Ion disabled (lots of heavy lifting) does it mean that we need more build time to generate SM library for shipment? Is it once-for-all? Or other efforts to take?

It is effort for each OS and platform combination (i.e. debian-buster-ppc64le, centos-8-x86_64, etc.) and since SM60 is still changing we'll need to rebuild and repackage it as it receives new patches. It's a lot of work.

fall back to SM 1.8.5 on all binary builds for 3.0 release and declare SM60 support beta quality (disappointing for end users who install our binaries) " all binary builds" means that it can't be installed using apt-get -y install libmozjs-60-dev or something like this?

No, because

you have to pick SM 60 or SM 1.8.5 at compile time, so we can't ship a binary that supports both easily. if we're falling back to SM 1.8.5, the binaries we ship will be configured for SM 1.8.5 at compile time, not SM 60.
even if we work out a way to support both at compile time (shipping 2x couchjs for instance), the upstream package is currently the problem, since it has Ion enabled and that's the source of the crashes in couchjs right now.

ideally we fix the problem with Ion, either in our code or SpiderMonkey. Barring that we have the above choice to make, and neither selection is ideal.

davisp commented 4 years ago

I managed to duplicate a segfault on x86_64 and it also appears to be related to the JIT compiler based on the garbage frames on the stack:

#0  0x00007f4bac91057c in JSRope::flatten (this=0x7f4ba0001fe8, maybecx=maybecx@entry=0x559cf0e444c0)
    at ./debian/build/dist/include/js/HeapAPI.h:169
#1  0x00007f4bac6d21eb in JSString::ensureLinear (cx=0x559cf0e444c0, this=<optimized out>)
    at ./js/src/vm/StringType.h:420
#2  js::str_split_string (cx=0x559cf0e444c0, group=..., str=..., sep=..., limit=2147483647)
    at ./js/src/builtin/String.cpp:3211
#3  0x0000366f6c2fab99 in ?? ()
#4  0x0000000000000002 in ?? ()
#5  0x00007ffd54cea4e8 in ?? ()
#6  0x0000000000000040 in ?? ()
#7  0x00007f4bacc5b8e0 in ?? () from /usr/lib/x86_64-linux-gnu/libmozjs-60.so.0
#8  0x0000366f6c31babc in ?? ()
#9  0x000000000000c020 in ?? ()
#10 0x00007f4ba8097370 in ?? ()
#11 0x00007f4ba0001fe8 in ?? ()
#12 0x00007f4ba8000780 in ?? ()
#13 0x000000007fffffff in ?? ()
#14 0x0000000000000000 in ?? ()

I discovered that there are options to disable the JIT compiler. Doing so seems to prevent the segfault from triggering. PR #2446 includes this diff and I'll merge it if/when it comes back green.

jiangphcn commented 4 years ago

Great finding, @davisp and just found some logic in SpiderMonkey for these two options.

    case JSJITCOMPILER_ION_ENABLE:
        if (value == 1) {
            JS::ContextOptionsRef(cx).setIon(true);
            JitSpew(js::jit::JitSpew_IonScripts, "Enable ion");
        } else if (value == 0) {
            JS::ContextOptionsRef(cx).setIon(false);
            JitSpew(js::jit::JitSpew_IonScripts, "Disable ion");
        }
        break;
      case JSJITCOMPILER_BASELINE_ENABLE:
        if (value == 1) {
            JS::ContextOptionsRef(cx).setBaseline(true);
            ReleaseAllJITCode(rt->defaultFreeOp());
            JitSpew(js::jit::JitSpew_BaselineScripts, "Enable baseline");
        } else if (value == 0) {
            JS::ContextOptionsRef(cx).setBaseline(false);
            ReleaseAllJITCode(rt->defaultFreeOp());
            JitSpew(js::jit::JitSpew_BaselineScripts, "Disable baseline");
        }

Also, found others related to ION in case we need in the future.

JSJITCOMPILER_ION_WARMUP_TRIGGER
JSJITCOMPILER_ION_GVN_ENABLE
JSJITCOMPILER_ION_FORCE_IC
JSJITCOMPILER_ION_CHECK_RANGE_ANALYSIS

wohali commented 4 years ago

We think this is closed by #2446 . Will reopen if necessary.

wohali commented 4 years ago

Re-opening this because we're still seeing coredumps from couchjs SM60 - but less often.

The two that have been found recently were both on x86_64.

https://gist.github.com/wohali/541e05b674c894c1245634813fec7f23

and

https://gist.github.com/wohali/bbc844322d741ad7fecb39535d096e7d

jiangphcn commented 4 years ago

the line is matched to https://github.com/apache/couchdb/blob/master/src/couch/priv/couch_js/60/main.cpp#L485 which is used to ExecuteScript

wohali commented 4 years ago

Not a segfault, but a failure i've never seen before:

[2020-01-25T07:32:05.985Z] test/javascript/tests/reduce_builtin.js                        

[2020-01-25T07:32:05.985Z]     Error: read_failure

[2020-01-25T07:32:05.985Z] Trace back (most recent call first):

[2020-01-25T07:32:05.985Z]     

[2020-01-25T07:32:05.985Z]  551:14: test/javascript/couch.js

[2020-01-25T07:32:05.985Z]       CouchError

[2020-01-25T07:32:05.985Z]  514:12: test/javascript/couch.js

[2020-01-25T07:32:05.985Z]       CouchDB.maybeThrowError

[2020-01-25T07:32:05.985Z]  180:5: test/javascript/couch.js

[2020-01-25T07:32:05.985Z]       CouchDB/this.query

[2020-01-25T07:32:05.985Z]  185:19: test/javascript/tests/reduce_builtin.js

[2020-01-25T07:32:05.985Z]       couchTests.reduce_builtin

[2020-01-25T07:32:05.985Z]  45:5: test/javascript/cli_runner.js

[2020-01-25T07:32:05.985Z]       runTest

[2020-01-25T07:32:05.985Z]  56:1: test/javascript/cli_runner.js

[2020-01-25T07:32:05.985Z]       

[2020-01-25T07:32:05.985Z] fail: 1

https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/detail/master/54/pipeline/181

Debian buster ppc64, SM60.

jiangphcn commented 4 years ago

thanks @wohali. it looks that this is not always happening. I re-triggered build, and got pass for above JS test in https://ci-couchdb.apache.org/blue/rest/organizations/jenkins/pipelines/jenkins-cm1/pipelines/FullPlatformMatrix/branches/master/runs/55/nodes/198/log/?start=0.

hi Paul, @davisp, From read_failure, i only found hit in https://github.com/apache/couchdb/blob/19c67e1e52a4071e1c41e6c36cf6b6ddd2e448e2/src/fabric/src/fabric_doc_open.erl#L161.

wohali commented 4 years ago

Tentatively closing this. I'm seeing various failures, esp. in mem3, but these don't appear to be SM60 related anymore.

apache / couchdb

couchjs segfault with SpiderMonkey 60 #2423