Got a crash when accessing archive.org

[roytam1]

rev range: https://github.com/rmottola/Arctic-Fox/compare/d664435b2c7c11bf6b185a9b7360e1722b5fad39...fd4f0e474420cbd3b5da4511a5945bf61313919d

stack seems to be corrupted, not showing any meaningful content.

[roytam1]

it seems to be javascript ion related, with javascript.options.ion=false it doesn't crash anymore when accessing archive.org.

[roytam1]

so most likely bug 1207922 cause the crash.

[roytam1]

for the record, firefox-46.0b1 doesn't crash when browsing archive.org

also tried to build https://hg.mozilla.org/mozilla-central/rev/46db83228504 and it works as well.

[roytam1]

trying to build with --enable-debug, but too many things are broken. created a hack to get it compiled: debug.patch

browser freezes when trying to access internet, and get a different crash when closing console window:

no clue whats going on.

[roytam1]

trying to disable HSTS, and networking seem to work now. and got another crash related to what bug 1207922 touches:

and ins->regexp()->type() becomes MIRType_Value for unknown reason.

[rmottola]

What do you mean "crash" ? literally crash in the browser? If I access archive.org on linux amd64, I see a white page (so for sure some unsupported JS thing) but no crash, I get CSP errors (I don't konw if legit or not)

[roytam1]

I get CSP errors (I don't konw if legit or not)

you have to disable the (broken) csp to trigger this bug.

[rmottola]

disabling CSP, I get a crash on Linux too... working on some incomplete applied patches for JS.. let's see if they help... right now build is broken, must check

[rmottola]

I landed more than 100 JS related patches .. huge improvements in JS tests!

[rmottola]

I tried web.archive.org and it works, yet only archive.org an it crashes, this hints to me more to a network related problem than JS, or not?

[rmottola]

trying to disable HSTS, and networking seem to work now. and got another crash related to what bug 1207922 touches:

and ins->regexp()->type() becomes MIRType_Value for unknown reason.

I wnet into preferences and disable HSTS, still get a crash .... doe sit work for you or do you just mean "a different crash"

[roytam1]

I wnet into preferences and disable HSTS, still get a crash

yeah you can't get rid this crash by disabling HSTS. (but you can workaround this by disabling javascript.options.ion for now) I tried to disable HSTS just because HSTS is buggy in --enable-debug build, disabling HSTS gets networking work again.

[roytam1]

I tried web.archive.org and it works, yet only archive.org an it crashes

web.archive.org and archive.org are serving different sets of javascript files, maybe this causes the differences.

[roytam1]

and I'm afraid we may finally need to branch out from https://github.com/rmottola/Arctic-Fox/commit/fd4f0e474420cbd3b5da4511a5945bf61313919d and trying to add commits to make /js look like https://github.com/mozilla/gecko-dev/commit/4304cf309c6763c1161eb039beb30e93d627d457 to see if we can get rid of that ion bug.

[rmottola]

I wnet into preferences and disable HSTS, still get a crash

yeah you can't get rid this crash by disabling HSTS. (but you can workaround this by disabling javascript.options.ion for now) I tried to disable HSTS just because HSTS is buggy in --enable-debug build, disabling HSTS gets networking work again.

I will look into more ion differences as well as debug stuff...

[rmottola]

and I'm afraid we may finally need to branch out from fd4f0e4 and trying to add commits to make /js look like mozilla/gecko-dev@4304cf3 to see if we can get rid of that ion bug.

maybe I did a mispatch somewhere, or a dependency is still missing... if you identify something specific let me know. I hope the huge bump in JS I did would help.

I will test also on ARM, just to see if it is a JS/VM/ION issue or an intel issue.

[rmottola]

Some more info... I can reproduce the issue on ARM. Disabling ION makes archive.org load.... so indeed ION exposes the issue. I tried importing more patches which I either missed/skipped or had incomplete patch sets. Sputnik tests are very happy now! But the issue is still there. I imported also some patches which are "renaming only", since that helps comparison with gecko files (or also your work by replacing files with Gecko as test). I am diffing files manually these days to spot differences. Did you do any work narrowing down? Maybe not only /js but narrowing to /js/jit only could also be of interest.

[roytam1]

Some more info... I can reproduce the issue on ARM. Disabling ION makes archive.org load.... so indeed ION exposes the issue.

Got another user report about the crash: https://msfn.org/board/?app=core&module=system&controller=content&do=find&content_class=forums_Topic&content_id=184051&content_commentid=1251393

and of course it relates to same issue. and since the crash is in random memory location, I think it is running generated(optimized) bytecode by jit.

Did you do any work narrowing down? Maybe not only /js but narrowing to /js/jit only could also be of interest.

tried but feels no clue on /js/src/jit since it seems that there is no notable differences between mozilla's and ours.

[roytam1]

and I'm afraid we may finally need to branch out from fd4f0e4 and trying to add commits to make /js look like mozilla/gecko-dev@4304cf3 to see if we can get rid of that ion bug.

tried to diff between them and difference is quite large, may because of missing patches in both sides: af-vs-mc-46db83228504.patch

[rmottola]

and I'm afraid we may finally need to branch out from fd4f0e4 and trying to add commits to make /js look like mozilla/gecko-dev@4304cf3 to see if we can get rid of that ion bug.

tried to diff between them and difference is quite large, may because of missing patches in both sides: af-vs-mc-46db83228504.patch

indeed, that's quite big... I'll try to narrow it doesn't. Im actually targeting a later version of MC, about 15 February, but focus on patches between december 2015 and january 2016. I found I missed some older stuff, it will surely help for other things.

[roytam1]

got a crash on another site: voh.com.vn

[rmottola]

I worked for days on JavaScript... I pulled in some big changesets, it was hard becasue they layered on some missing stuff which, if applied later, would mispatch. Also browser modules and toolkit needed to be patched or it would itself on startup The bonus is best score ever in sputnik javascript tests! However, no fix in your issue. Does it help you finding it?

[roytam1]

Also browser modules and toolkit needed to be patched

yeah this will be hard for me to fix my "downstream" browsers.

[rmottola]

I can reproduce now in debug mode... and have the same (even after the tons of patches I applied):

Assertion failure: ins->regexp()->type() == MIRType_Object, at /home/multix/code/Arctic-Fox/js/src/jit/Lowering.cpp:2155

Program /home/multix/code/Arctic-Fox/obj-x86_64-unknown-linux-gnu/dist/bin/arcticfox (pid = 19246) received signal 11.

[rmottola]

Also browser modules and toolkit needed to be patched

yeah this will be hard for me to fix my "downstream" browsers.

I was able to find always patches/set of patches (but I had to apply them manually mostly). I did not tweak something manually.... It should help you. It helps to open in with --jsconsole and see if you get an unexpected error, check the line and see it was changed in JavaScript and apply the corresponding patch- with half a dozen I got AF running again, everything is commited - immediately after the breaking.

[rmottola]

trying to build with --enable-debug, but too many things are broken. created a hack to get it compiled: debug.patch

Some more good news: I worked several evenings... now you can compile out-of-the-box with --enable-debug. I applied all relevant patches. Does not change the error, but it is good anyway also for other debugging!

[rmottola]

This is my stacktrace:

#0  0x00007f2f9c889aee in  () at /lib64/libc.so.6
#1  0x00007f2f9c88c261 in pthread_cond_wait () at /lib64/libc.so.6
#2  0x00007f2f9c5ded1b in PR_WaitCondVar (cvar=0x7f2f90b15540, timeout=4294967295)
    at /home/multix/code/Arctic-Fox/nsprpub/pr/src/pthreads/ptsynch.c:395
#3  0x00007f2f9a2deab3 in js::GlobalHelperThreadState::wait(js::GlobalHelperThreadState::CondVar, unsigned int)
    (this=0x7f2f9c676800, millis=0, which=js::GlobalHelperThreadState::CONSUMER) at /home/multix/code/Arctic-Fox/js/src/vm/HelperThreads.h:259
#4  0x00007f2f9a2e6796 in js::CancelOffThreadIonCompile(JSCompartment*, JSScript*)
    (compartment=compartment@entry=0x7f2f7119d800, script=script@entry=0x0) at /home/multix/code/Arctic-Fox/js/src/vm/HelperThreads.h:273
#5  0x00007f2f99f6787a in js::jit::JitCompartment::sweep(js::FreeOp*, JSCompartment*)
    (this=0x7f2f75db2600, fop=fop@entry=0x7ffcb2335af0, compartment=0x7f2f7119d800) at /home/multix/code/Arctic-Fox/js/src/jit/Ion.cpp:704
#6  0x00007f2f9a17bea4 in JSCompartment::sweepJitCompartment(js::FreeOp*) (this=<optimized out>, fop=fop@entry=0x7ffcb2335af0)
    at /home/multix/code/Arctic-Fox/js/src/jscompartment.cpp:714
#7  0x00007f2f9a1cf106 in js::gc::GCRuntime::beginSweepingZoneGroup() (this=this@entry=0x7f2f90be8428)
    at /home/multix/code/Arctic-Fox/js/src/jsgc.cpp:5196
#8  0x00007f2f9a1d4aca in js::gc::GCRuntime::beginSweepPhase(bool) (this=this@entry=0x7f2f90be8428, destroyingRuntime=destroyingRuntime@entry=false)
    at /home/multix/code/Arctic-Fox/js/src/jsgc.cpp:5379
#9  0x00007f2f9a1da442 in js::gc::GCRuntime::incrementalCollectSlice(js::SliceBudget&, JS::gcreason::Reason)
    (this=this@entry=0x7f2f90be8428, budget=..., reason=reason@entry=JS::gcreason::PERIODIC_FULL_GC)
    at /home/multix/code/Arctic-Fox/js/src/jsgc.cpp:6142
#10 0x00007f2f9a1db1bc in js::gc::GCRuntime::gcCycle(bool, js::SliceBudget&, JS::gcreason::Reason)
    (this=this@entry=0x7f2f90be8428, nonincrementalByAPI=nonincrementalByAPI@entry=false, budget=..., reason=reason@entry=JS::gcreason::PERIODIC_FULL_GC) at /home/multix/code/Arctic-Fox/js/src/jsgc.cpp:6352
#11 0x00007f2f9a1db79d in js::gc::GCRuntime::collect(bool, js::SliceBudget, JS::gcreason::Reason)
    (this=0x7f2f90be8428, nonincrementalByAPI=nonincrementalByAPI@entry=false, budget=..., reason=reason@entry=JS::gcreason::PERIODIC_FULL_GC)
    at /home/multix/code/Arctic-Fox/js/src/jsgc.cpp:6458
#12 0x00007f2f9a1dbd03 in js::gc::GCRuntime::startGC(JSGCInvocationKind, JS::gcreason::Reason, long)
    (this=<optimized out>, gckind=<optimized out>, reason=JS::gcreason::PERIODIC_FULL_GC, millis=<optimized out>)
    at /home/multix/code/Arctic-Fox/js/src/jsgc.cpp:6524
#13 0x00007f2f96d792e9 in mozilla::dom::AutoEntryScript::~AutoEntryScript() (this=this@entry=0x7ffcb2336120, __in_chrg=<optimized out>)
    at /home/multix/code/Arctic-Fox/dom/base/ScriptSettings.cpp:591
#14 0x00007f2f96eb62c7 in nsScriptLoader::EvaluateScript(nsScriptLoadRequest*, JS::SourceBufferHolder&)
    (this=this@entry=0x7f2f6c1526a0, aRequest=aRequest@entry=0x7f2f6c1b1780, aSrcBuf=...)
    at /home/multix/code/Arctic-Fox/dom/base/nsScriptLoader.cpp:1152
#15 0x00007f2f96eb68d8 in nsScriptLoader::ProcessRequest(nsScriptLoadRequest*)
    (this=this@entry=0x7f2f6c1526a0, aRequest=aRequest@entry=0x7f2f6c1b1780) at /home/multix/code/Arctic-Fox/dom/base/nsScriptLoader.cpp:966

[rmottola]

I can load voh.com.vn with and without CSP on my laptop. Can you reproduce it @roytam1 ? and does still reproduce with the latest update? Is it ION related like archive.org?

archive.org still crashes for me.

[rmottola]

@roytam1 latest commits make archive.org load for me!

[roytam1]

good to hear, but I have to test it tomorrow (not now because of too tired today)

[roytam1]

and yeah I can confirm that latest js/ changes helps the situation. closing this for now.

[rmottola]

and yeah I can confirm that latest js/ changes helps the situation. closing this for now.

excellent!