Deal with the inevitable stack overrun once and for all

[GoogleCodeExporter]

Issue 113 increases our stack to 1GB, which backs the stack pointer all the way 
up to 0xf0000000. Unless there are other tricks, this is the maximum we can go 
in 32-bit mode, and a 64-bit TenFourFox is not an option. It is better than 
crashing on some sites, but systems with less than 1GB of RAM will thrash like 
mad. We probably can't avoid this.

Tracejit will be disabled in Fx10 (though we may or may not reenable it). It 
will definitely be gone in Fx11 and there will only be methodjit. Methodjit 
will definitely change the properties of how we use the stack. It is not at all 
clear if it will make it better or worse. We are still required to abide by the 
PPC ABI, which means big fat stack frames.

We cannot a priori determine sites that will worst-case the stack, because a 
single recursive function that calls itself over and over will do it, and we 
can't detect this at JIT time.

Ideas Chris proposed:

- Blacklist. Well, we have one available, it's called NoScript :) It won't 
affect legit sites, but will require trial, error and probably a lot of crashes 
to construct.
- Use the dom script timeout or a similar construct. This will abort the 
tracer, which will slow the site down but will clean the stack up. It may also 
affect legitimate sites, but it has the advantage of "just working." We also 
need to find a magical value that correlates to stack abuse, and this will vary 
with the speed of the system in question.

We can't realistically do this until we have determined the status of the 
methodjit (issue 96). If methodjit is go, which has a decent chance, then we 
need to work with that and determine the stack usage characteristics before 
going forward with this. If not, then we need to consider the above ideas and 
do a lot of testing. If we don't do methodjit, then we will essentially be 
dropping source parity, so we could be free to implement our own NoScript like 
we do in Classilla.

Original issue reported on code.google.com by classi...@floodgap.com on 1 Dec 2011 at 2:35

• Blocked on: #96, #113

[GoogleCodeExporter]

One idea zubr brought up was to have the function prologue raise an exception 
for an imminent stack overrun (i.e., if it got below 0xb0000000). However, it's 
not at all clear that such a mechanism exists because we're pretty much the 
only arch that has to worry about it. This could be done in TrampolineOSXPPC, 
or possibly in the prologue generator in NativePPC.cpp if we don't make it to 
methodjit.

Original comment by classi...@floodgap.com on 1 Dec 2011 at 7:19

[GoogleCodeExporter]

If we could get a decent backtrace (maybe with a watchpoint 100 MB down the 
stack before loading the testcase, to catch when it's hitting crazy 
consumption?), I wonder whether the JSContext->stackLimit is even set. If it's 
not, maybe all that needs to happen is to patch in a call to 
JS_SetNativeStackQuota() so that the stack consumption limiting code already in 
js/ actually takes effect?

Original comment by magef...@gmail.com on 2 Dec 2011 at 8:52

[GoogleCodeExporter]

Curiouser and curiouser... I've been doing the following test:

1. Start 10.4Fx
2. attach to it in gdb
3. set a watch point with "watch *(int *)(stack-base - 1024*1024)" and 
stack-base is 0xf0000000 for 9beta or 0xb0000000 for <= 8.0
4. continue in gdb
5. load all the supposedly high stack-usage pages I can and wait for the 
watchpoint to hit
  5a. use ^C in gdb to pause 10.4Fx occasionally and use "info reg" to check the current value of sp (=r1) and see if it's excessive

And with all the test cases from the various closed bugs, I've never seen more 
than ~100 kB of usage... Although that may also be due to the test cases fixing 
their bad code, since I can't reproduce issue 113 in 10.4Fx 8 now, either.

Original comment by magef...@gmail.com on 4 Dec 2011 at 4:41

[GoogleCodeExporter]

https://bugzilla.mozilla.org/show_bug.cgi?id=685097 

This is only for IonMonkey, though it is still based on our old jsapi friends. 
Still, we should be able to wedge that.

Original comment by classi...@floodgap.com on 17 Dec 2011 at 3:39