optimize asan's stack frame layout

[GoogleCodeExporter]

The current stack frame layout generated by llvm's asan is far from optimal.

Today, even a 1-byte stack variable will generate 32*3 stack frame,
while a 64-byte frame should be enough 
(can't make it 32 bytes because we need a 32-byte header).  
If we have two small objects on stack we can also fit them into 64-byte frame:
32-byte header, 16 for each object and its redzone

stack frames are 32-byte aligned in llvm's variant (making the shadow 4-byte 
aligned)
but there seem to be no need for that in x86 (may want to flip the 
asan-realign-stack flag).

We do not support stack variables with alignment > 32 bytes,
although 64-bit alignment may become more popular with AVX-512

We need adaptive redzone sizes for stack vars  (issue 8) to increase 
the probability of catching OOBs in large objects.

We may want to poison the areas of stack that belong 
to the return address and compiler's redzone (could be tricky). 

For frames like "char a; LargeObj b; char c, LargeObj d, ..."
we may need to sort the objects (first by alignment, then within a given 
alignment by size).
But this will complicate bug analysis unless we implement
Issue 241 (provide more information for unnamed temps)

Today the inlined code poisons stack with 4-byte stores,
in some cases 8-byte stores may be faster. 

We may want to write the stack layout code in such a way that GCC can reuse it 
as is
(separate C++ module w/o deps on LLVM; thanks God and other fellows GCC is now 
C++)

Please comment. 

Original issue reported on code.google.com by konstant...@gmail.com on 18 Nov 2013 at 8:59

• Blocked on: #241

[GoogleCodeExporter]

http://llvm-reviews.chandlerc.com/D2324 mailed for review.

Original comment by konstant...@gmail.com on 4 Dec 2013 at 8:53

[GoogleCodeExporter]

http://llvm.org/viewvc/llvm-project?view=revision&revision=196568 implements a 
more compact and adaptive stack frame layout. 
I observe some performance improvements on SPEC in use-after-return mode due
to smaller frames (train data; w/o this change vs w/ this change):
       400.perlbench,        83.30,        79.00,         0.95
           445.gobmk,       222.00,       212.00,         0.95
           458.sjeng,       296.00,       293.00,         0.99
       483.xalancbmk,       205.00,       187.00,         0.91
          453.povray,        27.90,        28.80,         1.03

This is all I wanted to achieve for now, closing. 

Original comment by konstant...@gmail.com on 6 Dec 2013 at 9:12

• Changed state: Fixed