make operator new more standard compliant

[GoogleCodeExporter]

Users sometimes complain that asan's operator new is not fully 
standard-compliant
when it comes to handling out of memory situation.

In particular: 

  - new-nothrow does not return 0 by default (it does with ASAN_OPTIONS=allocator_may_return_null=1)
  - new does not throw std::bad_alloc
  - either variant does not call new_handler 

Implementing these features appears to be surprisingly difficult.

The implementation of asan's operator new resides in asan/asan_new_delete.cc
http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/asan/asan_new_delete.c
c?view=markup

asan's new/delete is essentially a user replacement of these operators. 
This has a side effect: a user can not replace new/delete with his own variants,
(this is undefined behavior: 
http://en.cppreference.com/w/cpp/memory/new/operator_new)

Throwing exceptions from asan/asan_new_delete.cc is currently not possible 
because
we build the entire run-time w/o exceptions. 
We can overcome this by having special flags for asan/asan_new_delete.cc.
Then there is a problem of linking with c++ lib.
asan run-time does not depend on c++ run-time and can be used with pure C w/o 
linking c++.
We probably may overcome this my splitting asan_new_delete.cc into a separate 
library
asan-rt-cxx that will be linked in asan mode only for C++ programs. 

Calling new_handler from asan's operator new would be possible in c++11, where 
std::get_new_handler() is present, but we can't rely on this yet
(libstdc++ in gcc 4.8.2 does not have std::get_new_handler).
And again, the same problem with linking with c++ lib.

We can also simply remove our new/delete interceptors.
This will route new/delete to the standard c++ library and will solve the 
problem completely
(assuming new/delete call malloc/free),
but we will lose the ability to detect new/free and new[]/delete mismatch bugs. 

So, the possible plan is: 
- split asan run-time, moving asan/asan_new_delete.cc into a separate library 
and link it only when libstdc++/libc++ is linked.
- make our operator new() throw bad_alloc
- declare std::get_new_handler() as weak and if it is present, handle the 
new_handler case.

OMG, that sounds so hard, and for very little outcome.
I am tempted to leave it as is unless we find an easier route.
Better implementation ideas are welcome. 

Original issue reported on code.google.com by konstant...@gmail.com on 15 Apr 2014 at 12:00

[GoogleCodeExporter]

By removing new/delete interceptors we also lose allocation stacks on ARM, 
because standard C++ libraries are often built without frame pointers, or with 
incompatible frame pointers (ARM has no standard FP location in the stack 
frame).

Original comment by euge...@google.com on 15 Apr 2014 at 12:11

[GoogleCodeExporter]

I agree with Evgeniy - dropping new/delete replacements is a bad idea - we will 
almost certainly have worse stack traces even on Linux. Splitting ASan runtime 
in two parts (i.e. moving C++-specific bits into a separate library and link it 
if necessary) is quite possible - we do it in UBSan already (there are 
clang_rt.ubsan.a and clang_rt.ubsan_cxx.a), and have the required logic in 
driver.

Original comment by samso...@google.com on 15 Apr 2014 at 9:26

[GoogleCodeExporter]

let us split the run-time first. 

Original comment by konstant...@gmail.com on 16 Apr 2014 at 7:20

[GoogleCodeExporter]

Oh, good.
Having an example to work from will make it much easier, I suspect.

� Marshall

Original comment by mclow.li...@gmail.com on 16 Apr 2014 at 1:41

[GoogleCodeExporter]

http://llvm.org/bugs/show_bug.cgi?id=19660 is asking to support 
custom operator new()

Original comment by konstant...@gmail.com on 6 May 2014 at 11:42

[GoogleCodeExporter]

http://llvm.org/viewvc/llvm-project?view=revision&revision=208609 splits ASan 
runtime in two parts. Now it should be possible to add better support for 
bad_alloc, and compile asan_cxx part with exceptions.

Original comment by samso...@google.com on 12 May 2014 at 6:48