Porting to a new target (AArch64)

[GoogleCodeExporter]

Hello,

I am currently working on enabling address-sanitizer in GCC for AArch64.

I am not sure if this is the right channel to ask for support?

I have made some changes necessary to have it build for this new target 
(GetPcSpBp, syscalls, ...) but now I am facing runtime problems:
==15518==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 
(15392894357504) bytes at address 0x02008fff7000 (12)
==15518==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 
bytes. Perhaps you're using ulimit -v

In my case, in asan_rtl.cc, full_shadow_is_available is false, and I guess I 
have to give sensible values to kMidMemBeg and kMidMemEnd.

How do I compute the right values, and will this fix the error message above?

Thanks

Original issue reported on code.google.com by christop...@linaro.org on 22 Nov 2013 at 12:14

[GoogleCodeExporter]

This is *the* right channel. 
I've never touched AArch64 before, so I have no idea.
You can start from posting the contents of /proc/self/maps here. 

Also, please note, that all work on the asan run-time should be done in llvm 
repository first and then it will get merged into gcc. 

Original comment by konstant...@gmail.com on 22 Nov 2013 at 12:17

[GoogleCodeExporter]

Here is the contents of /proc/self/maps (where self == /bin/cat)
# cat /proc/self/maps
00400000-0040b000 r-xp 00000000 fe:02 93                                 
/bin/cat.coreutils
0041a000-0041b000 rwxp 0000a000 fe:02 93                                 
/bin/cat.coreutils
3b799000-3b7ba000 rwxp 00000000 00:00 0                                  [heap]
7f7ca25000-7f7cb52000 r-xp 00000000 fe:02 611                            
/lib/libc-2.17-2013.07-2.so
7f7cb52000-7f7cb61000 ---p 0012d000 fe:02 611                            
/lib/libc-2.17-2013.07-2.so
7f7cb61000-7f7cb65000 r-xp 0012c000 fe:02 611                            
/lib/libc-2.17-2013.07-2.so
7f7cb65000-7f7cb67000 rwxp 00130000 fe:02 611                            
/lib/libc-2.17-2013.07-2.so
7f7cb67000-7f7cb6b000 rwxp 00000000 00:00 0
7f7cb6b000-7f7cb86000 r-xp 00000000 fe:02 652                            
/lib/ld-2.17-2013.07-2.so
7f7cb8b000-7f7cb8c000 rwxp 00000000 00:00 0
7f7cb93000-7f7cb94000 rwxp 00000000 00:00 0
7f7cb94000-7f7cb96000 r-xp 00000000 00:00 0                              [vdso]
7f7cb96000-7f7cb97000 r-xp 0001b000 fe:02 652                            
/lib/ld-2.17-2013.07-2.so
7f7cb97000-7f7cb99000 rwxp 0001c000 fe:02 652                            
/lib/ld-2.17-2013.07-2.so
7fc44c3000-7fc44e4000 rwxp 00000000 00:00 0                              [stack]

I don't work on LLVM, but I can make progress on the GCC side, and forward you 
the results when I think it's ready, so that it can be discussed with the LLVM 
team, or I'll ask Renato Golin.

Thanks

Original comment by christop...@linaro.org on 22 Nov 2013 at 12:52

[GoogleCodeExporter]

So, the address space is 0..2^39 and otherwise it looks pretty simple.
You need ShadowOffset=2^36 and kHighMemEnd=(2^39)-1 -- please try that.

>> but I can make progress on the GCC side,
Sure. But please don't commit changes tolibsanitizer directly to gcc tree. 
They should go via LLVM. 

It would be nice if someone can set up a buildbot for LLVM+asan on AArch64

Original comment by konstant...@gmail.com on 22 Nov 2013 at 1:00

[GoogleCodeExporter]

How do you know that ShadowOffset should be 2^36? It's not obvious to me, sorry.

As a quick hack, I have forced kHighMemEnd = 0xfffffffffULL in 
InitializeHighMemEnd(), and I get a new error:
libsanitizer/sanitizer_common/sanitizer_allocator.h:314 "((kSpaceBeg)) == 
((reinterpret_cast<uptr>(Mprotect(kSpaceBeg, kSpaceSize))))" (0x600000000000, 
0xffffffffffffffff)

(and I set shadow offset in aarch64_asan_shadow_offset to HOST_WIDE_INT_1 << 36)

Original comment by christop...@linaro.org on 22 Nov 2013 at 1:32

[GoogleCodeExporter]

2^36 is what you get if you divide 2^39 by 8 (the shadow scale factor)
it could be some other smaller value too. (e.g. in x86_64 we use 0x7FFF8000)

>> "((kSpaceBeg)) == ((reinterpret_cast<uptr>(Mprotect(kSpaceBeg, 
kSpaceSize))))" (0x600000000000, 0xffffffffffffffff)

That's the next failure. 
play with kAllocatorSpace and kAllocatorSize in asan/asan_allocator2.cc
It should be somewhere in the upper half of the address space.

Original comment by konstant...@gmail.com on 22 Nov 2013 at 2:09

[GoogleCodeExporter]

Hmmm..
I have set
const uptr kAllocatorSpace =  0x4000000000ULL;
const uptr kAllocatorSize  =  0x4000000000ULL;

and now the programs all end with segfault :-(

I tried to use gdb but it doesn't work, not sure why. It says:
Cannot find new threads: debugger service failed

However I am using a simulator, so it may be causing some problems; in 
particular when running (or trying to run) the GCC testsuite, the simulator 
always crashes at some point.

Not sure how to make more progress :-(

Thanks.

Original comment by christop...@linaro.org on 22 Nov 2013 at 3:37

[GoogleCodeExporter]

0x7fffffffff -- end of address space
0x4000000000 -- beginning of allocator space
0x8000000000 -- end of allocator space.
This is not going to work :)

try smaller kAllocatorSize first (e.g. 0x2000000000 or 0x1000000000)
Make sure you understand how the allocator works 
(sanitizer_common/sanitizer_allocator.h)

If nothing works, switch to using SizeClassAllocator32 instead of 
SizeClassAllocator64

Original comment by konstant...@gmail.com on 22 Nov 2013 at 5:09

[GoogleCodeExporter]

Why do you say this is not going to work? Need to have free space after the 
allocator space?

It seems I can't reduce kAllocatorSize; from what I understood I have 
kNumClassesRounded=64, and since kRegionSize must be >= 2^32, this means
kAllocatorSize >= 2^38 = 0x4000000000ULL

Since I must have kSpaceBeg % kSpaceSize == 0, this implies kAllocatorSpace =  
0x4000000000ULL.

As of switching to SizeClassAllocator32, I did a quick hack in 
sanitizer_internal_defs.h:
#ifdef __aarch64__
#undef SANITIZER_WORDSIZE
#define SANITIZER_WORDSIZE 32
#endif

but then in sanitizer_linux.cc, I had to make sure that 
SANITIZER_LINUX_USES_64BIT_SYSCALLS == 1
and in sanitizer_placement_new.h, make sure that
typedef uptr operator_new_ptr_type
for aarch64.

With this, the lib finally compiles, but at runtime:
==1399==AddressSanitizer CHECK failed: 
/XXX/libsanitizer/asan/asan_thread.cc:136 "((AddrIsInMem(stack_bottom_))) != 
(0)" (0x0, 0x0)
=================================================================
==1399==ERROR: AddressSanitizer: unknown-crash on address 0x7fc63948d0 at pc 
0x7faa929588 bp 0x7fc6393a70 sp 0x7fc6393aa8
WRITE of size 960 at 0x7fc63948d0 thread T0
==1399==AddressSanitizer CHECK failed: 
/XXX/libsanitizer/asan/asan_report.cc:239 "((0 && "Address is not in memory and 
not in shadow?")) != (0)" (0x0, 0x0)

Original comment by christop...@linaro.org on 25 Nov 2013 at 3:42

[GoogleCodeExporter]

>> Why do you say this is not going to work? Need to have free space after the 
allocator space?
Because the addressed like 7fc44c3000 are used for stack and DSOs.

Well, perhaps the 39-bit address space is too small to use our 64-bit 
allocator. 
See how we use SizeClassAllocator32 in 
sanitizer_common/sanitizer_allocator_internal.h on 64-bit.
You'll need similar changes in asan/asan_allocator2.cc
Make sure you check the fresh llvm code base, not gcc. 

>> #define SANITIZER_WORDSIZE 32
That's definitely not what you want. 

Original comment by konstant...@gmail.com on 26 Nov 2013 at 10:23

[GoogleCodeExporter]

http://llvm.org/viewvc/llvm-project?view=revision&revision=198044
should give you what you need for asan's allocator.
Just define SANITIZER_CAN_USE_ALLOCATOR64 to 0

Original comment by konstant...@gmail.com on 26 Dec 2013 at 1:59

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Please see http://gcc.gnu.org/PR64435
Aarch64 doesn't have always 39-bit address space, but configurably either 39, 
42 or 47 bit address space.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64435#c19 contains a partial patch 
for that, which lets at least the LowMem/LowShadow/ShadowGap/HighShadow/HighMem 
division be runtime decided to support all 3.  Unfortunately the 39-bit address 
space is quite small and it is hard to stick in the allocator64 at a fixed 
address, and 32-bit allocator doesn't really work with 42-bit addres space 
(assertion failure).

Original comment by svha...@gmail.com on 20 Jan 2015 at 5:00

[GoogleCodeExporter]

svhavel@, 
Let's fix the assertion in Allocator32.
All patches *must* be sent upstream, we can not accept patches from gcc mailing 
lists. 
https://code.google.com/p/address-sanitizer/wiki/HowToContribute

Alternatively, we can decide that we support sanitizers only with 47-bit AS. 
This will simplify lots of things. 

Original comment by konstant...@gmail.com on 20 Jan 2015 at 5:29

[GoogleCodeExporter]

I can send the current patch there, sure.
I don't think anybody ships with 47-bit AS right now, so requiring only that 
would be likely equal to not supporting it at all.

Original comment by svha...@gmail.com on 20 Jan 2015 at 5:51

[GoogleCodeExporter]

So, if we keep using Allocator32, that would supposedly mean we have to revert 
the
// The range of addresses which can be returned my mmap.
// FIXME: this value should be different on different platforms,
// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still work
// but will consume more memory for TwoLevelByteMap.
#if defined(__aarch64__)
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 39)
#else
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 47)
#endif
change and use 1ULL << 47 even on aarch64?  What implications does it have 
exactly (i.e. how much more memory it will consume?).

Or, what is the minimum usable size for Allocator64?  In case of 39-bit address 
space it is tight and we definitely can't find terrabytes of memory anywhere.  
42-bit and 47-bit AS has space for it, but the question is if the space will be 
always at the same base.

Original comment by svha...@gmail.com on 21 Jan 2015 at 9:26

[GoogleCodeExporter]

For the record, the asan tests passed on a recent GCC trunk, on a AArch64 juno 
board, running with 39-bits address space. That's the only HW I have access to 
at the moment.

Original comment by christop...@linaro.org on 21 Jan 2015 at 12:30

[GoogleCodeExporter]

FYI, following patch makes ASAN work fine on 42-bit AS, but will break the 
39-bit AS and won't fix 48-bit one.  So we really need something more dynamic.

--- libsanitizer/asan/asan_allocator.h  (revision 219833)
+++ libsanitizer/asan/asan_allocator.h  (working copy)
@@ -100,6 +100,10 @@
 # if defined(__powerpc64__)
 const uptr kAllocatorSpace =  0xa0000000000ULL;
 const uptr kAllocatorSize  =  0x20000000000ULL;  // 2T.
+# elif defined(__aarch64__)
+// Valid only for 42-bit VA
+const uptr kAllocatorSpace =  0x10000000000ULL;
+const uptr kAllocatorSize  =  0x10000000000ULL;  // 1T.
 # else
 const uptr kAllocatorSpace = 0x600000000000ULL;
 const uptr kAllocatorSize  =  0x40000000000ULL;  // 4T.
--- libsanitizer/sanitizer_common/sanitizer_platform.h  (revision 219833)
+++ libsanitizer/sanitizer_common/sanitizer_platform.h  (working copy)
@@ -79,7 +79,7 @@
 // For such platforms build this code with -DSANITIZER_CAN_USE_ALLOCATOR64=0 or
 // change the definition of SANITIZER_CAN_USE_ALLOCATOR64 here.
 #ifndef SANITIZER_CAN_USE_ALLOCATOR64
-# if defined(__aarch64__) || defined(__mips64)
+# if defined(__mips64)
 #  define SANITIZER_CAN_USE_ALLOCATOR64 0
 # else
 #  define SANITIZER_CAN_USE_ALLOCATOR64 (SANITIZER_WORDSIZE == 64)
@@ -88,10 +88,10 @@

 // The range of addresses which can be returned my mmap.
 // FIXME: this value should be different on different platforms,
-// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still 
work
+// e.g. on AArch64 it is most likely (1ULL << 42). Larger values will still 
work
 // but will consume more memory for TwoLevelByteMap.
 #if defined(__aarch64__)
-# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 39)
+# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 42)
 #else
 # define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 47)
 #endif

Original comment by svha...@gmail.com on 26 Jan 2015 at 3:20