Porting to a new target (AArch64)

[ramosian-glider]

Originally reported on Google Code with ID 246

Hello,

I am currently working on enabling address-sanitizer in GCC for AArch64.

I am not sure if this is the right channel to ask for support?

I have made some changes necessary to have it build for this new target (GetPcSpBp,
syscalls, ...) but now I am facing runtime problems:
==15518==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504)
bytes at address 0x02008fff7000 (12)
==15518==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps
you're using ulimit -v

In my case, in asan_rtl.cc, full_shadow_is_available is false, and I guess I have to
give sensible values to kMidMemBeg and kMidMemEnd.

How do I compute the right values, and will this fix the error message above?

Thanks

Reported by christophe.lyon@linaro.org on 2013-11-22 12:14:13

[ramosian-glider]

This is *the* right channel. 
I've never touched AArch64 before, so I have no idea.
You can start from posting the contents of /proc/self/maps here. 

Also, please note, that all work on the asan run-time should be done in llvm 
repository first and then it will get merged into gcc. 

Reported by konstantin.s.serebryany on 2013-11-22 12:17:50

[ramosian-glider]

Here is the contents of /proc/self/maps (where self == /bin/cat)
# cat /proc/self/maps
00400000-0040b000 r-xp 00000000 fe:02 93                                 /bin/cat.coreutils
0041a000-0041b000 rwxp 0000a000 fe:02 93                                 /bin/cat.coreutils
3b799000-3b7ba000 rwxp 00000000 00:00 0                                  [heap]
7f7ca25000-7f7cb52000 r-xp 00000000 fe:02 611                            /lib/libc-2.17-2013.07-2.so
7f7cb52000-7f7cb61000 ---p 0012d000 fe:02 611                            /lib/libc-2.17-2013.07-2.so
7f7cb61000-7f7cb65000 r-xp 0012c000 fe:02 611                            /lib/libc-2.17-2013.07-2.so
7f7cb65000-7f7cb67000 rwxp 00130000 fe:02 611                            /lib/libc-2.17-2013.07-2.so
7f7cb67000-7f7cb6b000 rwxp 00000000 00:00 0
7f7cb6b000-7f7cb86000 r-xp 00000000 fe:02 652                            /lib/ld-2.17-2013.07-2.so
7f7cb8b000-7f7cb8c000 rwxp 00000000 00:00 0
7f7cb93000-7f7cb94000 rwxp 00000000 00:00 0
7f7cb94000-7f7cb96000 r-xp 00000000 00:00 0                              [vdso]
7f7cb96000-7f7cb97000 r-xp 0001b000 fe:02 652                            /lib/ld-2.17-2013.07-2.so
7f7cb97000-7f7cb99000 rwxp 0001c000 fe:02 652                            /lib/ld-2.17-2013.07-2.so
7fc44c3000-7fc44e4000 rwxp 00000000 00:00 0                              [stack]

I don't work on LLVM, but I can make progress on the GCC side, and forward you the
results when I think it's ready, so that it can be discussed with the LLVM team, or
I'll ask Renato Golin.

Thanks

Reported by christophe.lyon@linaro.org on 2013-11-22 12:52:29

[ramosian-glider]

So, the address space is 0..2^39 and otherwise it looks pretty simple.
You need ShadowOffset=2^36 and kHighMemEnd=(2^39)-1 -- please try that.

>> but I can make progress on the GCC side,
Sure. But please don't commit changes tolibsanitizer directly to gcc tree. 
They should go via LLVM. 

It would be nice if someone can set up a buildbot for LLVM+asan on AArch64

Reported by konstantin.s.serebryany on 2013-11-22 13:00:29

[ramosian-glider]

How do you know that ShadowOffset should be 2^36? It's not obvious to me, sorry.

As a quick hack, I have forced kHighMemEnd = 0xfffffffffULL in InitializeHighMemEnd(),
and I get a new error:
libsanitizer/sanitizer_common/sanitizer_allocator.h:314 "((kSpaceBeg)) == ((reinterpret_cast<uptr>(Mprotect(kSpaceBeg,
kSpaceSize))))" (0x600000000000, 0xffffffffffffffff)

(and I set shadow offset in aarch64_asan_shadow_offset to HOST_WIDE_INT_1 << 36)

Reported by christophe.lyon@linaro.org on 2013-11-22 13:32:53

[ramosian-glider]

2^36 is what you get if you divide 2^39 by 8 (the shadow scale factor)
it could be some other smaller value too. (e.g. in x86_64 we use 0x7FFF8000)

>> "((kSpaceBeg)) == ((reinterpret_cast<uptr>(Mprotect(kSpaceBeg, kSpaceSize))))" (0x600000000000,
0xffffffffffffffff)

That's the next failure. 
play with kAllocatorSpace and kAllocatorSize in asan/asan_allocator2.cc
It should be somewhere in the upper half of the address space.

Reported by konstantin.s.serebryany on 2013-11-22 14:09:38

[ramosian-glider]

Hmmm..
I have set
const uptr kAllocatorSpace =  0x4000000000ULL;
const uptr kAllocatorSize  =  0x4000000000ULL;

and now the programs all end with segfault :-(

I tried to use gdb but it doesn't work, not sure why. It says:
Cannot find new threads: debugger service failed

However I am using a simulator, so it may be causing some problems; in particular when
running (or trying to run) the GCC testsuite, the simulator always crashes at some
point.

Not sure how to make more progress :-(

Thanks.

Reported by christophe.lyon@linaro.org on 2013-11-22 15:37:25

[ramosian-glider]

0x7fffffffff -- end of address space
0x4000000000 -- beginning of allocator space
0x8000000000 -- end of allocator space.
This is not going to work :)

try smaller kAllocatorSize first (e.g. 0x2000000000 or 0x1000000000)
Make sure you understand how the allocator works (sanitizer_common/sanitizer_allocator.h)

If nothing works, switch to using SizeClassAllocator32 instead of SizeClassAllocator64

Reported by konstantin.s.serebryany on 2013-11-22 17:09:55

[ramosian-glider]

Why do you say this is not going to work? Need to have free space after the allocator
space?

It seems I can't reduce kAllocatorSize; from what I understood I have kNumClassesRounded=64,
and since kRegionSize must be >= 2^32, this means
kAllocatorSize >= 2^38 = 0x4000000000ULL

Since I must have kSpaceBeg % kSpaceSize == 0, this implies kAllocatorSpace =  0x4000000000ULL.

As of switching to SizeClassAllocator32, I did a quick hack in sanitizer_internal_defs.h:
#ifdef __aarch64__
#undef SANITIZER_WORDSIZE
#define SANITIZER_WORDSIZE 32
#endif

but then in sanitizer_linux.cc, I had to make sure that SANITIZER_LINUX_USES_64BIT_SYSCALLS
== 1
and in sanitizer_placement_new.h, make sure that
typedef uptr operator_new_ptr_type
for aarch64.

With this, the lib finally compiles, but at runtime:
==1399==AddressSanitizer CHECK failed: /XXX/libsanitizer/asan/asan_thread.cc:136 "((AddrIsInMem(stack_bottom_)))
!= (0)" (0x0, 0x0)
=================================================================
==1399==ERROR: AddressSanitizer: unknown-crash on address 0x7fc63948d0 at pc 0x7faa929588
bp 0x7fc6393a70 sp 0x7fc6393aa8
WRITE of size 960 at 0x7fc63948d0 thread T0
==1399==AddressSanitizer CHECK failed: /XXX/libsanitizer/asan/asan_report.cc:239 "((0
&& "Address is not in memory and not in shadow?")) != (0)" (0x0, 0x0)

Reported by christophe.lyon@linaro.org on 2013-11-25 15:42:11

[ramosian-glider]

>> Why do you say this is not going to work? Need to have free space after the allocator
space?
Because the addressed like 7fc44c3000 are used for stack and DSOs.

Well, perhaps the 39-bit address space is too small to use our 64-bit allocator. 
See how we use SizeClassAllocator32 in 
sanitizer_common/sanitizer_allocator_internal.h on 64-bit.
You'll need similar changes in asan/asan_allocator2.cc
Make sure you check the fresh llvm code base, not gcc. 

>> #define SANITIZER_WORDSIZE 32
That's definitely not what you want. 

Reported by konstantin.s.serebryany on 2013-11-26 10:23:47

[ramosian-glider]

http://llvm.org/viewvc/llvm-project?view=revision&revision=198044
should give you what you need for asan's allocator.
Just define SANITIZER_CAN_USE_ALLOCATOR64 to 0

Reported by konstantin.s.serebryany on 2013-12-26 13:59:03

• Labels added: Type-Enhancement
• Labels removed: Type-Defect

[ramosian-glider]

Please see http://gcc.gnu.org/PR64435
Aarch64 doesn't have always 39-bit address space, but configurably either 39, 42 or
47 bit address space.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64435#c19 contains a partial patch for
that, which lets at least the LowMem/LowShadow/ShadowGap/HighShadow/HighMem division
be runtime decided to support all 3.  Unfortunately the 39-bit address space is quite
small and it is hard to stick in the allocator64 at a fixed address, and 32-bit allocator
doesn't really work with 42-bit addres space (assertion failure).

Reported by svhavel on 2015-01-20 17:00:46

[ramosian-glider]

svhavel@, 
Let's fix the assertion in Allocator32.
All patches *must* be sent upstream, we can not accept patches from gcc mailing lists.

https://code.google.com/p/address-sanitizer/wiki/HowToContribute

Alternatively, we can decide that we support sanitizers only with 47-bit AS. 
This will simplify lots of things. 

Reported by konstantin.s.serebryany on 2015-01-20 17:29:51

[ramosian-glider]

I can send the current patch there, sure.
I don't think anybody ships with 47-bit AS right now, so requiring only that would
be likely equal to not supporting it at all.

Reported by svhavel on 2015-01-20 17:51:15

[ramosian-glider]

So, if we keep using Allocator32, that would supposedly mean we have to revert the
// The range of addresses which can be returned my mmap.
// FIXME: this value should be different on different platforms,
// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still work
// but will consume more memory for TwoLevelByteMap.
#if defined(__aarch64__)
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 39)
#else
# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 47)
#endif
change and use 1ULL << 47 even on aarch64?  What implications does it have exactly
(i.e. how much more memory it will consume?).

Or, what is the minimum usable size for Allocator64?  In case of 39-bit address space
it is tight and we definitely can't find terrabytes of memory anywhere.  42-bit and
47-bit AS has space for it, but the question is if the space will be always at the
same base.

Reported by svhavel on 2015-01-21 09:26:34

[ramosian-glider]

For the record, the asan tests passed on a recent GCC trunk, on a AArch64 juno board,
running with 39-bits address space. That's the only HW I have access to at the moment.

Reported by christophe.lyon@linaro.org on 2015-01-21 12:30:21

[ramosian-glider]

FYI, following patch makes ASAN work fine on 42-bit AS, but will break the 39-bit AS
and won't fix 48-bit one.  So we really need something more dynamic.

--- libsanitizer/asan/asan_allocator.h  (revision 219833)
+++ libsanitizer/asan/asan_allocator.h  (working copy)
@@ -100,6 +100,10 @@
 # if defined(__powerpc64__)
 const uptr kAllocatorSpace =  0xa0000000000ULL;
 const uptr kAllocatorSize  =  0x20000000000ULL;  // 2T.
+# elif defined(__aarch64__)
+// Valid only for 42-bit VA
+const uptr kAllocatorSpace =  0x10000000000ULL;
+const uptr kAllocatorSize  =  0x10000000000ULL;  // 1T.
 # else
 const uptr kAllocatorSpace = 0x600000000000ULL;
 const uptr kAllocatorSize  =  0x40000000000ULL;  // 4T.
--- libsanitizer/sanitizer_common/sanitizer_platform.h  (revision 219833)
+++ libsanitizer/sanitizer_common/sanitizer_platform.h  (working copy)
@@ -79,7 +79,7 @@
 // For such platforms build this code with -DSANITIZER_CAN_USE_ALLOCATOR64=0 or
 // change the definition of SANITIZER_CAN_USE_ALLOCATOR64 here.
 #ifndef SANITIZER_CAN_USE_ALLOCATOR64
-# if defined(__aarch64__) || defined(__mips64)
+# if defined(__mips64)
 #  define SANITIZER_CAN_USE_ALLOCATOR64 0
 # else
 #  define SANITIZER_CAN_USE_ALLOCATOR64 (SANITIZER_WORDSIZE == 64)
@@ -88,10 +88,10 @@

 // The range of addresses which can be returned my mmap.
 // FIXME: this value should be different on different platforms,
-// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still work
+// e.g. on AArch64 it is most likely (1ULL << 42). Larger values will still work
 // but will consume more memory for TwoLevelByteMap.
 #if defined(__aarch64__)
-# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 39)
+# define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 42)
 #else
 # define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 47)
 #endif

Reported by svhavel on 2015-01-26 15:20:14

[ramosian-glider]

Reported by ramosian.glider on 2015-07-30 09:05:31

• Labels added: ProjectAddressSanitizer

[ramosian-glider]

Adding Project:AddressSanitizer as part of GitHub migration.

Reported by ramosian.glider on 2015-07-30 09:06:56

[ramosian-glider]

Closing as stale. There's some AArch64 support upstream already.