Get sigaltstack working

[NattyNarwhal]

Would make debugging easier. I think the guard pages have a problem; perhaps we don't pass the right flags to mmap?

[NattyNarwhal]

If you trace the calls:

• When setting up the guard page, it runs mprotect. This fails with ENOMEM, so we try the a fallback.

• The fallback tries to allocate from thin air with mmap. This fails with ENOMEM, so try mapping /dev/zero.

• That fails because AIX doesn't let you use MAP_ANONYMOUS with a valid fd. If we fix that (see patch) then we get ENOMEM again.

diff --git a/mono/utils/mono-mmap.c b/mono/utils/mono-mmap.c
index dc802c4..1605575 100644
--- a/mono/utils/mono-mmap.c
+++ b/mono/utils/mono-mmap.c
@@ -233,6 +233,11 @@ mono_valloc (void *addr, size_t length, int flags, MonoMemAccountType type)
        if (ptr == MAP_FAILED) {
                int fd = open ("/dev/zero", O_RDONLY);
                if (fd != -1) {
+#if defined(_AIX)
+                       /* we can't use MAP_ANONYMOUS with a file desc on AIX, per man page */
+                       if (mflags & MAP_ANONYMOUS)
+                               mflags &= ~MAP_ANONYMOUS;
+#endif
                        ptr = mmap (addr, length, prot, mflags, fd, 0);
                        close (fd);
                }

Per the man page of mmap:

       EBADF
            The fildes parameter is not a valid file descriptor, or the
            MAP_ANONYMOUS flag was set and the fildes parameter is not -1.

       ENOMEM
            There is not enough address space to map len bytes, or the application has not requested Single UNIX Specification, Version 2 compliant behavior and the MAP_FIXED flag was set and part of the
            address-space range (addr, addr+len) is already allocated.

And mprotect:

       ENOMEM
            The application has requested Single UNIX Specification, Version 2 compliant behavior, but addresses in the range are not valid for the address space of the process, or the addresses specify one or
            more pages that are not attached to the user's address space by a previous mmap or shmat subroutine call.

And the relevant sections of code:

/* mini-exceptions.c:mono_setup_altstack, snippet */
        if (mono_mprotect (tls->stack_ovf_guard_base, tls->stack_ovf_guard_size, MONO_MMAP_NONE))
                /* mprotect can fail for the main thread stack */
                gpointer gaddr = mono_valloc (tls->stack_ovf_guard_base, tls->stack_ovf_guard_size, MONO_MMAP_NONE|MONO_MMAP_PRIVATE|MONO_MMAP_ANON|MONO_MMAP_FIXED, MONO_MEM_ACCOUNT_EXCEPTIONS);
                g_assert (gaddr == tls->stack_ovf_guard_base);
                tls->stack_ovf_valloced = TRUE;
        }
/* mono-mmap.c */
void*
mono_valloc (void *addr, size_t length, int flags, MonoMemAccountType type)
{
        void *ptr;
        int mflags = 0;
        int prot = prot_from_flags (flags);

        if (!mono_valloc_can_alloc (length))
                return NULL;

        /* translate the flags */
        if (flags & MONO_MMAP_FIXED)
                mflags |= MAP_FIXED;
        if (flags & MONO_MMAP_32BIT)
                mflags |= MAP_32BIT;

        mflags |= MAP_ANONYMOUS;
        mflags |= MAP_PRIVATE;

        BEGIN_CRITICAL_SECTION;
        ptr = mmap (addr, length, prot, mflags, -1, 0);
        if (ptr == MAP_FAILED) {
                int fd = open ("/dev/zero", O_RDONLY);
                if (fd != -1) {
#if defined(_AIX)
                        /* we can't use MAP_ANONYMOUS with a file desc on AIX, per man page */
                        if (mflags & MAP_ANONYMOUS)
                                mflags &= ~MAP_ANONYMOUS;
#endif
                        ptr = mmap (addr, length, prot, mflags, fd, 0);
                        close (fd);
                }
        }
        END_CRITICAL_SECTION;

        if (ptr == MAP_FAILED)
                return NULL;

        mono_account_mem (type, (ssize_t)length);

        return ptr;
}