Standard autoconf test fails on mmap()

[dmik]

Valerius discovered that the standard autoconf test AC_FUNC_MMAP fails to succeed given our mmap() implementation. Here is the failing test case (from http://trac.netlabs.org/ports/browser/autoconf/trunk/lib/autoconf/functions.m4):

/* malloc might have been renamed as rpl_malloc. */
#undef malloc

/* Thanks to Mike Haertel and Jim Avera for this test.
   Here is a matrix of mmap possibilities:
    mmap private not fixed
    mmap private fixed at somewhere currently unmapped
    mmap private fixed at somewhere already mapped
    mmap shared not fixed
    mmap shared fixed at somewhere currently unmapped
    mmap shared fixed at somewhere already mapped
   For private mappings, we should verify that changes cannot be read()
   back from the file, nor mmap's back from the file at a different
   address.  (There have been systems where private was not correctly
   implemented like the infamous i386 svr4.0, and systems where the
   VM page cache was not coherent with the file system buffer cache
   like early versions of FreeBSD and possibly contemporary NetBSD.)
   For shared mappings, we should conversely verify that changes get
   propagated back to all the places they're supposed to be.

   Grep wants private fixed already mapped.
   The main things grep needs to know about mmap are:
   * does it exist and is it safe to write into the mmap'd area
   * how to use it (BSD variants)  */

#include <fcntl.h>
#include <sys/mman.h>

#if !defined STDC_HEADERS && !defined HAVE_STDLIB_H
char *malloc ();
#endif

/* This mess was copied from the GNU getpagesize.h.  */
#ifndef HAVE_GETPAGESIZE
# ifdef _SC_PAGESIZE
#  define getpagesize() sysconf(_SC_PAGESIZE)
# else /* no _SC_PAGESIZE */
#  ifdef HAVE_SYS_PARAM_H
#   include <sys/param.h>
#   ifdef EXEC_PAGESIZE
#    define getpagesize() EXEC_PAGESIZE
#   else /* no EXEC_PAGESIZE */
#    ifdef NBPG
#     define getpagesize() NBPG * CLSIZE
#     ifndef CLSIZE
#      define CLSIZE 1
#     endif /* no CLSIZE */
#    else /* no NBPG */
#     ifdef NBPC
#      define getpagesize() NBPC
#     else /* no NBPC */
#      ifdef PAGESIZE
#       define getpagesize() PAGESIZE
#      endif /* PAGESIZE */
#     endif /* no NBPC */
#    endif /* no NBPG */
#   endif /* no EXEC_PAGESIZE */
#  else /* no HAVE_SYS_PARAM_H */
#   define getpagesize() 8192   /* punt totally */
#  endif /* no HAVE_SYS_PARAM_H */
# endif /* no _SC_PAGESIZE */

#endif /* no HAVE_GETPAGESIZE */

int
main ()
{
  char *data, *data2, *data3;
  const char *cdata2;
  int i, pagesize;
  int fd, fd2;

  pagesize = getpagesize ();

  /* First, make a file with some known garbage in it. */
  data = (char *) malloc (pagesize);
  if (!data)
    return 1;
  for (i = 0; i < pagesize; ++i)
    *(data + i) = rand ();
  umask (0);
  fd = creat ("conftest.mmap", 0600);
  if (fd < 0)
    return 2;
  if (write (fd, data, pagesize) != pagesize)
    return 3;
  close (fd);

  /* Next, check that the tail of a page is zero-filled.  File must have
     non-zero length, otherwise we risk SIGBUS for entire page.  */
  fd2 = open ("conftest.txt", O_RDWR | O_CREAT | O_TRUNC, 0600);
  if (fd2 < 0)
    return 4;
  cdata2 = "";
  if (write (fd2, cdata2, 1) != 1)
    return 5;
  data2 = (char *) mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_SHARED, fd2, 0L);
  if (data2 == MAP_FAILED)
    return 6;
  for (i = 0; i < pagesize; ++i)
    if (*(data2 + i))
      return 7;
  close (fd2);
  if (munmap (data2, pagesize))
    return 8;

  /* Next, try to mmap the file at a fixed address which already has
     something else allocated at it.  If we can, also make sure that
     we see the same garbage.  */
  fd = open ("conftest.mmap", O_RDWR);
  if (fd < 0)
    return 9;
  if (data2 != mmap (data2, pagesize, PROT_READ | PROT_WRITE,
             MAP_PRIVATE | MAP_FIXED, fd, 0L))
    return 10;
  for (i = 0; i < pagesize; ++i)
    if (*(data + i) != *(data2 + i))
      return 11;

  /* Finally, make sure that changes to the mapped area do not
     percolate back to the file as seen by read().  (This is a bug on
     some variants of i386 svr4.0.)  */
  for (i = 0; i < pagesize; ++i)
    *(data2 + i) = *(data2 + i) + 1;
  data3 = (char *) malloc (pagesize);
  if (!data3)
    return 12;
  if (read (fd, data3, pagesize) != pagesize)
    return 13;
  for (i = 0; i < pagesize; ++i)
    if (*(data + i) != *(data3 + i))
      return 14;
  close (fd);
  return 0;
}

[dmik]

Apparently, it fails due to MAP_FIXED not being supported by our mmap. In this particular case, MMAP_FIXED basically just assigns another file to the same memory region. We actually may easily support at least this particular case, it's not a big deal at all. Given this is an autoconf test case, I now have a suspicion that this is may be the most commonly used MAP_FIXED case over there.

An alternative would be to change this test case in autoconf so that MAP_FIXED isn't used on OS/2 's now all configure-based projects will have MMAP support turned off. But I don't like this approach because there may be some projects that rely on MAP_FIXED given that the test passes. And also because it's quite easy to implement this on OS/2.

[dmik]

Well, as I now see, they first munmap() the region at addr2 then map a new one at the same location but with MAP_FIXED and a different file descriptor (the comment in the source is actually misleading). This isn't that easy to support as just changing the associated file...

[dmik]

On the second thought, It's not just "not easy", it's simply impossible to make the above snippet work on OS/2 because for shared memory OS/2 uses different address range than for private memory and we can't really control that.

[dmik]

The only theoretical approach to implement MAP_FIXED working this way is to use DosAliasMem for shared memory allocated for MAP_SHARED and return the resulting address which is from the private range to the caller. However, this means that for each MAP_SHARED mapping we will waste both the shared address space and the private address space (and we will have to call DosAliasMem multiple times to get the same address in all processes for MAP_SHARED mappings). I don't like this approach very much.

So I'm afraid for now we will have to hack autoconf to make it not use MAP_FIXED at all on OS/2.

[dmik]

Further testing shows that the above snippet fails not on MAP_FIXED but on creating a mapping of 4096 bytes (1 page) long on a file which is 1 bytes long. Apparently, Linux and other platforms support it, thus allowing to read and write beyond the end of the associated file. The comments in the snippet suggest that reading in such a case should return zeroes and writing should expand the file it seems. We will have to support that.

[dmik]

After fixing #18, conftest now fails right at MAP_FIXED, so it's the next task.

[dmik]

Well, the autoconf fix is very trivial (already did it and checked), but I will first learn if we should support MAP_FIXED in VirtualBox style (i.e. w/o unmapping the "root" mapping, see #14). If yes, then a fix to autoconf will be different (removing munmap in the middle rather than completely disable the MAP_FIXED code).

[dmik]

I committed the autoconf fix in http://trac.netlabs.org/ports/changeset/1752/autoconf and released new autoconf RPMs. In order for the mmap test to succeed LIBCx must be passed in LIBS (e.g. like LIBS="-lcx") prior to configure execution (the old mmap library fails much earlier so it won't trigger enabling autoconf mmap defines and such).

That's it for now.