test_debugging fails on x86-64 because %gs is 0

[GoogleCodeExporter]

test_debugging has started to fail for me on x86-64.

The cause seems to be Debug::enter()'s test for %gs.  It checks whether %gs is 
zero, and if so, Debug::enter() doesn't increment the recursion counter and it 
returns true.

However, we would expect %gs to be zero on x86-64.  See the test program below.

The result is that, in debugging mode, we get infinite recursion:  
defaultSystemCallHandler() calls Debug::syscall(), which calls gettimeofday(), 
which triggers a call to defaultSystemCallHandler().  Without the recursion 
check, this calls gettimeofday() again.

On my Ubuntu Lucid VM, this didn't just run out of stack, it triggered the OOM 
killer, and my window borders disappeared because the kernel killed Metacity 
(!).

What I don't understand is why the test was passing before.  I'm not sure what 
has changed.  Maybe syscall_entrypoint.cc's special case for gettimeofday() was 
making this work.  But if that is the case, I don't know why this has started 
failing.

I am not sure if %fs/%gs should ever show up as having non-zero values on 
x86-64.  The test program below gives the following output:

%gs = 0
%gs:0 = 1234
%fs = 0
%fs:0 = 139925201401600

#include <stdio.h>
#include <unistd.h>
#include <asm/unistd.h>
#include <asm/prctl.h>

int main() {
  long tls = 1234;
  long val;
  syscall(__NR_arch_prctl, ARCH_SET_GS, &tls);

  asm("mov %%gs, %0" : "=r" (val));
  printf("%%gs = %li\n", val);
  asm("mov %%gs:0, %0" : "=r" (val));
  printf("%%gs:0 = %li\n", val);

  asm("mov %%fs, %0" : "=r" (val));
  printf("%%fs = %li\n", val);
  asm("mov %%fs:0, %0" : "=r" (val));
  printf("%%fs:0 = %li\n", val);

  return 0;
}

Original issue reported on code.google.com by mseaborn@chromium.org on 26 Sep 2010 at 12:24

[GoogleCodeExporter]

On further investigation, syscall_entrypoint.cc's special case for 
gettimeofday() only applies to i386.  It does not apply to x86-64, and I assume 
that is done on the assumption that gettimeofday() is a vsyscall on x86-64 
(implemented using RDTSC) and therefore is fast and does not need to be 
optimised.

On two Ubuntu Hardy machines I have tested, x86-64 gettimeofday() is a 
vsyscall.  This means test_debugging passes, because gettimeofday() does not 
lead to a recursive call to the syscall handler.  When I run the following test 
program under strace --

#include <stdlib.h>
#include <sys/time.h>
#include <unistd.h>

int main() {
  while (1) {
    write(1, "x\n", 2);
    struct timeval tv;
    gettimeofday(&tv, NULL);
  }
  return 0;
}

I get only write() calls:

write(1, "x\n", 2x
)                      = 2
write(1, "x\n", 2x
)                      = 2
...

However, gettimeofday() is not always a vsyscall.  On the Ubuntu Lucid VM where 
test_debugging fails, this test program shows interleaved 
write()/gettimeofday() syscalls.

Original comment by mseaborn@chromium.org on 27 Sep 2010 at 10:14

[GoogleCodeExporter]

Fixed in r136, r137.

Original comment by mseaborn@chromium.org on 17 Oct 2010 at 6:28

• Changed state: Fixed