Caller save register clobbered by call instructions

[penberg]

Given this test program:

public class Hashtable<K, V> {
  private final float loadFactor;

  public Hashtable(int initialCapacity, float loadFactor)
  {
    HashEntry<K, V>[] buckets = (HashEntry<K, V>[]) new HashEntry[initialCapacity];
    if (!(loadFactor > 0))
      throw new IllegalArgumentException();
    this.loadFactor = loadFactor;
  }

  private static final class HashEntry<K, V> {
  }

  public static void main(String[] args) {
    Hashtable ht = new Hashtable(100, 0.75f);
  }
}

If you compile Jato with GCC optimization level "-O2", for example:

make CFLAGS=-O2

and then run the program as follows:

javac Hashtable.java

./jato -Xtrace:exceptions \
   -bootclasspath .:/usr/local/classpath/share/classpath/glibj.zip \
   -Djava.library.path=/usr/local/classpath/lib/classpath/ \
   -Xnosystemclassloader Hashtable

You will see the following crash:

[main] trace exception: exception object 0x236c630 (java/lang/IllegalArgumentException) thrown
[main]  message : 0x0 null
[main]  from    : 0x40fa8fc7: Hashtable.<init>(IF)V
[main]        (Hashtable.java:12)
[main]  action  : unwind to 0x40fa8db1: Hashtable.main([Ljava/lang/String;)V
[main]        (Hashtable.java:20)
[main] trace exception: exception object 0x236c630 (java/lang/IllegalArgumentException) thrown
[main]  message : 0x0 null
[main]  from    : 0x40fa8db0: Hashtable.main([Ljava/lang/String;)V
[main]        (Hashtable.java:20)
[main]  action  : unwind to native caller at 0x41e4b8

Exception in thread "main" java.lang.IllegalArgumentException at Hashtable.(Hashtable.java:12) at Hashtable.main(Hashtable.java:20)

Now while this doesn't seem to happen with the "-Os" GCC optimization level, it's by pure luck. Looking at the generated assembly, we see that XMM registers (which are caller saved registers on x86-64) are not saved before call instructions:

[main]   0x41d7ff51:   48 89 e5                   mov    %rsp,%rbp
[main]   0x41d7ff54:   48 83 ec 40                sub    $0x40,%rsp
[main]   0x41d7ff58:   53                         push   %rbx
[main]   0x41d7ff59:   41 54                      push   %r12
[main]   0x41d7ff5b:   41 55                      push   %r13
[main]   0x41d7ff5d:   41 56                      push   %r14
[main]   0x41d7ff5f:   41 57                      push   %r15
[main]   0x41d7ff61:   57                         push   %rdi
[main]   0x41d7ff62:   49 89 ff                   mov    %rdi,%r15
[main]   0x41d7ff65:   49 89 f6                   mov    %rsi,%r14
[main]   0x41d7ff68:   f3 0f 10 f8                movss  %xmm0,%xmm7
[main]   0x41d7ff6c:   4d 89 fd                   mov    %r15,%r13
[main]   0x41d7ff6f:   4d 85 6d 00                test   %r13,0x0(%r13)
[main]   0x41d7ff73:   4c 89 ef                   mov    %r13,%rdi
[main]   0x41d7ff76:   f6 04 25 00 90 d3 00 00    testb  $0x0,0xd39000
[main]   0x41d7ff7e:   e8 bd cd ff ff             callq  0x0000000041d7cd40       # java/lang/Object.<init>()V
[main]   0x41d7ff83:   45 89 f6                   mov    %r14d,%r14d
[main]   0x41d7ff86:   4c 89 f7                   mov    %r14,%rdi
[main]   0x41d7ff89:   e8 c2 51 6b be             callq  0x0000000000435150       # array_size_check+0 (/home/penberg/jato/vm/object.c:494)

which means that the any function that uses the XMM registers will clobber our registers.

I went through the code and I believe that the register allocator does not properly handle caller-saved registers. The special-casing in __update_live_ranges() seems incomplete because it doesn't update the "defs" bitmap nor updates use positions.

[penberg]

AFAICT, the way to fix this is to make sure caller save register are saved and restored by the spill/reload logic of the register allocator. However, we need to do this as a separate pass after we've assigned physical registers to struct var_infos; otherwise we fail to catch all clobbers.