kill improper instruction cache synchronization when patching tracee's syscall site.

[wangbj]

When patching tracee's syscall site, we did a dummy synchronize function synchronize_from, it did:

• insert a breakpoint instruction (0xcc) at the original syscall instruction
• do ptrace singlestep and let the breakpoint hit
• restore the old instruction from the breakpoint, then continue from the original syscall instruction

However, it causes SIGILL with PC=syscall_site+0x1, where syscall_site+0x1 is the instruction after the breakpoint instruction.

After the synchronize_from hack is removed, we didn't see any SIGILL after more than 100 runs.

In the future, if the SIGILL issue still afloat, we could change PC to a completely different location, such as 0x7000_xxxx, where we still have a lot of space space, the instruction sequence could be:

 0x0f 0x08       INVD; 
 0xcc            bkpt;

Then change PC back to where we're from. This should force the tracee do synchronization.

synchronize_from function is killed per commit: 2a87870ff79a8f43146708155db4a8699dec6069

[rrnewton]

where syscall_site+0x1 is the instruction after the breakpoint instruction.

If the syscall is always one byte, how does this generate an illegal instruction?

The binary instrumentation performed by the tracer could be happening on a different core, right? I guess ideally, after any instrumentation, the guest process would wake up in a state where it executes cpuid and then jumps to the desired PC...

@chamibuddhika - do you have a theory here?

[wangbj]

Syscall insn is two bytes in x86, bkpt is one byte

[chamibuddhika]

I am not quite sure what's happening here. Do we know which thread encounter the SIGILL? IIRC ptrace could be setup to halt a given thread only. But I guess it's not the mode of operation here.

[wangbj]

sometimes it happens because we modified the old 2-byte syscall instruction to:

1) callq trampoline (step1) 2) patch 1) to

  .byte 0xcc     // bkpt
  rest_of_callq   // partial instruction

3) rewind pc (pc -= 2), and let bkpt instruction hit 4) restore the partial instruction in step 2) 5) rewind pc (pc -= 1), remove bkpkt, and continue;

SIGILL sometimes happens at step 5), the proposed (merged) change:

1) callq trampoline 2) rewind pc (pc -=2)

I don't think the old method is entirely necessary and it over complicates things. Thus merged the change.

[rrnewton]

Ah, yes if there is a two byte syscall, then certainly syscall+1 is an anvilid PC.

[wangbj]

Close this issue for now will reopen when this problem reproduce