asan should instrument calls via pointer

[ramosian-glider]

Originally reported on Google Code with ID 75

Like here: 

void caller(void (*callback)(int), int input) {
  callback(input);
}

Reported by konstantin.s.serebryany on 2012-05-29 14:58:46

[ramosian-glider]

On second thought I don't see a good test case where this would be interesting. 
Closing until we find such test. 

Reported by konstantin.s.serebryany on 2012-05-30 10:55:29

• Status changed: WontFix

[ramosian-glider]

This would be really useful if we want to do native execution of statically instrumented
code in a hybrid tool.

If the LLVM instrumented code is not running in the code cache, then DR (or any other
DBI framework) will not be able to detect indirect calls across module boundaries.

For x86, I was thinking this should be something like this.  Instead of a standard
indirect call:
call reg/mem

Do:
mov rax, reg/mem
call indirect_call_trampoline
...
indirect_call_trampoline:
jmp %rax

Then DR can patch indirect_call_trampoline when it is present, and performance shouldn't
suffer much when it isn't.

If you did this at the LLVM level, you'd want to find some way to make the arguments
match the callee perfectly, so the trampoline is optimized to a jmp without any parameter
marshaling.  It might be possible to do this with a custom calling convention.  Query
the CC used, find a clobbered register not used for params, create a new CC with that
reg as the first param, and pass the function pointer in it.

Reported by rnk@google.com on 2012-05-30 12:54:56

[ramosian-glider]

That's another story. Agree, this might be helpful. 
In the original report I meant that we need to check if callback points to addressable
memory, but then realized that I can't imagine a test where this would be useful. 

Reported by konstantin.s.serebryany on 2012-05-30 13:04:15

[ramosian-glider]

Adding Project:AddressSanitizer as part of GitHub migration.

Reported by ramosian.glider on 2015-07-30 09:12:59

• Labels added: ProjectAddressSanitizer