Return value of call trampolines.

[Emegua]

If I have my own implementation of malloc called myMalloc, and if I want to instrument a binary to use myMalloc instead of malloc, what arguments should I pass for e9tool. I used the following command:

./e9tool --option --debug=true -s -M '(call and target==&malloc)' -P 'replace myMalloc@my_malloc' test

The call to malloc is replaced with a corresponding jump instruction, but how can I make sure the pointer to the allocated memory space is correctly returned from myMalloc?

Thanks.

[GJDuck]

Some knowledge of x86_64 calling conventions is necessary. Specifically, the first argument (the size) is stored in register %rdi, and the return value will be stored in register %rax. So myMalloc can take two arguments, e.g.:

    #include "stdlib.c"
    void myMalloc(intptr_t rdi, intptr_t *rax)
    {
        // Example: replace with stdlib.c malloc():
        *rax = (intptr_t)malloc(rdi);
    }

    e9tool -M  '((call or jmp) and target==&malloc)' -P 'replace myMalloc(%rdi,&%rax)@my_malloc' test

Note the (call or jump) since the program may use jmp for tail calls.

Note that replacing the program's malloc() using this method will have other problems. You'd also need to replace free(), realloc(), calloc() as well as other memory allocation variants. You'd also need to (at least) replace free()/realloc() in all shared library dependencies too, since it is somewhat common that the main program will allocate a pointer that is free'ed by a library. Generally, I find it easier to replace malloc/free using the LD_PRELOAD trick rather than E9Patch.

[Emegua]

Thanks. That resolved the issue.

[Emegua]

I know using LD_PRELOAD is easier, but why do we need to replace free() if we replace malloc?

[GJDuck]

I know using LD_PRELOAD is easier, but why do we need to replace free() if we replace malloc?

It depends if you are changing the allocator. If you are, then the pointer created by your malloc may be passed to the glibc free() which will cause problems.