lsc2001 commented 8 months ago

Environment

KERNEL: 5.4.0-148-generic (Ubuntu 20.04) CPU: Intel(R) Xeon(R) CPU E7- 4807 @ 1.87GHz LIBC: glibc 2.31 CC: gcc 9.4.0 LIBOPCODES: libopcodes 2.34

NO.1

Outputs of the hook does not match that of strace. Here, a.out is a simple "hello, world" program:

#include <stdio.h>

int main() {
    printf("Hello, world!\n");
    return 0;
}

NO.2

Core dumped when running gcc. Here, hello.c is the same "hello, world" program:

# LIBZPHOOK=./apps/basic/libzphook_basic.so LD_PRELOAD=./libzpoline.so gcc hello.c -o hello
output from __hook_init: we can do some init work here
output from hook_function: syscall number 16
output from hook_function: syscall number 16
output from hook_function: syscall number 12
output from hook_function: syscall number 257
output from hook_function: syscall number 5
output from hook_function: syscall number 9
output from hook_function: syscall number 3
output from hook_function: syscall number 257
output from hook_function: syscall number 5
output from hook_function: syscall number 0
output from hook_function: syscall number 0
output from hook_function: syscall number 3
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 16
output from hook_function: syscall number 16
output from hook_function: syscall number 16
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 13
output from hook_function: syscall number 302
output from hook_function: syscall number 302
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 4
output from hook_function: syscall number 6
output from hook_function: syscall number 6
output from hook_function: syscall number 6
output from hook_function: syscall number 89
output from hook_function: syscall number 6
output from hook_function: syscall number 89
output from hook_function: syscall number 6
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 4
output from hook_function: syscall number 6
output from hook_function: syscall number 6
output from hook_function: syscall number 6
output from hook_function: syscall number 89
output from hook_function: syscall number 6
output from hook_function: syscall number 89
output from hook_function: syscall number 6
output from hook_function: syscall number 21
output from hook_function: syscall number 79
output from hook_function: syscall number 6
output from hook_function: syscall number 79
output from hook_function: syscall number 6
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 4
output from hook_function: syscall number 21
output from hook_function: syscall number 21
output from hook_function: syscall number 39
output from hook_function: syscall number 257
output from hook_function: syscall number 3
output from hook_function: syscall number 4
output from hook_function: syscall number 21
output from hook_function: syscall number 293
output from hook_function: syscall number 58
output from hook_function: syscall number 3
output from hook_function: syscall number 59
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 257
output from hook_function: syscall number 1
gcc: main.c:108: is_replaced_instruction_addr: Assertion `addr < (1UL << 48)' failed.
output from hook_function: syscall number 9
output from hook_function: syscall number 14
output from hook_function: syscall number 14
output from hook_function: syscall number 39
output from hook_function: syscall number 186
output from hook_function: syscall number 234
output from hook_function: syscall number 14
[1]    2580381 abort (core dumped)  LIBZPHOOK=./apps/basic/libzphook_basic.so LD_PRELOAD=./libzpoline.so gcc  -o

NO.3

I compiled a.out (the same as above), libzphook_basic.so and libzpoline.so in the above envirionment (with glibc 2.38, however).

Then I made a try on another environment using QEMU: KERNEL: Linux 6.6.4 GLIBC: glibc 2.38 LIBOPCODES: libopcodes 2.34

A segment fault occurs and the syscall numbers are wrong:

# LIBZPHOOK=./libzphook_basic.so LD_PRELOAD=./libzpoline.so ./a.out
output from __hook_init: we can do some init work here
output from hook_function: syscall number 262
output from hook_function: syscall number 16
[   56.789620] a.out[144]: segfault at 1400b7f02814 ip 00007fe99b26d913 sp 00007ffeb7f02808 error 6 in libc.so.6[7fe99b1ce000+105000] likely on CPU 0 (core 0, socket 0)
[   56.791876] Code: d0 48 3d 00 f0 ff ff 76 11 48 8b 15 0f 85 0b 00 f7 d8 64 89 02 83 ca ff eb 4c 89 c2 85 c0 75 46 48 8b 44 24 d4 0f 10 44 24 e5 <49> 89 00 48 8b 44 24 dc 41 0f 11 40 11 49 89 40 08 8a 44 24 e4 41
Segmentation fault

yasukata commented 8 months ago

Thank you for your report.

NO.1

Regarding NO.1, I think this behavior is fine; essentially, the system call numbers are printed only after the hook function implemented in apps/basic/main.c is activated.

To see this, when I run the program with strace by the following command,

strace --env LIBZPHOOK=./apps/basic/libzphook_basic.so --env LD_PRELOAD=./libzpoline.so ./a.out

I have the following output.

...
mprotect(0x5f92da000000, 16384, PROT_READ) = 0
mprotect(0x7f92da3a0000, 4096, PROT_READ) = 0
munmap(0x5f92da00a000, 127993)          = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5f92d9d18000
write(1, "output from __hook_init: we can "..., 55output from __hook_init: we can do some init work here
) = 55
write(1, "output from hook_function: sysca"..., 44output from hook_function: syscall number 5
) = 44
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(1, "output from hook_function: sysca"..., 44output from hook_function: syscall number 1
) = 44
write(1, "Hello, world!\n", 14Hello, world!
)         = 14
write(1, "output from hook_function: sysca"..., 46output from hook_function: syscall number 231
) = 46
exit_group(0)                           = ?
+++ exited with 0 +++

Up to the line printing we can do some init work here implemented in __hook_init of apps/basic/main.c, we do not see output from hook_function: syscall number because the hook function (hook_function in apps/basic/main.c) is not activated yet, and after its activation by *((syscall_fn_t *) sys_call_hook_ptr) = hook_function; in __hook_init, the hook function prints the system call numbers of subsequently invoked system calls (fstat : 5, write : 1, exit_group : 231).

I think this is the reason why the hook function only prints system call numbers 5, 1, and 231 for the hello world program.

NO.2

I also have an error when I run gcc. Currently, I do not have an idea why this happens, so, please give me time to look into more details.

NO.3

Regarding NO.3, I could not have reproduced this yet; for further investigation, could you provide the following information?

please let me confirm first; are these three (i, ii, iii) correct?
1. a.out, libzphook_basic.so, and libzpoline.so are compiled on the bare-metal machine having KERNEL: 5.4.0-148-generic (Ubuntu 20.04), CPU: Intel(R) Xeon(R) CPU E7- 4807 @ 1.87GHz, LIBC: glibc 2.31, CC: gcc 9.4.0, LIBOPCODES: libopcodes 2.34.
2. the generated files (a.out, libzphook_basic.so, and libzpoline.so) are copied to a QEMU-based VM (by for example scp and so on).
3. the QEMU process runs on the same bare-metal machine.
what is the command used for launching the QEMU process? (e.g., qemu-system-x86_64 -enable-kvm ...) specifically, I am interested in what is specified for the -cpu option.
how is the OS environment for the QEMU-based VM prepared? (e.g., by using debootstrap on the bare-metal machine described in the Environment section, or OS image installer provided by the official Ubuntu distributor) particularly, I would like to know 1) how are the library files, including libc and libopcodes, obtained and 2) which OS distribution is installed.
what is the commit id of this repository you have? I expect it is either 4c9362d2fffb85f8323be66d70361dbffe922c3d or 0a349e65c102f8f9bdbbf6da0a52c4006589178b , and I guess the change made in 4c9362d2fffb85f8323be66d70361dbffe922c3d would affect this issue.

The following questions are optional; if it is not too much trouble, I would appreciate it if you try (any of) them.

currently, both Makefile and apps/basic/Makefile have -O3; if you change -O3 to -O0 for the two Makefiles, does the error remain?
if you specify -cpu host for the QEMU command so that the vCPU will have the features of the underlying physical CPU, do you see the same error?
if you use version 4c9362d2fffb85f8323be66d70361dbffe922c3d of this repository, how is the output when you try 0a349e65c102f8f9bdbbf6da0a52c4006589178b ?
if you use version 0a349e65c102f8f9bdbbf6da0a52c4006589178b of this repository, how is the output when you try 4c9362d2fffb85f8323be66d70361dbffe922c3d ? (this is opposite to the above)

Thank you very much for reporting the issues.

lsc2001 commented 8 months ago

Thank you for your reply.

NO.1

It's my fault. I thought that two brks were missing at first, but actually they have been called before zpoline was activated.

NO.3

After using 4c9362d, it doesn't trigger segment fault for a.out any more.

(i, ii, iii) are correct.
I use this qemu command: qemu-system-x86_64 -smp 4 -m 1024M -kernel $(LINUX)/build/arch/x86_64/boot/bzImage -hda $(ROOTFS) -append "root=/dev/sda rw console=ttyS0" -nographic, I didn't specify -cpu but -smp.
It is not a well prepared OS envrionment. I use Linux-6.6.4 and glibc-2.38 compiled by myself and rootfs compiled by buildroot. And I copied libopcodes from host Ubuntu-22.04 into the rootfs.
I used 0a349e6. I tried 4c9362d and it works.

yasukata commented 8 months ago

Thank you very much for spending your time to have the information that I have requested.

NO.3

After using 4c9362d, it doesn't trigger segment fault for a.out any more.

(i, ii, iii) are correct.

I use this qemu command: qemu-system-x86_64 -smp 4 -m 1024M -kernel $(LINUX)/build/arch/x86_64/boot/bzImage -hda $(ROOTFS) -append "root=/dev/sda rw console=ttyS0" -nographic, I didn't specify -cpu but -smp.

It is not a well prepared OS envrionment. I use Linux-6.6.4 and glibc-2.38 compiled by myself and rootfs compiled by buildroot. And I copied libopcodes from host Ubuntu-22.04 into the rootfs.

I used 0a349e6. I tried 4c9362d and it works.

According to these, I guess that the reason for the issue of NO.3 is the red zone ( https://github.com/yasukata/zpoline/issues/9 ), and a temporary workaround added in 4c9362d2fffb85f8323be66d70361dbffe922c3d seemed to work this time.

Regarding NO.2, I have not figured out the root cause of it yet; please give me a bit more time.

Thank you very much for trying the implementation and reporting the issues.

yasukata / zpoline

Some bugs when using apps/basic hook #12

Environment

NO.1

NO.2

NO.3

NO.1

NO.2

NO.3

NO.1

NO.3

NO.3