ltp: fcntl34 - Githubissues

Martins3 commented 3 years ago

Sometimes, it will crash.

[x] understand how stack works. That's right, we can't understand what we are writing.

Martins3 commented 3 years ago

➜  test git:(main) ✗ ./hello
  Info: top = fffbef8000, limit = 800000
  Info: open /dev/kvm
  Info: KVM_GET_API_VERSION
  Info: KVM_CREATE_VM
  Info: KVM_SET_USER_MEMORY_REGION
  Info: ebase address : fff519c000
Parent: PID=22901 PPID=16496
child new stack 0xfff5194000 in guest_clone

 Registers:
 ----------
$0   : 0000000000000000 fffffffffffffff0 00000000000013bf 000000fffbef68e4
$4   : 0000000000000112 000000fff5193ff0 0000000000002000 0000000000000031
$8   : 0000000000000035 0000000000000031 0000000000000035 000000000000000a
$12  : 000000006c696863 0000000120017830 ffffffffffffffff ffffffff81440000
$16  : 0000000000005975 0000000000000001 000000fffbef6978 0000000120001c44
$20  : 000000fff492d438 00000001201081d0 0000000000000000 000000fff492d480
$24  : 000000000000000c 000000012000ad10 980000fff51a9848 0000000000000000
$28  : 0000000120032890 000000fffbef6880 000000fffbef68c0 0000000120006708
hi   : 0000000000000000
lo   : 0000000000000000
pc  : 980000fff519c1cc

fuck
Child:  PID=22904 PPID=22901
hello, this is a child process
    Child PID=22904

The stack top is 0xfff5194000 and syscall parameter is 000000fff5193ff0, it's ok.

Martins3 commented 3 years ago

tst_test.c:1263: TINFO: Timeout per run is 0h 05m 00s
  Info: top = fffbddc000, limit = 800000
  Info: open /dev/kvm
  Info: KVM_GET_API_VERSION
  Info: KVM_CREATE_VM
  Info: KVM_SET_USER_MEMORY_REGION
  Info: ebase address : fff7c7c000
fcntl34.c:91: TINFO: write to a file inside threads with OFD locks
fcntl34.c:37: TINFO: spawning '15' threads
stack : fff7a06ae0
stack : fff71feae0
stack : fff69f2ae0
stack : fff61eaae0
stack : fff59e2ae0
stack : fff51daae0
stack : fff49ceae0
stack : fff41c2ae0
stack : fff39baae0
stack : fff31aeae0
stack : fff29a6ae0
stack : fff219eae0
stack : fff1996ae0
stack : fff118eae0
stack : fff0986ae0
fcntl34.c:46: TINFO: waiting for '15' threads
  Fatal: KVM_EXIT_IS_NOT_HYPERCALL vcpu=8 exit_reason=6

[357142.945414] kvm [25968]: TLB LD fault: cause 0x10800008, PC: 00000000a13e0b69, BadVaddr: 0xfff7a076b8
[357142.945420] kvm [25968]: Failed to find VMA for hva 0xfff7a04000
[357142.945424] kvm [25968]: update_pc(): New PC: 0xfff7b9ce60
[357142.945429] kvm [25968]: kvm_mips_handle_exit : cause register = 40008020 exit_reason=6
[357142.945431] kvm [25968]: huxueshi trace leave kvm_vz_vcpu_run:3586 cause=40008020

Martins3 commented 3 years ago

/usr/local/musl/bin/musl-gcc -g -pthread -I../dune  pthread.c ../dune/libdune.a -o pthread
  Info: top = fffbd28000, limit = 800000
  Info: open /dev/kvm
  Info: KVM_GET_API_VERSION
  Info: KVM_CREATE_VM
  Info: KVM_SET_USER_MEMORY_REGION
  Info: ebase address : fff70dc000
pthread join never works 0
just before pthread join fff7017b20
0 stack = 0xfff7017a80

vcpu=0 sysno=5014:  2 fffbd26c18 0 10 fff70ee180 0
vcpu=0 sysno=5318:  10 0 0 14c6 fff70ee180 0
vcpu=0 sysno=5009:  0 28000 0 802 ffffffffffffffff 0
vcpu=0 sysno=5010:  fff6ff4000 24000 3 3fff ffffffffffffc000 0
vcpu=0 sysno=5014:  1 fff70d8460 fffbd26b60 10 ffffffffffffc000 0
vcpu=0 sysno=5055:  7d0f00 fff7017ae0 fff7017b40 fff701ebe8 fff70ed1dc fff701ebe8
vcpu=0 sysno=5014:  3 fffbd26b60 0 10 fff70ed1dc fff701ebe8
vcpu=1 sysno=5014:  3 fff7017b10 0 10 fff70ed1dc fff701ebe8
vcpu=0 sysno=5019:  1 fffbd26980 2 fff70f3670 2000 0
vcpu=1 sysno=5034:  fff7017a60 fff7017a60 0 0 0 0
vcpu=0 sysno=5019:  1 fffbd26940 2 fff70f3670 2000 0
vcpu=0 sysno=5194:  fff7017b48 80 2 0 0 0
vcpu=1 sysno=5019:  1 fff7017700 2 fff70f3670 2000 30
vcpu=1 sysno=5014:  1 fff70d8460 fff70179f0 10 fff70eb5bc 30
vcpu=1 sysno=5194:  fff70efa48 81 1 1 144a fff70f3670
vcpu=1 sysno=5058:  0 0 1 13c2 fff709620c fff70f3670

[x] is it abnormal to futex on a different address ? Yes

parent locked on fff7017b48, but child unlock it at fff70efa48

Martins3 commented 3 years ago

➜  test git:(main) ✗ strace -f ./pthread
execve("./pthread", ["./pthread"], [/* 50 vars */]) = 0
set_thread_area(0xfff5e0aae8)           = 0
set_tid_address(0xfff5e011dc)           = 1881
brk(0)                                  = 0x1252e4000
brk(0x1252ec000)                        = 0x1252ec000
mmap(0x1252e4000, 16384, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x1252e4000
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 16) = 0
syscall_5318(0x10, 0, 0, 0x14c6, 0xfff5e02180, 0) = 0
mmap(NULL, 163840, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfff5d08000
mprotect(0xfff5d0c000, 147456, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RT_0 RT_1 RT_2], [], 16) = 0
clone(Process 1882 attached
child_stack=0xfff5d2fae0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, p
arent_tidptr=0xfff5d2fb40, tls=0xfff5d36be8, child_tidptr=0xfff5e011dc) = 1882
[pid  1882] rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
[pid  1882] nanosleep({1, 0},  <unfinished ...>
[pid  1881] rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
[pid  1881] ioctl(1, 0x40087468, 0xfffbb2dd60) = 0
[pid  1881] writev(1, [{"pthread join never works 0", 26}, {"\n", 1}], 2pthread join never works 0
) = 27
[pid  1881] writev(1, [{"just before pthread join fff5d2f"..., 35}, {"\n", 1}], 2just before pthread join fff5d2fb20
) = 36
[pid  1881] futex(0xfff5d2fb48, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  1882] <... nanosleep resumed> 0xfff5d2fa60) = 0
[pid  1882] writev(1, [{"0 stack = 0xfff5d2fa80", 22}, {"\n", 1}], 20 stack = 0xfff5d2fa80
) = 23
[pid  1882] rt_sigprocmask(SIG_BLOCK, ~[RT_0 RT_1 RT_2], [], 16) = 0
[pid  1882] futex(0xfff5d2fb48, FUTEX_WAKE_PRIVATE, 1) = 1
[pid  1881] <... futex resumed> )       = 0
[pid  1882] exit(0)                     = ?
[pid  1882] +++ exited with 0 +++
munmap(0xfff5d08000, 163840)            = 0
writev(1, [{"pthread succeed", 15}, {"\n", 1}], 2pthread succeed
) = 16
writev(1, [{"all child returned", 18}, {"\n", 1}], 2all child returned
) = 19
writev(1, [{"OFD locks synchronized access be"..., 53}, {"\n", 1}], 2OFD locks synchronized access between threads succeed
) = 54
exit_group(0)                           = ?
+++ exited with 0 +++

Martins3 commented 3 years ago

I guess, in libc implementation, the child stack contains much more data than our expected, dune should write to them.

Martins3 commented 3 years ago

➜  test git:(main) ✗ make && ./pthread
make -C ../dune
make[1]: Entering directory '/home/loongson/dune/dune'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/loongson/dune/dune'
/usr/local/musl/bin/musl-gcc -g -pthread -I../dune  pthread.c ../dune/libdune.a -o pthread
  Info: top = fffb8a4000, limit = 800000
  Info: open /dev/kvm
  Info: KVM_GET_API_VERSION
  Info: KVM_CREATE_VM
  Info: KVM_SET_USER_MEMORY_REGION
  Info: ebase address : fff6fdc000
parent stack 0xfffb8a1478
0 fff6f17b20
pthread join never works 0
just before pthread join fff6f17b20
0 stack = 0xfff6f17a80
^C
➜  test git:(main) ✗ cat syscall.txt
vcpu=0 sysno=5019:  1 fffb8a10b0 2 fff6ff3670 2000 38
vcpu=0 sysno=5014:  2 fffb8a1308 0 10 fff6fee180 38
vcpu=0 sysno=5318:  10 0 0 14c6 fff6fee180 38
vcpu=0 sysno=5009:  0 28000 0 802 ffffffffffffffff 0
vcpu=0 sysno=5010:  fff6ef4000 24000 3 3fff ffffffffffffc000 0 // fff6ef2000 + 28000 = fff6f1a000
vcpu=0 sysno=5014:  1 fff6fd8460 fffb8a1250 10 ffffffffffffc000 0 b // first parameter is &app_mask, static library
vcpu=0 sysno=5055:  7d0f00 fff6f17ae0 fff6f17b40 fff6f1ebe8 fff6fed1dc fff6f1ebe8
vcpu=0 sysno=5014:  3 fffb8a1250 0 10 fff6fed1dc fff6f1ebe8 // restore is fine !
vcpu=0 sysno=5019:  1 fffb8a10b0 2 fff6ff3670 2000 66
vcpu=0 sysno=5019:  1 fffb8a1070 2 fff6ff3670 2000 0
vcpu=0 sysno=5019:  1 fffb8a1030 2 fff6ff3670 2000 0
vcpu=0 sysno=5194:  fff6f17b48 80 2 0 0 0
vcpu=1 sysno=5014:  3 fff6f17b10 0 10 fff6fed1dc fff6f1ebe8 // 
vcpu=1 sysno=5034:  fff6f17a60 fff6f17a60 0 0 0 0
vcpu=1 sysno=5019:  1 fff6f17700 2 fff6ff3670 2000 66
vcpu=1 sysno=5014:  1 fff6fd8460 fff6f179f0 10 fff6feb5bc 66 // ?????????
vcpu=1 sysno=5194:  fff6fefa48 81 1 1 144a fff6ff3670
vcpu=1 sysno=5058:  0 0 1 13c2 fff6f9620c fff6ff3670

Martins3 commented 3 years ago

I think the bug is related with tls.

[x] SYS_set_tid_address is never handled seriously
[x] in fork, we create the vcpu before simulated the clone, so the userlocal register parent's.

Martins3 commented 3 years ago

In fact, all the register, modified by host, but used by guest, should write to guest explicitly. My original assumption is that syscall's simulation result is memory modification, but in fact, it also changed some cp0 register.

I think Counter and Perf related is dune's job, it belongs to vz and kvm
SYNCI_step is similar. They are maintained by the hardware.
[ ] As for CPUnum, I trace related code in kernel, it seems not used by kernel.

MIPS Privilege Manual Page 200 Table 9.43

Martins3 / loongson-dune

ltp: fcntl34 #29