Tests fail on ppc64el

[ebroder]

The tty-steal test on ppc64el fails with the following logs:

Unable to find `prctl.set_ptracer`, skipping `PR_SET_PTRACER`.
spawned children: me=14648 victim=14649 reptyr=14650
world
[+] Making sure we have permission to attach...
[+] session leader of pid 14649 = 14649
[+] found terminal emulator process: 14648
[+] Listening on socket: /tmp/reptyr.VhiRVA/reptyr.sock
[+] Attaching terminal emulator pid=14648
[+] Allocated scratch page: 7fffacce0000
[+] Attached to terminal emulator (pid 14648)
[+] Checking fd: 0: st_dev=8800
[+] Checking fd: 1: st_dev=8800
[+] Checking fd: 2: st_dev=8800
[+] Checking fd: 5: st_dev=502
[+] found a ptmx fd: 5
[+] found a master fd: 5
[+] Checking fd: 6: st_dev=502
[+] found a ptmx fd: 6
[+] Opened fd 94 in the child.
[+] Connected to the shared socket.
[+] Sent the pty fd, going to receive it.
[-] Error receiving message.
Unable to attach to pid 14649: Resource temporarily unavailable
world
[...]

I've discussed this with @nelhage in person but since I have access to a ppc64el machine now, opening a ticket to track the debugging work. I'm currently trying to understand why the powerpc implementation of arch_set_syscall is using PTRACE_POKEUSER, since AIUI the syscall number goes in a register, not a memory address.

[ebroder]

OK that was a red herring - ptrace has interfaces through PEEKUSER and POKEUSER for interacting with registers (in addition to GETREGSET and SETREGSET.

I'm currently suspicious that the return value from a syscall ends up in the result field of struct pt_regs, not in gpr[3]. But if I make that change, I get a new, earlier error:

[+] Checking fd: 6: st_dev=502
[+] found a ptmx fd: 6
Unable to attach to pid 1954: Socket type not supported

which implies that socket(AF_UNIX, SOCK_DGRAM, 0) is failing with ESOCKTNOSUPPORT which makes no sense.

[nelhage]

I find straceing the reptyr process pretty helpful for debugging weird issues, since so much of what it does is ~directly at the syscall layer.

You can probably modify https://github.com/nelhage/reptyr/blob/e78f1f41603f159c42e52d483a3941d7e64f9f52/test/tty-steal.py#L33 to add strace -o reptyr.strace or something and then pore over that.

[ebroder]

Yeah I did try something similar earlier, but it doesn't let me see the registers that are being poked into place:

ptrace(PTRACE_GETREGSET, 20911, NT_PRSTATUS, [{iov_base=0x7fffe1e014d8, iov_len=352}]) = 0
ptrace(PTRACE_SETREGSET, 20911, NT_PRSTATUS, [{iov_base=0x7fffe1e014d8, iov_len=352}]) = 0
ptrace(PTRACE_SETREGSET, 20911, NT_PRSTATUS, [{iov_base=0x7fffe1e02848, iov_len=352}]) = 0
ptrace(PTRACE_POKEUSER, 20911, r0, 0x118) = 0
ptrace(PTRACE_DETACH, 20911, NULL, 0)   = 0
close(3)                                = 0
unlink("/tmp/reptyr.ugBArk/reptyr.sock") = 0
rmdir("/tmp/reptyr.ugBArk")             = 0
write(2, "Unable to attach to pid 20924: S"..., 57Unable to attach to pid 20924: Socket type not supported

Having thought about it more, though, I do think that the socket call is actually returning with ESOCKTNOSUPPORT. The victim process only has ~6 fds open, so this line from the original output stream does actually make no sense:

[+] Opened fd 94 in the child.

Still not sure why, though, or what arguments are actually getting fed into the syscall.

[ebroder]

Aha:

(gdb) p arch_syscall_numbers[0].nr_socket
$5 = 0

[ebroder]

powerpc seems to have both a socketcall syscall and the direct socket etc. syscalls, but because socketcall exists, it was going through this logic: https://github.com/nelhage/reptyr/blob/e78f1f41603f159c42e52d483a3941d7e64f9f52/platform/linux/linux.c#L415-L418

which, as promised, does not seem to work on 64-bit architectures. Swapping the precedents in default-syscalls.h (to prefer socket over socketcall unless __NR_socket is undefined) seems to work. I'll send a patch in a sec.