blitz branch seems broken, at least for aarch64 chroots.

[dfr]

In my test setup, I have binmisc entries setup without pre-open. I have a small arm64 chroot containing pkgbase's FreeBSD-runtime and supporting packages. I tested with both sysutils/qemu-user-static and sysutils/qemu-user-static-devel by copying qemu-aarch64-static into the chroot's /usr/local/bin and run chroot <dir> sh sucessfully. I built an emulator from blitz which does not contain my execve changes:

$ mkdir build
$ cd build && ../configure --prefix=/usr/local --cc=cc --localstatedir=/var --extra-ldflags=-L/usr/local/lib --extra-cflags=-I/usr/local/include --enable-debug --enable-debug-info --static --python=/usr/local/bin/python3.9 --disable-docs --disable-tools --disable-system --disable-capstone --extra-cflags='-I/usr/local/include -DPREFIX=\""/usr/local\""' --target-list=i386-bsd-user,arm-bsd-user,ppc-bsd-user,x86_64-bsd-user,riscv64-bsd-user,ppc64-bsd-user,aarch64-bsd-user
$ gmake -j16 
...

If I copy this emulator into my chroot, I get this:

# chroot $m sh
ld-elf.so.1: Shared object "libedit.so.8" not found, required by "sh"

poking around with ktrace shows that the emulator runs and appears to be executing the chroot's /libexec/ld-elf.so.1, opens /var/run/ld-elf.so.hints and then prints the above error message. Interestingly, it doesn't even try to open libedit.so.8.

Originally posted by @dfr in https://github.com/qemu-bsd-user/qemu-bsd-user/issues/39#issuecomment-1344155962

[dfr]

The libedit.so.8 not found error first starts appearing in git commit https://github.com/qemu-bsd-user/qemu-bsd-user/commit/0e0cb0540bdbbb3b8c831f7eb4441b5b8500784a

[dfr]

https://github.com/dfr/qemu-bsd-user/commit/83eb90235033fa76a0854da22bb3072be4bfb6fe fixes it but I'm not sure why. I split 0e0cb05 to separate the syscall changes and the iovec changes and reverted just the syscall parts.

[dfr]

A simple recipe for reproducing, assuming a fresh built set of emulators in $(pwd)/build:

# pkg install -y buildah
# c=$(buildah from quay.io/dougrabson/freebsd-minimal:13)
# m=$(buildah mount $c)
# cp build/aarch64-bsd-user/qemu-aarch64 $m/usr/local/bin/qemu-aarch64-static
# chroot $m sh -c 'echo Hello World'

This only uses buildah as a convenient means of pulling a pre-built small arm64 image - manually unpacking a suitable base.txz into a local directory would work just as well.

[bsdimp]

Yea, I do stuff like that all the time and have never ever seen this error.

[bsdimp]

I'll have to run the test suite, though to see if something else is showing up there.

[dfr]

Attached truss output for good and bad examples of my test syscalls-good.txt syscalls-bad.txt

The two traces diverge early in /bin/sh, just after the call to issetugid. The good trace searches for libmap.d and the bad trace skips that and opens libmap.conf. Both then search for and open ld-elf.so.hints and then the bad trace dies.

[dfr]

One interesting thing is that both traces open libmap.conf but only the good trace reads from it.

[dfr]

I see emulated read syscalls in the qemu trace but not with truss. In the emulator, they seem to return EFAULT:

IN:
0x00000040006d5ec0:  d2800068      mov x8, #0x3
0x00000040006d5ec4:  d4000001      svc #0x0

 PC=00000040006d5ec0 X00=0000000000000003 X01=0000000003244008
X02=000000000000002f X03=fffffff820c96c88 X04=0000000000000000
X05=0000000000000000 X06=0000000080808080 X07=fefefeff302d6e72
X08=0000000003244040 X09=0000000003244000 X10=00000000000000ef
X11=0000000003244fc0 X12=0000000003244fc0 X13=0000000000000000
X14=00000040006f8850 X15=00000040006f8520 X16=00000040006f8000
X17=0000000000000000 X18=00000040006f8000 X19=00000040006b16fc
X20=0000000003244008 X21=0000000000000003 X22=00000040006f8000
X23=0000000003242008 X24=00000040006f8000 X25=00000040006f8000
X26=00000040006f8000 X27=00000040006f8000 X28=00000040006f8000
X29=fffffff820c96e10 X30=00000040006d1b2c  SP=fffffff820c96d30
PSTATE=60000000 -ZC- EL0t  BTYPE=0
guest_user_syscall cpu=0x825d2f9c0 num=0x0000000000000003 arg1=0x0000000000000003 arg2=0x0000000003244008 arg3=0x000000000000002f arg4=0xfffffff820c96c88 arg5=0x0000000000000000 arg6=0x0000000000000000 arg7=0x0000000080808080 arg8=0xfefefeff302d6\
e72
guest_user_syscall_ret cpu=0x825d2f9c0 num=0x0000000000000003 ret=0xfffffffffffffff2

which seems to imply that lock_user is returning NULL

[dfr]

It turns out that an earlier call to mmap had its return value truncated - the result from freebsd_syscall was assigned to an 'int ret' variable. Changing that to abi_long seems to fix this problam at least but probably is unrelated to the random memory corruption problem.

[dfr]

Is there anything blocking merging #41? I'd like to revisit #39 so that the blitz branch will work with 'binmiscctl --pre-open'. I also plan to make a PR to back-port it to bsd-user-rebase-3.1 so that I can update the port.

[bsdimp]

Nothing is blocking it. I rebased it on the merge, then went in and edited the commit message a bit so I can more easily upstream it when the time comes (soon, I hope). I also plan on merging qemu-project's master now that 7.2 is out.

[dfr]

Great, thanks for taking a look. I will rebase #39 this week and also make a PR for bsd-user-rebase-3.1.

[dfr]

Fixed in #41