Open corbinlc opened 3 years ago
Here is what the error looks like to the user:
say you are in a director that has two files a.txt
and b.txt
and you run ls
me@localhost:~$ ls
ls: cannot access 'a.txt': No such file or directory
ls: cannot access 'b.txt': No such file or directory
a.txt
b.txt
So it says none of them exist but then prints them out anyway. Doing cat a.txt
or other things that use the files work normally. Trying to ls
anything within a directory that is part of a binding returns the errors but doesn't print them out afterwards.
Most probably this is related to statx(2)
syscall, I've confirmed that old version doesn't use it while new does, although so far I haven't looked further as issue didn't reproduce on my phone.
(Some commits related to statx
are referenced in #122)
I saw those commit when looking through how stale UserLAnd proot was yesterday. I don't see statx being called though, but I should catch up. Here is the interesting part of the proot log for the failing and passing cases (this is running ls
on my home directory before and after apt upgrade
).
Failing:
proot info: vpid 20: sysenter start: openat(0xffffffffffffff9c, 0x300003b730, 0x84800, 0x0, 0x300001ae82, 0x300003b732) = 0xffffffffffffff9c [0x7ff45c3eb0, 0]
proot info: vpid 20: translate("/home/corbin" + ".")
proot info: vpid 20: -> "/data/user/0/tech.ula/files/1/home/corbin/."
proot info: vpid 20: sysenter end: openat(0xffffffffffffff9c, 0x7ff45c3e84, 0x84800, 0x0, 0x300001ae82, 0x300003b732) = 0xffffffffffffff9c [0x7ff45c3eb0, 0]
proot info: vpid 20: sysenter start: fstat(0x3, 0x7ff45c3e88, 0x7ff45c3e88, 0x3, 0x300001ae82, 0x300003b732) = 0x3 [0x7ff45c3e60, 0]
proot info: vpid 20: sysenter end: fstat(0x3, 0x7ff45c3e88, 0x7ff45c3e88, 0x3, 0x300001ae82, 0x300003b732) = 0x3 [0x7ff45c3e60, 0]
proot info: vpid 20: sysexit start: fstat(0x0, 0x7ff45c3e88, 0x7ff45c3e88, 0x3, 0x300001ae82, 0x300003b732) = 0x0 [0x7ff45c3e60, 0]
proot info: vpid 20: sysexit end: fstat(0x0, 0x7ff45c3e88, 0x7ff45c3e88, 0x3, 0x300001ae82, 0x300003b732) = 0x0 [0x7ff45c3e60, 0]
proot info: vpid 20: sysenter start: getdents64(0x3, 0x300003b780, 0x8000, 0x7fffffff, 0x4, 0x3) = 0x3 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysenter end: getdents64(0x3, 0x300003b780, 0x8000, 0x7fffffff, 0x4, 0x3) = 0x3 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysexit start: getdents64(0xd8, 0x300003b780, 0x8000, 0x7fffffff, 0x4, 0x3) = 0x1c0 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysexit end: getdents64(0xd8, 0x300003b780, 0x8000, 0x7fffffff, 0x4, 0x3) = 0x158 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysenter start: getdents64(0x3, 0x300003b780, 0x8000, 0x7fffffff, 0x300003b82e, 0x300004379b) = 0x3 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysenter end: getdents64(0x3, 0x300003b780, 0x8000, 0x7fffffff, 0x300003b82e, 0x300004379b) = 0x3 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysexit start: getdents64(0x0, 0x300003b780, 0x8000, 0x7fffffff, 0x300003b82e, 0x300004379b) = 0x0 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysexit end: getdents64(0x0, 0x300003b780, 0x8000, 0x7fffffff, 0x300003b82e, 0x300004379b) = 0x0 [0x7ff45c3ec0, 0]
proot info: vpid 20: sysenter start: fstat(0x1, 0x7ff45c1ce8, 0x7ff45c1ce8, 0x1, 0x7afc70f944, 0x8080808080800000) = 0x1 [0x7ff45c1cb0, 0]
proot info: vpid 20: sysenter end: fstat(0x1, 0x7ff45c1ce8, 0x7ff45c1ce8, 0x1, 0x7afc70f944, 0x8080808080800000) = 0x1 [0x7ff45c1cb0, 0]
proot info: vpid 20: sysexit start: fstat(0x0, 0x7ff45c1ce8, 0x7ff45c1ce8, 0x1, 0x7afc70f944, 0x8080808080800000) = 0x0 [0x7ff45c1cb0, 0]
proot info: vpid 20: sysexit end: fstat(0x0, 0x7ff45c1ce8, 0x7ff45c1ce8, 0x1, 0x7afc70f944, 0x8080808080800000) = 0x0 [0x7ff45c1cb0, 0]
proot info: vpid 20: exited with status 1
Passing:
proot info: vpid 20: sysenter start: openat(0xffffffffffffff9c, 0x300003a6f0, 0x84800, 0x0, 0x300001ab42, 0x300003a6f2) = 0xffffffffffffff9c [0x7ffaa6bc80, 0]
proot info: vpid 20: translate("/home/corbin" + ".")
proot info: vpid 20: -> "/data/user/0/tech.ula/files/2/home/corbin/."
proot info: vpid 20: sysenter end: openat(0xffffffffffffff9c, 0x7ffaa6bc54, 0x84800, 0x0, 0x300001ab42, 0x300003a6f2) = 0xffffffffffffff9c [0x7ffaa6bc80, 0]
proot info: vpid 20: sysenter start: fstat(0x3, 0x7ffaa6bc58, 0x7ffaa6bc58, 0x3, 0x300001ab42, 0x300003a6f2) = 0x3 [0x7ffaa6bc30, 0]
proot info: vpid 20: sysenter end: fstat(0x3, 0x7ffaa6bc58, 0x7ffaa6bc58, 0x3, 0x300001ab42, 0x300003a6f2) = 0x3 [0x7ffaa6bc30, 0]
proot info: vpid 20: sysexit start: fstat(0x0, 0x7ffaa6bc58, 0x7ffaa6bc58, 0x3, 0x300001ab42, 0x300003a6f2) = 0x0 [0x7ffaa6bc30, 0]
proot info: vpid 20: sysexit end: fstat(0x0, 0x7ffaa6bc58, 0x7ffaa6bc58, 0x3, 0x300001ab42, 0x300003a6f2) = 0x0 [0x7ffaa6bc30, 0]
proot info: vpid 20: sysenter start: getdents64(0x3, 0x300003a740, 0x8000, 0x7fffffff, 0x4, 0x3) = 0x3 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysenter end: getdents64(0x3, 0x300003a740, 0x8000, 0x7fffffff, 0x4, 0x3) = 0x3 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysexit start: getdents64(0xd8, 0x300003a740, 0x8000, 0x7fffffff, 0x4, 0x3) = 0xd8 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysexit end: getdents64(0xd8, 0x300003a740, 0x8000, 0x7fffffff, 0x4, 0x3) = 0xd8 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysenter start: fstatat64(0xffffffffffffff9c, 0x7ffaa6b930, 0x3000035ee8, 0x100, 0x8, 0x0) = 0xffffffffffffff9c [0x7ffaa6b930, 0]
proot info: vpid 20: translate("/home/corbin" + "index.html")
proot info: vpid 20: -> "/data/user/0/tech.ula/files/2/home/corbin/index.html"
proot info: vpid 20: sysenter end: fstatat64(0xffffffffffffff9c, 0x7ffaa6b8fb, 0x3000035ee8, 0x100, 0x8, 0x0) = 0xffffffffffffff9c [0x7ffaa6b8fb, 0]
proot info: vpid 20: sysexit start: fstatat64(0x0, 0x7ffaa6b8fb, 0x3000035ee8, 0x100, 0x8, 0x0) = 0x0 [0x7ffaa6b8fb, 0]
proot info: vpid 20: sysexit end: fstatat64(0x0, 0x7ffaa6b930, 0x3000035ee8, 0x100, 0x8, 0x0) = 0x0 [0x7ffaa6b930, 0]
proot info: vpid 20: sysenter start: getdents64(0x3, 0x300003a740, 0x8000, 0x7fffffff, 0x300003a7ee, 0x300004275b) = 0x3 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysenter end: getdents64(0x3, 0x300003a740, 0x8000, 0x7fffffff, 0x300003a7ee, 0x300004275b) = 0x3 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysexit start: getdents64(0x0, 0x300003a740, 0x8000, 0x7fffffff, 0x300003a7ee, 0x300004275b) = 0x0 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysexit end: getdents64(0x0, 0x300003a740, 0x8000, 0x7fffffff, 0x300003a7ee, 0x300004275b) = 0x0 [0x7ffaa6bc90, 0]
proot info: vpid 20: sysenter start: fstat(0x1, 0x7ffaa69728, 0x7ffaa69728, 0x1, 0x7a8062f944, 0xa) = 0x1 [0x7ffaa696f0, 0]
proot info: vpid 20: sysenter end: fstat(0x1, 0x7ffaa69728, 0x7ffaa69728, 0x1, 0x7a8062f944, 0xa) = 0x1 [0x7ffaa696f0, 0]
proot info: vpid 20: sysexit start: fstat(0x0, 0x7ffaa69728, 0x7ffaa69728, 0x1, 0x7a8062f944, 0xa) = 0x0 [0x7ffaa696f0, 0]
proot info: vpid 20: sysexit end: fstat(0x0, 0x7ffaa69728, 0x7ffaa69728, 0x1, 0x7a8062f944, 0xa) = 0x0 [0x7ffaa696f0, 0]
proot info: vpid 20: exited with status 0
I am wondering if they compiled coreutils with READDIR_LIES_ABOUT_MOUNTPOINT_D_INO
. ls
will fail if the ino
returned by readir/getdents
doesn't match the real ino
in that case. I will try catching up on proot commits, need to re-merge anyway at some point. Your way of handling shmget
and related to probably better than what I put in a while back. I will also check if the ino
matching does in fact matter.
That is not it. The inodes returned are the same as what you get from calling stat. We running with the hidden_files
extension, but the issue shows up when we do or do not use that extension.
Clearly something statx
-related, I've LD_PRELOAD
'ed following into ls
#define _GNU_SOURCE
#include <stddef.h>
#include <stdio.h>
#include <linux/stat.h>
#include <fcntl.h>
#include <dlfcn.h>
#include <errno.h>
static int(*real_statx)(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf);
int statx(int dirfd, const char *pathname, int flags, unsigned int mask, struct statx *statxbuf) {
int saved_errno;
int ret;
char buf[100];
// Find real statx() from libc
if (real_statx == NULL) {
real_statx = dlsym(RTLD_NEXT, "statx");
}
// Note before call
saved_errno = errno;
snprintf(buf, sizeof(buf), "/aaa_before_errno=%d", ret, errno);
open(buf, O_RDONLY);
errno = saved_errno;
// Call real function
ret = real_statx(dirfd, pathname, flags, mask, statxbuf);
// Note after call
saved_errno = errno;
snprintf(buf, sizeof(buf), "/aaa_after_ret=%d_errno=%d", ret, errno);
open(buf, O_RDONLY);
errno = saved_errno;
return ret;
}
and have got
proot info: vpid 30: sysenter start: openat(0xffffffffffffff9c, 0x7fefdc2230, 0x0, 0x0, 0x40100401, 0x5550000055100000) = 0xffffffffffffff9c [0x7fefdc2180, 0]
proot info: vpid 30: translate("/" + "/aaa_before_errno=48")
proot info: vpid 30: -> "/data/data/tech.ula/files/1/aaa_before_errno=48"
proot info: vpid 30: sysenter end: openat(0xffffffffffffff9c, 0x7fefdc2150, 0x0, 0x0, 0x40100401, 0x5550000055100000) = 0xffffffffffffff9c [0x7fefdc2180, 0]
proot info: vpid 30: seccomp SIGSYS: void(0xffffffffffffff9c, 0x7fefdc23e0, 0x100, 0x2, 0x7fefdc22d8, 0x7aa5f045f0) = 0xffffffffffffff9c [0x7fefdc2200, 0]
proot info: SIGSYS PR_void handled, but not for SYSCALL_AVOIDER. Falling through to default handling
proot info: SIGSYS. Return set to -ENOSYS
proot info: vpid 30: sysenter start: fstatat64(0xffffffffffffff9c, 0x7fefdc23e0, 0x7fefdc2078, 0x100, 0x100, 0xffffff9c) = 0xffffffffffffff9c [0x7fefdc2050, 0]
proot info: vpid 30: translate("/home/user" + "old_ls")
proot info: vpid 30: -> "/data/data/tech.ula/files/1/home/user/old_ls"
proot info: vpid 30: sysenter end: fstatat64(0xffffffffffffff9c, 0x7fefdc2023, 0x7fefdc2078, 0x100, 0x100, 0xffffff9c) = 0xffffffffffffff9c [0x7fefdc2023, 0]
proot info: vpid 30: sysexit start: fstatat64(0x0, 0x7fefdc2023, 0x7fefdc2078, 0x100, 0x100, 0xffffff9c) = 0x0 [0x7fefdc2023, 0]
proot info: vpid 30: sysexit end: fstatat64(0x0, 0x7fefdc23e0, 0x7fefdc2078, 0x100, 0x100, 0xffffff9c) = 0x0 [0x7fefdc2050, 0]
proot info: vpid 30: sysenter start: openat(0xffffffffffffff9c, 0x7fefdc2230, 0x0, 0x0, 0x40100401, 0x51000040000000) = 0xffffffffffffff9c [0x7fefdc2180, 0]
proot info: vpid 30: translate("/" + "/aaa_after_ret=0_errno=38")
proot info: vpid 30: -> "/data/data/tech.ula/files/1/aaa_after_ret=0_errno=38"
proot info: vpid 30: sysenter end: openat(0xffffffffffffff9c, 0x7fefdc214b, 0x0, 0x0, 0x40100401, 0x51000040000000) = 0xffffffffffffff9c [0x7fefdc2180, 0]
"seccomp SIGSYS" here was for statx
, which here triggered errno
to be set to 38
and therefore triggered fallback in glibc.
glibc statx
source is here, __NR_statx
was defined, __ASSUME_STATX
wasn't, checked on libc6/kali-rolling,now 2.31-9 arm64 [installed]
.
PR_void
suggests that statx
syscall is not even registered in proot's syscall table, so chances are that this issue is already fixed in Termux fork (although you'd need to check that, either way if you'd get this to reproduce in Termux that it is easier for me to debug)
For statx
there are multiple combinations to support (depending on Android and Linux version there are following options, I think each of them is present on some real devices):
SIGTRAP|0x80
)SIGTRAP|0x80
)I can confirm this is already fixed in the latest code base. I have a little more work to merge, but it does look like this issue is solved.
Btw... Thanks for looking into this.
@michalbednarski I finished up the merge ( https://github.com/CypherpunkArmory/proot/tree/merge-it ). I mostly went with what you had, except for a couple of differences related to specific bugs users encountered. It looks to be mostly working but one issue popped out on testing.
I previously had a incomplete solution for handling shmget/shmat and a few other things, probably the most substantial addition to UserLAnd proot since last merge back to termux (you might find it interesting: https://github.com/CypherpunkArmory/proot/blob/master/src/extension/fake_id0/shm.c ). I never merged it back because there were some bugs I never had time to debug them and I had to mostly stop work on UserLAnd (I think I forgot to close one file handle, because people were reporting running out of file handles after quite a bit of time using applications that use shared memory). Your solution sounds more robust and certainly handles more system calls, so I merged that in and deleted mine.
I am getting one error from sysvipc_shm.c currently when running pg_createcluster 13 main --start
after sudo apt install postgresql
. Specifically this seems to be failing:
if (request->op == SHMHELPER_ALLOC) {
int fd = -1;
read(helper2proot, &fd, sizeof(fd)); <-- this returns a negative fd
return fd;
}
I haven't analyzed what all this extension does yet, just the intent and where the error pops up. What do you think I should look at next?
Note: The interesting differences are that I still build off of the android-5
branch ( https://github.com/CypherpunkArmory/UserLAnd-Assets-Support/blob/docker-build/input/main.sh ) and use a variety of environment variables and bindings when calling proot
( https://github.com/CypherpunkArmory/UserLAnd-Assets-Support/blob/staging/assets/all/execInProot.sh ).
Any thoughts would be appreciated.
Any thoughts on this @michalbednarski ? Maybe I will just merge the statx part of this right now, but I do want to get all merged up in the near future. Thanks!
So far I've checked that building and running package in Termux from your branch works correctly.
So far I haven't looked further but few things that might help debugging further:
ashmem
through SCM_RIGHTS
. One difference is that your implementations runs accepting unix socket connections from thread while I have this running in separate process.proot --shm-helper
process will listen on (non-abstract) unix socket listening at create_temp_name(NULL, "prootshm")
, make sure that get_temp_directory()
is writeableproot --shm-helper
uses unix socket only to send ashmem fd to tracees, communication with proot and selection of fd to send is done through stdin/stdoutif (fd < 0) return -5;
doesn't make difference between failed read
and read filling fd
buffer with minus one, that is worth checkingBefore bringing this up, I did confirm that the prootshm file was getting created. I will work through the other suggestions and see where I end up. Thank you!
@michalbednarski I finally got back to debugging this. This is related to UserLAnd targeting Android 10. See this behavior change:
Shared memory
Ashmem has changed the format of dalvik maps in /proc/<pid>/maps, affecting apps that directly parse the maps file. Application developers should test the /proc/<pid>/maps format on devices that run Android 10 or higher and parse accordingly if the app depends on dalvik map formats.
Apps targeting Android 10 cannot directly use ashmem (/dev/ashmem) and must instead access shared memory via the NDK’s ASharedMemory class. In addition, apps cannot make direct IOCTLs to existing ashmem file descriptors and must instead use either the NDK’s ASharedMemory class or the Android Java APIs for creating shared memory regions. This change increases security and robustness when working with shared memory, improving performance and security of Android overall.
So when the shm helper gets to trying to allocated memory it is failing to open /dev/ashmem. Any thoughts on this? I am going to look at how easy it is to use the mentioned class.
In the same branch I have been playing with ( https://github.com/CypherpunkArmory/proot/tree/merge-it ) it looks to be mostly working now, but very little testing so far. Had to do the following, include <android\sharedmem.h>
, switch to using ASharedMemory_create()
, link with -landroid
and make it so the build targeted API level 26 or newer. That last one will require a different build for earlier versions of android which don't have ASharedMemory_create()
. Going to have to see how to improve upon that and do a bit of testing.
@michalbednarski I am not sure yet if this is a proot issue, but I am currently thinking it is not. Regardless, I wanted your thoughts on this matter, because it does seem like the type of problem you often get to the bottom of. In UserLAnd, which uses PRoot (you know this, but writing that for anyone else reading) Kali Linux works fine currently, though the image installed is a bit stale. If after you run
sudo apt upgrade
to catch up to current, thels
command starts returning the normal list of files, but then prints out that each one does not exist. The image is based on kali-rolling which pick some pretty recent packages into its build, so I am a little bothered that this might start showing up on other prooted images running on Android in the future. I will drop some logs related to this on this issue for your review, but I am also wondering if you have any other thoughts on what could be going on.