Open mu150 opened 3 years ago
...To remedy this install strace/gdb/etc from same stratum as the traced program and use 'strat' to specify appropriate strace/gdb/etc.
Maybe try installing strace and gdb in all strata and see if that works for that problem? (I don't know anything about pmOS) If that doesn't work, check that chroot works, I think that might be a potential culprit.
As for the other issue, are you sure bad internet isn't responsible for it? I thought that Fedora does checksums on packages, but if it doesn't, internet is defenitly a potential issue.
I think the strace/gdb is just an example of why the error would happen, but I'm not using those programs and installing them doesn't change anything. I'm pretty sure my internet is not the issue since I tried on multiple fast networks. I think the fetching works fine if release and mirror is specified, but the installation fails because bedrock can't install certain packages. I will upload some logs soon. It's odd because pmos is based on Alpine Linux which is listed as compatible.
Ok I have some logs, these are for issues with fetching and another issue I found when running sudo brl update
.
Also I was successful in fetching void, arch-arm, debian among others by using --mirror
, so the main issue with fetching is with centos and fedora, and both of these give very similar logs.
Here's a log for centos:
And issue with brl update:
@mn26 I wonder if that last one would work if you added a User-Agent to the wget? Generally like wget --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22"
.
Also for the first, problemo seems to be here:
error: unpacking of archive failed on file /usr/lib/.build-id/18/c636104f5b332bc96b2efed579e2716006c50f;5f7b7d22: cpio: symlink failed - Inappropriate ioctl for device
error: grub2-tools-1:2.02-87.el8_2.aarch64: install failed
EDIT: what's your CPU arch?
The cpu is a Cortex-A53 so it should be arm64/aarch64.
Actually if I use wget directly it saves the file just fine:
wget https://raw.githubusercontent.com/bedrocklinux/bedrocklinux-userland/0.7/releases
will save releases
file.
I've done some research and cap_sys_chroot
is a kernel capability. So it makes sense that the issue appeared in Mobian when using a different kernel. And since running as root has full permissions, it works when using sudo
.
Also in the error when fetching it says something about cap_set_file
failing:
error: unpacking of archive failed on file /usr/bin/newgidmap;5f7b7d22: cpio: cap_set_file failed - Inappropriate ioctl for device
so maybe both of these errors are related to capabilities.
I found this post: https://www.reddit.com/r/bedrocklinux/comments/av4cg9/strat_wrong_cap_sys_chroot_capability_when/
but running sudo /bedrock/libexec/setcap cap_sys_chroot=ep /bedrock/bin/strat
gives me a segfault.
This
I've always had this issue on pmos, however on Mobian I also got these errors when switching the kernel from 5.7 to 5.6. Switching the kernel back solved the issue.
combine with this:
but running
sudo /bedrock/libexec/setcap cap_sys_chroot=ep /bedrock/bin/strat
gives me a segfault.
pointed me in the right direction to find something. It looks like there is a bug in libcap (the library that setcap is built around) which causes it to segfault under certain situations with newer kernels. This was fixed in early May but the last stable Bedrock release was April 30th and just barely missed getting this fix in.
While I can't make any promises - I'm completely swamped with support requests and am having difficulty finding time for development - I'm going to try to push a beta update this weekend. I'll be sure this (and the eventually following stable release) includes the latest libcap fixes. Try the 0.7.18beta5 release once it's out and see if it fixes things.
In the mean time, if you can install a new enough version of setcap
from some other distro (e.g. Ubuntu's libcap2-bin), you can adjust the command you found:
sudo strat <new-setcap-providing-stratum> cap_sys_chroot=ep /bedrock/bin/strat
and it might fix the strat issue.
The might qualifier is because of the Inappropriate ioctl for device
errors you've mentioned. These make me suspect you're running a filesystem that does not support xattrs. Bedrock normally detects this before executing the hijack maneuver, but it uses setcap to do so; if setcap was segfaulting, the check might have failed. Bedrock 0.7 Poki strictly requires this filesystem feature. I wouldn't be surprised if your CentOS and Fedora errors are because they, too, require such things. What filesystem are you on? Can Postmarketos or Mobian easily utilize another one (e.g. ext4)?
brl update
uses Bedrock's busybox
distribution. I suspect when you run wget
directly you're using another distro's wget
, and the issue with brl update
(and possibly finding mirrors) is specific to Bedrock's busybox
's wget
. Not unlike with libcap, I see a new busybox release since Bedrock's last stable release, and this release does mention wget changes. While I have less confidence I'm on the right track here than I do with libcap, I'll update busybox in the next beta as well. Give it a try then.
Looking at df -Th
my partitions are all ext4, I think mobian and pmos use it default.
Unfortunately when I run sudo strat ubuntu cap_sys_chroot=ep /bedrock/bin/strat
after installing libcap2-bin I get this error:
strat: could not run
cap_sys_chroot=ep
from stratum
ubuntu
due to: unable to find file (ENOENT)
and doing sudo strat ubuntu setcap cap_sys_chroot=ep /bedrock/bin/strat
gives:
Failed to set capabilities on file '/bedrock/bin/strat' (Operation not supported)
The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file
I was also suspicious of busybox's wget, however since Alpine Linux also uses busybox and is reported to work I figured it was something else.
When I run brl which wget
it returns postmarketos
so wget isn't from another distro.
Anyway thanks for your response, it will be interesting to see if the beta release will fix anything.
Apologies, I the word setcap
; your second command was what I intended.
Failed to set capabilities on file '/bedrock/bin/strat' (Operation not supported) The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file
This continues to look like something about the corresponding components here are missing xattr support. Maybe it's compiled out of your kernel, or your filesystem is mounted with it as a disabled option?
Try
sudo mount -o remount,acl,user_xattr /
then assuming that doesn't error try
sudo strat ubuntu setcap cap_sys_chroot=ep /bedrock/bin/strat
again.
I'm able to run the mount command but it doesn't fix the error on the second command, it's the same error as before.
Also I ran these commands in Mobian to make sure they work, didn't get any errors there
When I run sudo cat /proc/fs/ext4/mmcblk0p2/options | grep xattr
I get: user_xattr
which means I should have xattr support
I just pushed 0.7.18beta5. While I suspect you'll still get operation not supported errors, it's worth a try if you have the time.
I tried updating using the downloaded script but I get error.
sudo ./bedrock-linux-0.7.18beta5-aarch64.sh --update
:
Failed to set capabilities on file `/bedrock/bin/strat` (Not supported)
usage: setcap [-q] [-v] [-n <rootid>] (-r|-|<caps) <filename> [ ... (-r|-|<capsN>) <filenameN> ]
Note <filename> must be a regular (non-symlink) file.
ERROR: Unexpected error occurred.
I willl just reinstall pmos and then hijack again.
If you're reinstalling, try playing around with setfattr
and getfattr
on pmos before hijacking. If they're not working, debugging for why should probably happen at that point without Bedrock in the mix. It might be something like CONFIG_EXT3_FS_XATTR
being disabled in the kernel.
We'll also want to check if 0.7.18beta5 now properly detects setfattr
/getfattr
don't work before actually executing the hijacking and aborting appropriately. I recognize that's at odds with my prior request; balance those conflicting interests per your own preference.
When I read your comment I had already hijacked, but I can just reinstall again. Unfortunately I'm getting the same errors as before. I did run setfattr
in the already hijacked os and got the cap_sys_chroot error (brl which setfattr
returned bedrock
). I did notice however that pmos does not install the attr
package by default which is needed for setfattr
and getfattr
. After doing your tests I will try hijacking again but with this package installed.
So after reinstall and installing attr
it seems getfattr
and setfattr
are working properly, I ran the first few test commands from this site:https://www.linux.com/news/cli-magic-use-extended-attributes-better-file-management/.
Interesting. In that case I guess we proceed to test with Bedrock.
I hijacked again, currently fetching ubuntu. Edit: I get the same errors as before.
Ok I think I figured it out, when I run zcat /proc/config.gz | grep 'XATTR'
I don't see any entry for CONFIG_EXT4_FS_XATTR
. So although the filesystem supports xattr, the kernel was compiled without xattr support for ext4. Looks like I can either compile a kernel with the proper configuration or wait and see if new kernels for allwinner cpu are compiled with support.
Also of note, sudo /bedrock/libexec/setcap cap_sys_chroot=ep /bedrock/bin/strat
no longer segfaults, instead gives me a Not supported
error, it looks like the beta update did fix something.
In fact setcap
and getcap
result in Not supported
error with any binary, I'm not sure why getfattr
and setfattr
work though.
Ok I think I figured it out, when I run
zcat /proc/config.gz | grep 'XATTR'
I don't see any entry forCONFIG_EXT4_FS_XATTR
. So although the filesystem supports xattr, the kernel was compiled without xattr support for ext4. Looks like I can either compile a kernel with the proper configuration or wait and see if new kernels for allwinner cpu are compiled with support.
For what it's worth, I don't have that config either:
$ zcat /proc/config.gz | grep 'XATTR' | grep -ic ext4
0
and this works for me. This might be a red herring.
Also of note,
sudo /bedrock/libexec/setcap cap_sys_chroot=ep /bedrock/bin/strat
no longer segfaults, instead gives me aNot supported
error, it looks like the beta update did fix something.
That's good.
In fact
setcap
andgetcap
result inNot supported
error with any binary, I'm not sure whygetfattr
andsetfattr
work though.
Since both setcap
/getcap
and getfattr
/setfattr
were failing previously, I was focused on how they were similar, which was that they both use xattrs. It seems I pushed us to look in the wrong direction. Now that getfattr
/setfattr
is working but setcap
/getcap
is failing, we should look at how they differ.
Sometimes unprivileged processes need to do something that requires root permissions. This is usually done by setting a setuid
bit on the corresponding executable. For example, if you look at permissions on your sudo
binary you've probably got an s
in there to allow for this. The program is expected to do its own permission checks to ensure non-root users can't abuse it to get more permissions then they're supposed to. This is what /etc/sudoers
and sudo
's password prompt are for. However, this is a bit risky: if there's a bug in a setuid
program (or config), this could let non-root users get unrestricted root.
Linux has a subsystem called "capabilities" which lets you do something similar to setuid
, but instead of granting all root powers to whomever runs the program, it can be used to only provide specific capabilities. For example, if you set cap_block_suspend
on a program, this means the program can block Linux from suspending. If a program with cap_block_suspend
is exploited, the most a malicious user can get from it is the ability to stop the system from suspending. That's it.
Bedrock's strat
executable uses cap_sys_chroot
, which means it can chroot()
when run as any user. strat
does a bunch of security checks to make sure it only chroot()
s to a selected list of locations, but in theory if someone finds a bug in Bedrock and breaks strat
's security checks, the worst the attacker can do with it is chroot()
unrestricted.
This is why running cross-stratum software on Bedrock as root or with work for you in your original post: as root, you have all capabilities, including cap_sys_chroot
. What's broken is the ability to let non-root users do this.
Normal UNIX permissions predate this concept and don't have room to store the information, which is why Linux puts them in xattrs. For example, Bedrock expects:
$ getfattr -d -m '^security.' /bedrock/bin/strat
getfattr: Removing leading '/' from absolute path names
# file: bedrock/bin/strat
security.capability=0sAQAAAgAABAAAAAAAAAAAAAAAAAA=
I don't know pmOS any what well. Can you use this background as a jumping off point to check if pmOS is doing something weird to block Linux capabilities?
Thanks for the explanation, can I see your output for zcat /proc/config.gz | grep 'EXT4' | grep 'SECURITY'
? I read on this forum that EXT3_FS_SECURITY =y (EXT4 in this case) is needed for using capabilities.
I tested the previous zcat
command and it doesn't appear on my pc either, it seems it's enabled default if there's no entry:
we are better removing this config option, and so that it is effectively always enabled.
I was reading from an older forum before this kernel change. Since my pc has the security flag set but pmos doesn't, I'd like to explore this further before dismissing kernel config.
I'm compiling the kernel again using pmbootstrap
, I noticed in the config that EXT4_FS_SECURITY is default set to '=n'.
$ zcat /proc/config.gz | grep 'EXT4' | grep 'SECURITY'
CONFIG_EXT4_FS_SECURITY=y
I am on ext4. This seems a promising lead to pursue.
I finally managed to reinstall pmos using the configured kernel, the instructions for pmbootstrap
weren't so clear.
Now when I run zcat /proc/config.gz | grep 'EXT4_FS_SECURITY'
I get CONFIG_EXT4_FS_SECURITY=y.
Using getcap no longer gives Not supported
error.
I installed the latest Bedrocklinux script and it looks like there's no longer any cap_sys_chroot
error when doing strat [distro]
.
However I still get the same error as before when doing brl update
:
* Checking for updates
Checking https://raw.githubusercontent.com/bedrocklinux/bedrocklinux-userland/0.7/releases... ERROR!
Connecting to raw.githubusercontent.com (151.101.0.133:443)
wget: error getting response: Connection reset by peer
ERROR: No working mirrors found
brl which wget
returns global
, and I know that wget
works because I used it to download the latest bedrock-linux script.
In addition fetching fedora still gives me errors about setting --release
and --mirror
. However fetching fedora works now and I can run fedora commands without any error. Centos fetches and works without any issues.
I installed wget
on my ubuntu strat, and when I run sudo strat ubuntu brl update
it works properly. Similarly I can run sudo strat ubuntu brl fetch fedora
and it fetches properly. So these errors seem to be due to busybox
's wget
.
The wget issues should be fixed if you install the wget package on alpine
When testing bedrock on pmOS, hijacking was done successfully. However there are still 2 issues:
strat postmarketos
(Only pmos and bedrock programs can be run)For fetching I tried all that were officially available: Ubuntu: success Alpine: success Centos: success fetching but fails on grub2-tools, shadow-tools, and unbound-libs (similar errors as Fedora) However all other distros had the following error:
I have not used manual mirrors for any distros other than Fedora, I can do so if it would be helpful. Fedora gives the error:
After setting both release and mirror flags to avoid errors, Fedora does fetch packages successfully, however it fails on certain packages (libsnl2, shadow-utils, and unbound-libs for Fedora32, different versions give different package errors). Looking through the output I see:
Error unpacking rpm package
for all 3 packages as well asError in PREIN scriptlet in rpm package unbound-libs
It seems fetching can get broken or fixed depending on what version pmos is.The second issue is where running any program with a strat that isn't postmarketos (including
strat bedrock
) causes this error:The only solution is running as root or sudo, using
strat -r
does nothing and neither does using explicit paths. I've always had this issue on pmos, however on Mobian I also got these errors when switching the kernel from 5.7 to 5.6. Switching the kernel back solved the issue. My device is Pinephone ubports edition, and I'm using pmos edge channel. Need to try testing with stable channel. I will try to do more testing and update this post accordingly, if anyone has any ideas on what else I can try let me know.