BusyBox no longer installs with Chrome OS 65 (/system is read-only)

[jbardi]

As of Chrome OS Stable version 65, you can no longer install BusyBox. I downgraded back to 64 and re-rooted and BusyBox installed instantly, so I once again upgraded to 65 and although I am able to root, BusyBox will no longer install. Google changed something in 65 and none of the 4 BusyBox installers tested on the Play Store can write to /system even after being granted root access.

I know it is not directly related to your root scripts, but I was hoping that maybe it was something that could be worked around with a change of some kind in your scripts. Busybox is a necessity for me, so without being able to install it, I guess I'll have to move back to using my laptop and tablet. Was hoping a Chromebook could replace them.

Is there any way you can inject busybox with your scripts just like you install su into the image?

[jbardi]

You can delete this issue. I resolved it by manually copying the busybox executable to the /opt/google/containers/android/rootfs/root/system/xbin folder, which Android sees as /system/xbin, the location that the BusyBox installer from the Play Store attempts to install to, however, the /system/xbin folder is readonly, so the install fails on Chrome OS 65. I did look at your script and see that you have the option to install busybox if SuperSU is not already in the downloads folder, however, your script tries to download it using wget, but there is no wget on chrome os until after you install busybox, so you may want to change the busybox download command to use curl instead. Also, some apps expect to find busybox in /system/xbin or /system/bin, so that is why I couldn't use it in the /usr/local/bin directory as stated in your script.

[jbardi]

I spoke too soon... I have been trying to get this working for days now, and on my last attempt I didn't realize I had restored back to Chrome OS 64, so no, I didn't fix the problem. Even manually putting busybox in /usr/local/bin or /opt/google/containers/android/rootfs/root/system/xbin (aka /system/xbin) does not allow the apps to utilize busybox. So Chrome OS 65 is still broke as far as BusyBox is concerned. Oh well, back to the drawing board.

Running this on an ASUS Chromebook Flip C302

[nolirium]

Oh thanks for the heads up re: curl/wget. I actually thought I'd changed the wget to curl in the scripts a while back (after they removed wget from CrOS). It appears I only changed it in one of the script versions. Cheers, as you suggest, that should be swapped over everywhere.

Yes, I shouldn't think it would be a problem to fix up the scripts to install busybox properly in the Android container. I had an idea to do that at some point, as you've seen in the script, but didn't get around to it...

Edit: I see I actually added it in to the script at one point, at least in part. I'm going to comment it out again for the moment, temporarily.

Actually, thinking about it, I think I might know what the issue with manually installing busybox could be. Two things to maybe try:

Firstly: Although some of the newer Intel Chromebooks seem to be moving over to 64 bit Android, I am not sure if the c302 has done so yet. If it's still running a 32 bit container, and if you were using the version that the script attempts to download, maybe this could be the reason why it's not working after you manually added it.

Secondly: Should also maybe just double check it's not an issue with permissions or similar. I'll share the set of commands I just used to get a working BB on a fresh container image.

So, for the CPU arch, you could try checking the build.prop. In a root CrOS shell, if you enter

printf "getprop ro.product.cpu.abilist" | android-sh; printf "getprop ro.product.cpu.abilist64" | android-sh

It should return two lines, if the second is blank then I think it's definitely 32 bit.

If it is 32 bit, if you previously tried with the x86_64 version from the script, you may need https://busybox.net/downloads/binaries/1.26.2-defconfig-multiarch/busybox-i686 for the Android container instead.

And then secondly, to rule out any problem with the permissions etc - Using the latter file as the example:

sudo su
cd /tmp/
curl https://busybox.net/downloads/binaries/1.26.2-defconfig-multiarch/busybox-i686 -o busybox
cp busybox /opt/google/containers/android/rootfs/root/system/xbin/
chown 655360 /opt/google/containers/android/rootfs/root/system/xbin/busybox 
chgrp 655360 /opt/google/containers/android/rootfs/root/system/xbin/busybox 
chmod a+x /opt/google/containers/android/rootfs/root/system/xbin/busybox 
printf "busybox --install -s /system/xbin" | android-sh

Regarding Chrome OS version 65: It's strange that /system couldn't be written to at all, though I wonder if this may have possibly been due to some change in CrOS v65 that was perhaps more recently reverted, as I haven't had any issues on v66 or 67 lately. I don't remember having issues on 65, but that was a while back on dev/canary, and I may not have even noticed at the time...

Definitely no problem writing to /system here at the moment though, in fact I'm just finishing off an xposed install script which writes it all to /system - even that is working fine (more or less).

If I get some spare time to go through and back up all my /usr/local files, I might drop down to stable, to see if I can replicate the issue. In the meantime, if using a less stable CrOS build is an option you're willing/able to consider, it might be worth trying v66 or 67, to see if you still have the issue...

[jbardi]

Wow, that was SUPER informative. Thank you so much for your detailed response. I am VERY confident I can get it working with all this great info. And yes, the ASUS Flip C302 is running a 32-bit android container, so I'm going to use the busybox version you linked to. Awesome job, can't possibly thank you enough.

[jbardi]

Ok, so now we get to the crux of the issue... when I type the following command:

printf "busybox --install -s /system/xbin" | android-sh

I get the following error, for each of the symlinks it is trying to create, obviously with the actual symlink name being different for each command:

busybox: /system/xbin/addgroup: Read-only file system

I have reinstalled Chrome OS 64, rooted with your RootandSEpatch.sh script and ran the same command without any errors, which means the /system is read/write, but back in Chrome OS 65 stable, also rooted using the RootandSEpatch.sh, the /system seems to always be read-only.

So I thought doing --remove_rootfs_verification automatically made it writable, but apparently something changed in Chrome OS 65 on the stable channel, because it is definitely read-only.

If I type the mount command in the CrOS shell, I see this:

/dev/loop1 on /opt/google/containers/android/rootfs/root type ext4 (rw,nosuid,nodev,noexec,relatime,seclabel)

but when I type it using android-sh, I see this:

/dev/loop1 on / type ext4 (ro,seclabel,relatime)

So it is mounted RW as far as Chrome OS can tell, but the android side of things sees it as RO

Just some more info, I am testing this on the newest stable release 65.0.3325.209, which is the 3rd v65 release to the stable channel, which I am told is very rare, as they usually release one for Chromebooks without Android and the second for Chromebooks with android, so for them to release a 3rd update on the same version branch is apparently very rare and would denote a major bug was found that impacted enough users to require a 3rd update. So I have no idea if the first release for my device on 65 had the same problem or if it was introduced only with the latest 65.0.3325.209

[nolirium]

Yeah, I've now had a chance to check out the current v65 stable, and can replicate this issue. As you describe, it seems that the Android /system can't be written to, even after both removing rootfs verification in CrOS (which allows the CrOS filesystem to be mounted writeable) and also having the script set the WRITABLE_MOUNT flag in /etc/init/arc-setup-env (which usually makes the Android filesystem writeable),

My first thought was that perhaps there could be an issue with the latter, setting the "export WRITABLE_MOUNT=1" flag in /etc/init/arc-setup-env, but, according to the logs (in /var/log) this was still getting processed as expected by the arc-setup binary.

After testing on v65 stable, I moved up to v66 beta, and found that the situation was the same. I'm now on v67 dev, and the issue is no longer present i.e. I can write to the Android /system as usual.

So, still not sure what's caused it at the moment, but can confirm that it is not present on the dev and canary builds. If I get a chance, I may try to investigate further.

In the meantime, there should be a solution to get BusyBox, at least, working on v65/66. In RootandSEpatch.sh, I've switched out the Intel x64 BusyBox binary in favor of the 32 bit version for now, and I've added the following function, which from my testing appears to correctly add BB to Android on all of the OS versions, with all the symlinked commands then accessible in e.g. Android terminal emulator...

if [ -e /usr/local/bin/busybox ] ; then
 cp  /usr/local/bin/busybox $arc_system/xbin
  chown 655360 $arc_system/xbin/busybox
  chgrp 655360 $arc_system/xbin/busybox
  chmod a+x $arc_system/xbin/busybox
  cd $arc_system/xbin/
  echo "Executing './busybox --install -s ../xbin'"
  ./busybox --install -s ../xbin
  echo "Replacing absolute symlinks created by 'busybox --install' with relative symlinks"
  find ../xbin -lname "*" -exec  sh -c 'ln -sfr busybox $0' {} \;
fi

I also found that it was possible to do this in the CrOS shell, without having to redo the rooting, by mounting the system.raw.expanded.img to another location. After doing this, making the symlinks, replacing them with relative ones, then rebooting, the modified /system was mounted up and BusyBox's symlinked commands all seemed to be fully functional.

So if you now have BusyBox present in your Android xbin, and you're loth to redo the whole rooting script, you could try something like the following (in a CrOS root shell) for the symlinks:

mount /usr/local/Android_Images/system.raw.expanded.img /usr/local/Android_Images/Mounted/
cd /usr/local/Android_Images/Mounted/system/xbin
./busybox --install -s ../xbin
find ../xbin -lname "*" -exec  sh -c 'ln -sfr busybox $0' {} \;

It worked for me, so hopefully should for you, too. The problem with running busybox --install outside of the container is that it creates absolute symlinks (so inside the container they would be broken). The last command there should replace those absolute symlinks with relative ones.

[jbardi]

That is really weird that it is not allowing the /system to be writable. I hope this is just a mistake and perhaps it will be fixed in a later version of Chrome OS. Your workaround for BusyBox works perfectly, I really appreciate it. This allows me to do some of things I'm working on, unfortunately, half the other stuff I am doing requires me to periodically write to /system, so I'm still dead in the water. If this is the new direction Google is taking to lock down the /system permanently, then I'm going to have to give up on Chromebook and move back to my laptop. For the time being, I'm downgrading back to v64 and then breaking the auto update feature so that I will stay on v64 in hopes that you are somehow able to find a way to make /system writable again, or Google fixes the problem.

Thank you again for all your help and the time you have put in for these scripts!

[nolirium]

Some good news and some bad news. The good news is that this seems to just be a temporary bug/glitch of sorts, an incidental consequence of some of the ongoing development work related to arc++ and containers in general on Chrome OS. The issue appears to be an unintentional by-product of some recent changes made to the container implementation.

Reading through some of the change logs and commit comments, it looks like there may have been a few teething troubles with run_oci (/usr/bin/run_oci), their new "minimalistic container runtime that is (mostly) compatible OCI runtime spec". This has had some active development lately, and some code changes relating to this binary and its*.json file(s) (which have recently appeared under /opt/google/containers/android/) seem to have been the cause of the issue at hand.

It looks like the issue may have been fixed in the following commit: https://chromium-review.googlesource.com/c/chromiumos/platform2/+/930577

From that commit, and its neighbor in the relation chain, I gather that the move to using (a certain configuration of) run_oci introduced the issue where the writable flag (which is apparently intended to only be set by developers), when set, wasn't getting passed through to the container properly.

That commit was made on Feb 23 and v65 was already in beta at that point, so I think that the commit would have only made it into v66+. Since the arc++ related source was only made public around the same time, some of the source code for v65 relating to this issue does not appear to be available publicly as far as I can tell, and the container config.json (and some related behavior of run_oci) in v65 differs quite significantly from the most recent source code, which is available online.

Unfortunately I have not, so far, had success in re-enabling the writable rootfs for v65 by way of editing the *.json files, particularly as run_oci appears to be compiled to accept slightly different flags/syntax being passed to it in each version of Chrome OS (as its functionality is improved/streamlined). For v66, while I initially experienced the read-only issue, I found that loop1 would successfully mount as writeable in Android after I had changed one line in config.json (under 'rootfs/root'), to bring the file in line with the current source. However, the earlier container config .json files as found in v65 are rather different, and from what I can see it seems to not be as easily possible to 'fix' the behavior in v65.

Depending on use case, one possibly workable approach resulting in a writeable /system (of sorts) on v65, could be to use a bind mount. I tested this out a few different ways and although I got it working at one point, it turns out that this approach is sketchy at best, and plain non-functional at worst. Not recommended.

It looks like you could be better off staying on v64 rather than 65 for the time being anyway, as there seems to be quite a number of reports of crashing and freezing issues on v65 (unrelated to arc++, apparently). Regarding the 3rd update (to 65.0.3325.209) you mentioned your device received, the Google release blog says the following:

Update 4/13: Asus Chromebook Flip C302, Google Pixelbook, HP Chromebook 13 G1 and Samsung Chromebook Pro have been updated to Platform version: 10323.67.5 to patch a kernel crash loop when using bluetooth

And there's been a fair amount of movement on the bug tracker relating to issues with v65 on some devices, including the C302 such as this bug report, in which CrOS developers allude to 'skipping' 65 for some platforms.

So unless you get an unexpected update to 65, I'd probably recommend just keeping on 64 for now. I expect that by the time 66 rolls around to the stable builds this 'read-only /system' issue may have been resolved with updated config.json/run_oci, or if not then at least the fix is (at the moment) quite trivial for v66.

I shouldn't think we need to worry too much about Google trying to lock down the system, while security is obviously a priority they seem to be keeping a good balance between 'developer oriented' open features which can be tinkered with, (to varying degrees of difficulty), and 'user oriented' features which are, presumably, more about keeping things secure and easy to use.

Lately there have also been moves towards open sourcing some more of the arc related code, and various other things like leveraging Open Container Initiative Runtime specification, and this, I think, bodes well for the future.

[jbardi]

Once I saw "and some bad news" my heart sunk... but now after reading everything, I see it as all good news to me. I have no pressing need to be on v65, other than preferring to have the latest features and not wanting to forget to unbreak the update mechanism and being stuck on an old version. Other than that, I have no problem waiting for v66 now that I know it should hopefully be working once again.

I can not thank you enough for posting back and updating me on the current situation with v65. You have definitely made my day! 👍

[jbardi]

Well v66 was finally released as stable, but ASUS C302 just got the update today. And I guess not too surprising, v66 is just like v65. The /system partition is locked to read-only on the Android side, so looks like I'm going to have to wait for v67 after all and hope this isn't going to be a pattern.

[nolirium]

There's a fairly easy workaround for v66. You just need to remove the "dev" argument from the mount options for "/" in /opt/google/containers/android/config.json (then reboot), and the container should then mount as writeable.

On my device this string is on line 32 - this line can just be deleted. I expect the container config.json is likely the same for all devices with the same arc version, but for clarity, the relevant part of the "mounts" section of config.json originally looks like this:

"mounts": [

  {
      "_comment": "TODO(yusukes): Swap dev for nodev and adjust auto-test",
      "destination": "/",
      "type": "bind",
      "source": "rootfs/root",
      "options": [
          "rslave",
          "dev",
          "suid",
          "exec"
      ],
      "performInIntermediateNamespace": true
  },

And after removing "dev" from the mount options for "/", it looks like this:

`"mounts": [

  {
      "_comment": "TODO(yusukes): Swap dev for nodev and adjust auto-test",
      "destination": "/",
      "type": "bind",
      "source": "rootfs/root",
      "options": [
          "rslave",
          "suid",
          "exec"
      ],
      "performInIntermediateNamespace": true
  },

(The way I did this was to open the file withvi /opt/google/containers/android/config.json, then move the caret down to line 32 with the cursor key, then press dd to delete the line, then press :wq! to save and exit).

It doesn't seem to cause any issue mounting / as nodev, as the /dev directory is mounted separately, and as you can see from the comment in the code it looks like they're doing this going forward anyway. In v67+ this problem with the mount config should be completely fixed, with no workarounds or such required.

[nolirium]

Just to update on this, on the most recent versions of CrOS on my device currently (v69/70), the above is no longer applicable, but it is still possible to enable the RW container by editing Android's /init.rc (in a CrOS root shell).

The following works for me (all one line): sed -i 's|mount rootfs rootfs / remount bind ro|mount rootfs rootfs / remount bind rw|g' /opt/google/containers/android/rootfs/root/init.rc

[jbardi]

Well, I just updated to v69 stable, and I was able to make the /system writable using the "sed" line provided, but it makes no difference because any attempt to install busybox using the busybox and busybox pro apps from android play store will immediately crash the system and reboot.

I was able to get busybox installed by manually copying the busybox executable from:

/data/data/stericson.busybox.donate/files/bb/busybox into /system/xbin

I then had to run the following: /system/xbin/busybox --install -s /system/xbin

So, for others that may be having this issue while trying to install busybox on a root v69 Chrome OS using the busybox apps from the app store, you may have to manually install it from an Android root terminal app. (I used termius)

[mmirg]

HP Chromebook X2 71.0.3578.21 11151.11.0 (Official Build) beta-channel soraka-unibuild (soraka) Google_Soraka.10431.75.0 ARC Version 5086868

system.raw.expanded.img appears to be mounted read-only on Chrome OS 71 which means that I'm unable to edit init.rc in the Android root file system.

sed -i.old 's|mount rootfs rootfs / remount bind ro|mount rootfs rootfs / remount bind rw|g' /opt/google/containers/android/rootfs/root/init.rc
sed: couldn't open temporary file /opt/google/containers/android/rootfs/root/sedgCdjut: Read-only file system

/usr/local/Android_Images/system.raw.expanded.img on /opt/google/containers/android/rootfs/root type ext4 (ro,nosuid,nodev,noexec,relatime,seclabel,block_validity,delalloc,barrier,user_xattr,acl)

I can neither change permissions on /opt/google/containers/android/rootfs/root nor figure out how to remount system.raw.expanded.img read-write so I can modify it. I also tried mounting and editing init.rc to remount rootfs as rw but that didn't seem to work either.

Original:

more /usr/local/Android_Images/Original/init.rc | grep -i remount
    # The bind+remount combination allows this to work in containers.
    mount rootfs rootfs / remount bind ro nodev

Currently active and rooted:

more /opt/google/containers/android/rootfs/root/init.rc | grep -i remount
    # The bind+remount combination allows this to work in containers.
    mount rootfs rootfs / remount bind rw nodev