Investigate Pixel's new repair mode feature

[chenxiaolong]

The December 2023 Pixel OTA update added support for a "repair mode". I'd like to investigate to see if this has any impact to how things are signed for AVB.

The feature is not open source. There are parts of the implementation in /system_ext/priv-app/SettingsGoogle and /system_ext/app/RepairMode. At first glance, it seems to be implemented on top of Android's DSU mechanism.

[chenxiaolong]

Repair mode is fully working in an avbroot setup, but with a big caveat (see bottom of post).

But first, this is how the feature is implemented:

1. When repair mode is turned on, it creates a new DSU (dynamic system update) slot named repairmode.lock:

   DynamicSystemManager.startInstallation("repairmode.lock")

2. It then writes an empty userdata partition image in /data/gsi/dsu/repairmode.lock/. The minimum size of the image file is 2 GiB and the maximum size is min(<free space> - 300 MiB, 10 GiB) rounded down to the nearest GiB.

   DynamicSystemManager.createPartition("userdata", size, /* readOnly */ false)
   DynamicSystemManager.closePartition()
   DynamicSystemManager.finishInstallation()

   Because no other partition images are created, the original system/system_ext/product/vendor/etc. are used as-is.

3. If the DSU creation is successful, then it marks the new repairmode.lock slot as the default slot for all future reboots.

   DynamicSystemManager.setEnable(/* enable */ true, /* oneShot */ false)

   If you're quick, running gsi_tool before the device has a chance to reboot will output:

   # gsi_tool status
   installed
   enabled
   [0] repairmode.lock
   installed: userdata_gsi
   AVB public key (sha1): [NONE]

4. After rebooting, the device will mount the new userdata image at /data instead of the real userdata partition. Even though the image lives on the real partition, the real partiition is never mounted. This works because the image creation procedure uses Linux's fiemap API to record which sectors/blocks the image file occupies on disk. When booted into repair mode, only those specific blocks are accessed (via dm-linear).

5. The (proprietary) RepairMode app will set up Android's device policy mechanism to disable the initial setup wizard and gray out the factory reset and OEM unlocking options.

6. The device remains in repair mode until the user exits it via the notification. This will configure the system to boot back into the original system and the repairmode.lock DSU slot will be deleted after rebooting.

---

WARNING!

There's is one big caveat with avbroot + repair mode. Repair mode will also be rooted! All it takes is installing the Magisk/KernelSU app and root access will be granted. This is problematic because with root access, it's trivial for someone to manually mount the original userdata partition and install a malicious Magisk/KernelSU module. Emphasis on trivial since repair mode has no authentication (because of what the feature is intended for). A malicious module installed this way would persist after exiting repair mode.

EDIT: Per @pascallj's suggestion below, a good way to work around this is by sideloading an unrooted avbroot-signed OTA prior to entering repair mode. This can be done by patching with the --rootless option. After the device is repaired, exit repair mode and sideload the rooted patched OTA again. Switching between unrooted and rooted Android builds this way does not require wiping data.

EDIT 2: This helps close the unauthenticated root access vulnerability, but it's not perfectly safe. In repair mode, Google's original OTA updater is enabled by default and could potentially run. avbroot's clearotacerts module won't be active to prevent this since the OS would be unrooted. There is currently no solution for this. Best to just flash the stock OS before sending the device in for repairs.

[pascallj]

Thanks for researching this! When I noticed this feature in the feature drop, it immediately peaked my interest.

I agree that giving a device with avbroot to any sort of official repair shop is a bad idea. It will raise red flags and I think they will just reset the device then anyway. However if you're just getting your screen replaced at the local phone shop around the corner, this mode would still be sufficient.

Flashing the stock OS will force you to wipe your device as we have established that it cannot (or won't) decrypt userdata written with custom keys. Would flashing a rootless variant of the system fix this issue? There won't be any way to get root on the system now and because the bootloader is locked it is impossible to revert to the other version. You can flash a root-enabled variant when the device returns and everything should work as is.

[chenxiaolong]

Would flashing a rootless variant of the system fix this issue?

That's brilliant! I didn't think of that at all. Yes, that completely solves the problem. Switching between avbroot-signed rootless and rooted builds is just an adb sideload away. I'll update my message above to suggest that.

[pascallj]

One downside I was later thinking of, is that the 'clearotacerts' module is not running. Therefore maybe the system might get an accidental OTA update (if that works in repair mode) which breaks the system.

[chenxiaolong]

~I'll check AOSP tomorrow and see if I can find a definitive answer to that. My guess is that update_engine or something would hopefully disable itself while booted into an DSU. When booted into a DSU slot, any partition (eg. system) might be an arbitrary image. There are also post-install scripts that run in the host environment (Android normally or recovery if sideloading) when an OTA is installed. A DSU is not guaranteed to have a proper environment for that. It'd be pretty crazy/amazing if update_engine somehow worked.~

[chenxiaolong]

I was wrong. I tested with Custota and it looks like OTA updates go through just fine in repair mode. It also looks like repair mode enables the OEM updater app too, which I suppose makes sense since that's the default. I don't see a good solution around this.

I was trying finding a way to do what the clearotacerts module does without Magisk/KernelSU, but after looking at the code for /system/bin/init, I highly doubt this is possible without wrapping the binary like what Magisk does. I think that's a bit too invasive and may conflict with Magisk.

I can't think of a way to use repair mode that's 100% safe against the accidental OTA update scenario at the moment.

[pascallj]

Thanks for testing! Avbroot supports repacking of images now right? What if we just replace the otacerts.zip and repack the image?

[chenxiaolong]

That would be the best solution. avbroot can deal with all the AVB, dm-verify, and FEC parts, but it doesn't have the ability to write to an ext4 image or recreate an erofs image (since Android's copy of otacerts.zip lives in system.img). I'm not aware of any libraries that can read/write those filesystems in userspace.

[pascallj]

I'm not aware of any libraries that can read/write those filesystems in userspace.

In userspace, as in userspace on the device avbroot is running on?

We can just corrupt the file 🙈 Ext4 doesn't have checksums as far as I'm aware.

[chenxiaolong]

In userspace, as in userspace on the device avbroot is running on?

I meant on the system that's running avbroot ota patch. The filesystem would have to be modified and then avbroot can recompute the dm-verity hashes and FEC.

We can just corrupt the file 🙈 Ext4 doesn't have checksums as far as I'm aware.

:joy: I guess that is indeed theoretically possible. Still would require some sort of ext4/erofs parser to find the blocks where the file is located though.

(If there actually is a good way to find the block(s) containing otacerts.zip, we could do better than corrupting it. We could insert one with the user's key and pad out the file to equal the original size. avbroot's generated otacerts.zip is usually a few hundred bytes smaller than Google's original one.)

[chenxiaolong]

Well, it turns out recovery's copy of otacerts.zip is byte-for-byte identical to the copy inside system.img. Blindly searching for those bytes inside system.img and replacing them actually works: https://gist.github.com/chenxiaolong/da42491c0b7d4fb8262b6fc3c2850777.

I tested with ext4 and uncompressed erofs. fsck passes, the filesystem mounts, and the replaced otacerts.zip data shows up as expected.

Although this actually works well, I probably won't implement this because:

• Patching would be slow since system.img would need to be extracted, modified, and recompressed. Inside a temp directory, the image would need to be fully read 3 times and fully written once in order to make the modifications and regenerate the dm-verity metadata. This would probably increase the patching times by an order of magnitude.
• This doesn't work at all for compressed erofs. I wouldn't be surprised if Google switches to this in the near future. I believe Samsung's newest devices already use it.

---

It sure would be nice if we could just:

mount --bind <recovery's otacerts> <system's otacerts>

during init's first stage... There's just no easy way to hook into that process.

[pascallj]

Interesting! I agree, it is not worth implementing it this way. It's a very niche use case.