Would a honest to god homebrew OTA server be feasible? [feature request, understood to be unlikely]

[nad0vs]

The general idea is to make one's own home-hosted OTA server instead of native one. I realize it is way too much work probably but with root (and magisk "strategically" substituting some files) it seems like it might be conceptually possible.

I understand chenxiaolong is probably busy enough just keeping all other projects up to date :-) so a whole custom OTA server + OTA agent may bee too much work, but maybe someone else will get inspired to write and maintain that as a separate project (at least for Pixels which have pretty documented and "standardized" process of updating)

[chenxiaolong]

This was actually one of my "shower thoughts" recently (yes, I have a problem...), so I did some research. I won't be making an OTA updater app myself, but here are my notes in case someone else is interested in doing so:

• The actual A/B signature verification and flashing process is done by Android's built-in update_engine service. The OTA updater app is responsible for downloading the OTA, verifying the metadata (model number, version number, etc.), and then telling update_engine the location of payload.bin inside the OTA zip.
• There are good implementations of OTA updater apps available from GrapheneOS (https://github.com/GrapheneOS/platform_packages_apps_Updater) as well as the AOSP sample (https://android.googlesource.com/platform/bootable/recovery/+/master/updater_sample/). The GrapheneOS one is nice and simple and only requires the server to serve static files. These use are large number of @SystemAPI Android APIs though, so they need to be compiled as part of the AOSP build, not with eg. Android Studio (at least not without some hacks).
• The Settings app and other things that launch the OTA updater thankfully don't hard code which app to open. They launch whichever app can handle intents with the android.settings.SYSTEM_UPDATE_SETTINGS action. I don't think there's anything fancy here. If two OTA updaters are installed, Android should just show the usual "select an app"-style prompt.
• The updater must be installed as a system app, along with the corresponding /system/etc/permissions/ file to grant the relevant system app permissions. This can be easily done in a Magisk module.
• The updater must have SELinux rules that grant it access to the update_engine service. This is how AOSP gives permissions to Google Play Services: https://android.googlesource.com/platform/system/sepolicy/+/refs/tags/android-13.0.0_r66/private/gmscore_app.te#17. And this is how GrapheneOS gives access to its updater app: https://github.com/GrapheneOS/platform_system_sepolicy/commit/70227c607b87ff41652cf9fda9260e5286d914a6. I don't know if Magisk's magiskpolicy is sufficient to make the required changes.
• The Magisk module would need to override /system/etc/security/otacerts.zip with one that contains the custom signing certificate.

I definitely think this is possible to do and there's no reason that fancy features, like incremental OTAs or streaming downloads wouldn't work. The hard part is getting something like this built and packaged as a Magisk module. OTA updaters are very much designed to be built as part of the AOSP build.

[pixincreate]

https://github.com/MuratovAS/grapheneos-magisk I found this golden repo.

[nad0vs]

There's also this thing https://github.com/ur0/UpdateEngineInterface which is horribly out of date but conceptually seems pretty close

[chenxiaolong]

I ended up getting more interested in this, so I'm working on an app to do this now :slightly_smiling_face:

No promises when or if it'll be released. I'm not to the point of testing things yet, so I'm not sure if I'll run into something that makes this completely unfeasible.

As far what to expect, UI and functionality-wise, it'll be very similar to my other project, BCR. Just a basic settings UI, with the progress bar and pause/resume/cancel options shown in a notification only. It'll exclusively use update_engine's streaming install method, meaning the download and install run at the same time and the download process is managed by update_engine itself. It's possible to pause/resume during this process, but pre-downloading an update to install later will never be supported.

For the server side, the only thing required is a static file host that supports the standard HTTP Range header. Even a simple python -m http.server 8080 will work. The server will need to serve a JSON file containing metadata about the latest OTA available for the device. The JSON file contains either a relative path to the .zip or a full URL in case you want to host the large files elsewhere.

Finally, assuming I'm able to get this to work, it'll end up being a separate project living in a separate repo (still open source under GPLv3). It should work with any A/B-style OTA, regardless if it was created by avbroot or not, so it probably doesn't make much sense to put it in this repo.

[chenxiaolong]

OK, I just ran into the first limitation:

Both gmscore (Google Play Services' OTA updater) and GrapheneOS' updater are sandboxed via a separate SELinux domain (gmscore_app and updater_app respectively). These sandboxed domains are the only things allowed to talk to update_engine.

Trying to do the same for an app via a Magisk module is impossible. The way an app as assigned an SELinux domain is via seapp_contexts files, which contain lines like:

# AOSP / gmscore
user=_app isPrivApp=true name=com.google.android.gms domain=gmscore_app type=privapp_data_file levelFrom=user

# GrapheneOS
user=_app isPrivApp=true name=app.seamlessupdate.client domain=updater_app type=app_data_file levelFrom=user

The seapp_contexts recognized by Android 13 are:

/system/etc/selinux/plat_seapp_contexts
/dev/selinux/apex_seapp_contexts
/system_ext/etc/selinux/system_ext_seapp_contexts
/product/etc/selinux/product_seapp_contexts
/vendor/etc/selinux/vendor_seapp_contexts
/odm/etc/selinux/odm_seapp_contexts

These files are loaded by /system/bin/init very, very early in the boot process before /data and thus, any Magisk modules, are even mounted. These files are also never reloaded. There's no easy or sane way to hook into init to modify these files at the right point in time, even if we were to modify the boot image.

---

With that out the window, the best we can do is add some rules for priv_app (the default SELinux domain for anything in /system/priv-app). This would allow any system app to talk to update_engine. This doesn't inherently make things more insecure because update_engine is always going to verify signatures for data given to it, but it does slightly increase the potential attack surface. Now a weak/insecure system app could potentially be used to find and exploit vulnerabilities in update_engine.

It's not the ideal solution, but unless I'm missing something, it's probably the best solution we have.

---

EDIT: I'm dumb. Yes, init loads all these seapp_contexts files very early, but init's copy of this information in memory doesn't matter. What matters is when zygote's loads these files into memory itself. And Magisk modules can definitely do things before that happens.

[chenxiaolong]

Creating a new SELinux domain at runtime is a royal pain in the ass (and not because of Magisk--magiskpolicy is wonderful).

If the updater app's SELinux policy were included in an AOSP build, things would be very easy. It's just a few rules:

type test_app, domain;
typeattribute test_app coredomain;
app_domain(test_app)
untrusted_app_domain(test_app)
net_domain(test_app)
allow test_app ota_package_file:dir rw_dir_perms;
allow test_app ota_package_file:file create_file_perms;
binder_call(test_app, update_engine)
allow test_app update_engine_service:service_manager find;
binder_call(update_engine, test_app)

Directly translating these to magiskpolicy commands is pretty simple. After evaluating the macros (like binder_call(...)), it's only 20 rules or so. However, the AOSP SELinux policy has tons of statements like these:

allowxperm { appdomain -bluetooth } self:{ rawip_socket tcp_socket udp_socket } ioctl { unpriv_sock_ioctls unpriv_tty_ioctls };

("allow everything with the appdomain attribute except bluetooth to perform some network socket operations")

These everything except <blah> rules are what's incredibly painful. The "everything" needs to include the new SELinux domain for the updater app, resulting in several hundred additional rules after the policy is compiled.

Replicating that outside out of the AOSP build isn't really possible:

• There's no way to find these kinds of rules on the device itself to handle them programmatically because the information doesn't exist in the /sepolicy binary. The rules were already expanded when the policy was originally compiled.
• Compiling AOSP and AOSP + custom policy, saving the diff between them, and then loading that with magiskpolicy works, but only for the specific device and version AOSP was compiled for (it's not portable).

I'm probably going to go back this approach:

With that out the window, the best we can do is add some rules for priv_app (the default SELinux domain for anything in /system/priv-app). This would allow any system app to talk to update_engine. This doesn't inherently make things more insecure because update_engine is always going to verify signatures for data given to it, but it does slightly increase the potential attack surface. Now a weak/insecure system app could potentially be used to find and exploit vulnerabilities in update_engine.

[pixincreate]

Instead, can we modify the existing Updater such that, it checks for Official OTAs (GrapheneOS as of now), if exist, download that, instead of installing, the updater applies the patches to the ROM that is downloaded, and install them. Would that go? I feel like the things that you mentioned are complicated.

Given the existing updater is open-source, we can modify that.

Also, I personally, now prefer KernelSU over Magisk since it allows more robust rooting mechanism and hides root is better and safer way (which is what I believe).
So, patching kernel on every OTA automatically would be really feasible IMO.

What do you think? @chenxiaolong

[chenxiaolong]

:slightly_smiling_face:

All SELinux issues have been worked around with no compromises on security.

What works:

• Checking for and installing updates
• Showing process and success/failure alerts
• Letting the update process continue running in the background while the screen is off

Performance on my Pixel 7 Pro (plugged in, 100% battery, screen off):

• ~25 minutes for download/install phase
• ~1 minute for verification phase
• ~30 minutes for the finalization phase. This is where Android runs the OTA's "post install" script. For AOSP, this runs dexopt to precompile every app's dex files in the new boot slot. Other ROMs may perform additional tasks. The OTA updater app will have an option to skip post install scripts, but only if it's marked as optional in the OTA (update_engine always runs mandatory post install scripts).

(It will always be slower than adb sideload because Android deprioritizes update_engine to avoid slowing down the device. The post install script also skips the dexopt step if it detects that the update is being sideloaded.)

Remaining work:

• [x] [EDIT: DONE] UI to configure OTA server URL
• [x] [EDIT: DONE] UI to view list of OTA verification certificates (/system/etc/security/otacerts.zip) so that it's easy to confirm that the custom certificate was correctly installed
• [x] [EDIT: DONE] Hidden option to allow reinstalling the current OTA for testing (downgrading is never possible)
• [x] [EDIT: DONE] Test pause/resume/cancel behavior
• [x] [EDIT: DONE] Test behavior when attempting an update after an app crash
• [x] [EDIT: DONE] Test behavior when attempting an update after a previous update completed but before a reboot
• [x] [EDIT: DONE] Automatic updates
• [x] [EDIT: DONE] Documentation + preparation for release

---

The server side metadata is a JSON file that looks like the example below. The server can be any plain old HTTP server that supports the Range header. I was wrong in my previous post about python -m http.server working, but any of the normal webservers, like Apache, Nginx, or Caddy would work fine. For testing, Caddy is great because you can just run caddy file-server -access-log -listen :8080 inside the folder containing the metadata and OTA zips--no need to mess with config files.

// (A script will be provided to generate this file.)
{
    // The location for the full OTA.
    "full": {
        // The location can be a relative path or a full URL.
        "location": "cheetah-ota-tq3a.230705.001-dfcb7951.zip.patched",

        // The values of the `metadata` entry from `ota-property-files` inside
        // the OTA's META-INF/com/android/metadata file.
        "metadata_offset": 2343312763,
        "metadata_size": 646
    }
}

EDIT: Remove incremental options from JSON example. I don't plan on implementing it unless/until someone implements a way to actually generate incremental updates from two zips. (I do not plan on adding this functionality to avbroot.)

[chenxiaolong]

Custota has been released! https://github.com/chenxiaolong/Custota

I've tested it pretty extensively on my device, so hopefully there aren't too many bugs. If you run into issues, feel free to open a bug report over there.

[pascallj]

Very cool, will have a look at this in the future!