anestisb / android-prepare-vendor

Set of scripts to automate AOSP compatible vendor blobs generation from factory images
348 stars 155 forks source link

remove product from radio images #156

Closed thestinger closed 5 years ago

thestinger commented 5 years ago

This is already provided by AOSP for blueline/crosshatch:

# product.img
BOARD_PRODUCTIMAGE_PARTITION_SIZE := 314572800
BOARD_PRODUCTIMAGE_FILE_SYSTEM_TYPE := ext4
TARGET_COPY_OUT_PRODUCT := product

Including this as a radio image has unclear consequences and appears to be incorrect.

Closes #155

thestinger commented 5 years ago

In a standard AOSP build, this image literally only contains a build.prop describing the build. I'm unsure on the consequences of including a duplicate mismatched with the build. It may be one of the issues leading to over-the-air update breakage, but I don't think it has been an issue up to this point. It's verified via dm-verity based on the vbmeta information and gets mounted at /product. It's documented here:

https://source.android.com/devices/bootloader/product-partitions

Google's factory images product.img has contents inside it including the Verizon apps, so you might want to set it up in your full builds. You could extract the files and add them to the proper output directory. It's wrong to include it as a duplicate that simply gets ignored though.

product
├── app
│   ├── MobileFeliCaClient
│   │   └── MobileFeliCaClient.apk
│   ├── MobileFeliCaMenuApp
│   │   └── MobileFeliCaMenuApp.apk
│   ├── MobileFeliCaMenuMainApp
│   │   └── MobileFeliCaMenuMainApp.apk
│   ├── MobileFeliCaSettingApp
│   │   └── MobileFeliCaSettingApp.apk
│   ├── MobileFeliCaWebPlugin
│   │   └── MobileFeliCaWebPlugin.apk
│   ├── MobileFeliCaWebPluginBoot
│   │   └── MobileFeliCaWebPluginBoot.apk
│   ├── SSRestartDetector
│   │   └── SSRestartDetector.apk
│   ├── VZWAPNLib
│   │   └── VZWAPNLib.apk
│   └── WallpapersBReel2018
│       └── WallpapersBReel2018.apk
├── build.prop
├── etc
│   ├── cne
│   │   └── andsfCne.xml
│   └── permissions
│       ├── com.android.sdm.plugins.connmo.xml
│       ├── com.android.vzwomatrigger.xml
│       ├── com.customermobile.preload.vzw.xml
│       ├── com.verizon.apn.xml
│       ├── com.verizon.llkagent.xml
│       ├── com.verizon.services.xml
│       ├── obdm_permissions.xml
│       └── vzw_mvs_permissions.xml
├── lost+found
├── media
│   ├── audio
│   │   ├── alarms
│   │   │   ├── A_real_hoot.ogg
│   │   │   ├── Bright_morning.ogg
│   │   │   ├── Cuckoo_clock.ogg
│   │   │   ├── Early_twilight.ogg
│   │   │   ├── Full_of_wonder.ogg
│   │   │   ├── Gentle_breeze.ogg
│   │   │   ├── Icicles.ogg
│   │   │   ├── Jump_start.ogg
│   │   │   ├── Loose_change.ogg
│   │   │   ├── Rolling_fog.ogg
│   │   │   ├── Spokes.ogg
│   │   │   └── Sunshower.ogg
│   │   ├── notifications
│   │   │   ├── Beginning.ogg
│   │   │   ├── Coconuts.ogg
│   │   │   ├── Duet.ogg
│   │   │   ├── End_note.ogg
│   │   │   ├── Gentle_gong.ogg
│   │   │   ├── Mallet.ogg
│   │   │   ├── Orders_up.ogg
│   │   │   ├── Ping.ogg
│   │   │   ├── Pipes.ogg
│   │   │   ├── Popcorn.ogg
│   │   │   ├── Shopkeeper.ogg
│   │   │   ├── Sticks_and_stones.ogg
│   │   │   ├── Tuneup.ogg
│   │   │   ├── Tweeter.ogg
│   │   │   └── Twinkle.ogg
│   │   ├── ringtones
│   │   │   ├── Copycat.ogg
│   │   │   ├── Crackle.ogg
│   │   │   ├── Flutterby.ogg
│   │   │   ├── Hotline.ogg
│   │   │   ├── Leaps_and_bounds.ogg
│   │   │   ├── Lollipop.ogg
│   │   │   ├── Lost_and_found.ogg
│   │   │   ├── Mash_up.ogg
│   │   │   ├── Monkey_around.ogg
│   │   │   ├── Schools_out.ogg
│   │   │   ├── The_big_adventure.ogg
│   │   │   └── Zen_too.ogg
│   │   └── ui
│   │       ├── audio_end.ogg
│   │       ├── audio_initiate.ogg
│   │       ├── camera_click.ogg
│   │       ├── camera_focus.ogg
│   │       ├── ChargingStarted.ogg
│   │       ├── Dock.ogg
│   │       ├── Effect_Tick.ogg
│   │       ├── InCallNotification.ogg
│   │       ├── KeypressDelete.ogg
│   │       ├── KeypressInvalid.ogg
│   │       ├── KeypressReturn.ogg
│   │       ├── KeypressSpacebar.ogg
│   │       ├── KeypressStandard.ogg
│   │       ├── Lock.ogg
│   │       ├── LowBattery.ogg
│   │       ├── NFCFailure.ogg
│   │       ├── NFCInitiated.ogg
│   │       ├── NFCSuccess.ogg
│   │       ├── NFCTransferComplete.ogg
│   │       ├── NFCTransferInitiated.ogg
│   │       ├── Trusted.ogg
│   │       ├── Undock.ogg
│   │       ├── Unlock.ogg
│   │       ├── VideoRecord.ogg
│   │       ├── VideoStop.ogg
│   │       └── WirelessChargingStarted.ogg
│   └── bootanimation.zip
└── priv-app
    ├── AppDirectedSMSService
    │   └── AppDirectedSMSService.apk
    ├── ConnMO
    │   └── ConnMO.apk
    ├── DCMO
    │   └── DCMO.apk
    ├── DiagMon
    │   └── DiagMon.apk
    ├── LLKAgent
    │   └── LLKAgent.apk
    ├── MyVerizonServices
    │   └── MyVerizonServices.apk
    ├── OBDM_Permissions
    │   └── OBDM_Permissions.apk
    ├── obdm_stub
    │   └── obdm_stub.apk
    ├── Showcase
    │   └── Showcase.apk
    └── VzwOmaTrigger
        └── VzwOmaTrigger.apk

31 directories, 95 files
thestinger commented 5 years ago

An interesting story of debugging that led to this, not particularly relevant but interesting enough to write it up:

After the April security update, this led to a very strange bug in GrapheneOS which took me a long time to unravel and trace back to this. It caused over-the-air updates to soft brick the device, breaking booting the newly installed OS after rebooting to boot the new update. The device would fail to boot a until the retry limit and then roll back to the previous version, which would then also somehow fail to boot despite working before the update. I have a Suzy-Q debug cable, but it's for ChromeOS devices and apparently lacks support for obtaining the kernel logs on Pixel phones so I wasn't able to obtain the logs and had to try debugging and narrowing down this problem blind. I first assumed that it was a bug in the updated OS triggered in early boot, and that it was somehow breaking rollback. I spent a long time pursuing that with no luck. I tried to rule out it being a bug in the source OS by testing a bunch of updates from it to builds of earlier versions, which all worked fine until the April update. I slowly ruled out all of my hardening and carefully looked through the android-9.0.0_r34..android-9.0.0_r35 AOSP and vendor diffs many times, unable to see anything able to cause it.

Eventually, @ScottyBauer was able to replicate the problem for me and get me kernel logs via his custom made sketchy as hell Suzy-Q cable. It turned out that f2fs was detecting that userdata was corrupt on boot. That was strange and unexpected. It explained why rollback failed, I was quite sure that it indicated it was a bug in the source OS in the update implementation, triggered by some strange change in the update package. I spent a huge amount of time debugging update_engine to identify the problem. I determined that this problem only occurs for over-the-air updates within the OS via the update client, and never via sideloaded updates despite that also using the same update_engine backend. I finally ended up narrowing it down to this: a very strange update package created due to the double inclusion of product.img, which I guess has become significantly larger in this release. It manages to cause update_engine to corrupt a tiny little bit of userdata, but so little that it still works fine until actually rebooting and trying to mount / fsck it.

I think that update_engine is buggy, along with the brillo_update_payload generation script, and this was simply a very strange way of triggering various bugs due to a poorly handled edge case. I'm very curious about how it manages to mostly work in both cases, but with userdata corrupted when the full OS is booted up and not corrupted when it isn't booted. Something to do with differences in SELinux policy or whether userdata is mounted perhaps. It's very strange, since I wouldn't think that would be possible even with a buggy update_engine. It makes me thinks there are deeper storage bugs. I don't want to think too much more about it right now though. I just spent 2.5 long days debugging this already,