thierryg commented 7 years ago

Hi,

I have tried to re-generate vendors for the two new Google Pixels :

$ PATH=/home/tgayet/_aosp-git-repos/android-prepare-vendor/_tmp-jdk/jdk1.8.0_101/bin:$PATH ; ./execute-all.sh --device sailfish --buildID NDE63V --output $(pwd) --yes

[-] 'sailfish' is not supported I wanna know if you have ever tried to rebuild vendors for the new pixel (pixel and pixel XL : marlin and sailfish) ?

I have both mobile on my side.

What do you need to investigate in order to add the two support for your script ?

"marlin" for Pixel XL

Version Download SHA-256 Checksum 7.1.0 (NDE63H, Oct 2016) Link 96adcecf136d0389569e28a3396eba92880d77df3ceb8e39b0a765d99ddf0929 7.1.0 (NDE63L, Oct 2016) Link 71a65e7ef49f88a8f376831378984c019e8f078dbd39499eb4ca9e1a475df9f0 7.1.0 (NDE63P, Oct 2016) Link dcdaaa51d4b4c0a67c4b670f76fe53b852b38d7b2ce65399ac56799bfc1ca7bb 7.1.0 (Europe, NDE63U, Nov 2016) Link 24e8abeb4e1f09bff31577b46661110bf7d5745abb9269b638f6fb86a338d271 7.1.0 (NDE63V, Nov 2016) Link a66866ba4b8f9f1baa856828d46a747d769098add86098bece50854983ecea3e 7.1.0 (Verizon, NDE63X, Nov 2016) Link 8b520419e32141f68d9f12d7e302bfb77885ed5587908d30b2a0e94ed07038a7

"sailfish" for Pixel

Version Download SHA-256 Checksum 7.1.0 (NDE63H, Oct 2016) Link 43ba5f8130a58770edce6cb0c6c43920798e038cd7cdec2c54f22a1dd474a7c8 7.1.0 (NDE63L, Oct 2016) Link 49876a7c9e8f14da672dad27a0e404cccbb1330a4c9ebd398c34583a576eba6c 7.1.0 (NDE63P, Oct 2016) Link aaaa2f1a592ba43de10396938220c4ed65ff3341edd01afd4b5a4f506b39042c 7.1.0 (Europe, NDE63U, Nov 2016) Link 3fa367096c251cd5ab1a6c465cb60fc075b08b999148bbf0036f1533e3efd364 7.1.0 (NDE63V, Nov 2016) Link 1b3155ff623e8a872a788c54e650a0077175f54d723a185f52fff8a3170478ff 7.1.0 (Verizon, NDE63X, Nov 2016) Link dc45f6c83547266bf9e49edeae0978c3d734278febec2f7023aae4bf299e4fec

https://developers.google.com/android/images

Thanks in advance for your reply.

BR Thierry

anestisb commented 7 years ago

No I haven't tried to generate blobs against the new Pixel devices. At the moment I don't have a Pixel device so any further support will be delayed until I get one (not sure when this will happen).

Have you checked these Google provided binaries: https://developers.google.com/android/drivers

Are there still items missing from provided blobs that are required to have a fully functional AOSP build against Pixel or Pixel XL? If you can enumerate what items are missing maybe can we can start working some configuration files against these devices.

thestinger commented 7 years ago

It would be nice to start with regenerating vendor.img since they don't provide that. It might just work already if it's added as a supported device. I don't have hardware to test on yet.

anestisb commented 7 years ago

The Makefile to generate vendor.img is dynamically constructed based on the contents extracted from factory image archive. So if you edit the execute-all.sh script and add the codenames for Pixel devices at the availDevices array at the top you should have a vendor.img output.

However, someone still needs to dig deeper and see which elements are missing from /system for the two new devices (hopefully they will be the same). Plus needs to trace again which portions of the AOSP device configuration are not aligned with production releases (e.g. the dm-verity disabled at aosp fstab) and hack around them.

thierryg commented 7 years ago

Hi,

Thanks for your reply.

i have well rebuild an android image from scratch with the binaries.

the reboot was correct except a warning boot message and a another one at the end of the android starting. It remind me a hack i have done on the vendor partition (i have comment all the properties).

About the boot warning i have find a way to remove it on the Nexus 5x. I need to find a hack for the Pixel.

In the next step i will :

check the vendors between the one rebuild and an official image ;
check if we can recompile a GNU/Linux kernel (was not possible with the nexus 6 for instance)
1. Getting the AOSP source code (android-7.1.0_r7) :

mkdir buildPixelSailfish/ && cd buildPixelSailfish/ repo init -u /mnt/aospMirror/platform/manifest.git -b android-7.1.0_r7 -g all,tools,default repo sync -j72

Create a cache :

mkdir -p .ccache export CCACHE_DIR=${PWD}/.ccache export USE_CCACHE=1 prebuilts/misc/linux-x86/ccache/ccache -M 25G

Getting vendors :

Binaries for Android 7.1.0 (NDE63X) : https://developers.google.com/android/drivers#sailfish

wget https://dl.google.com/dl/android/aosp/google_devices-sailfish-nde63x-6e9ab619.tgz tar zxvf google_devices-sailfish-nde63x-6e9ab619.tgz ./extract-google_devices-sailfish.sh --> 8.e

wget https://dl.google.com/dl/android/aosp/qcom-sailfish-nde63x-cda901ec.tgz tar zxvf qcom-sailfish-nde63x-cda901ec.tgz ./extract-qcom-sailfish.sh --> 8.e

Build of the Android 7.1.0 image in userdebug :

export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" source build/envsetup.sh lunch aosp_sailfish-userdebug make -j72 ls -al out/target/product/sailfish/*.img -rw-rw-r-- 1 tgayet tgayet 25621800 Nov 11 19:37 out/target/product/sailfish/boot.img -rw-rw-r-- 1 tgayet tgayet 1485538 Nov 11 19:37 out/target/product/sailfish/ramdisk.img -rw-rw-r-- 1 tgayet tgayet 6719698 Nov 11 19:37 out/target/product/sailfish/ramdisk-recovery.img -rw-rw-r-- 1 tgayet tgayet 867983588 Nov 11 19:52 out/target/product/sailfish/system.img -rw-rw-r-- 1 tgayet tgayet 230526804 Nov 11 19:49 out/target/product/sailfish/system_other.img -rw-r--r-- 1 tgayet tgayet 152755084 Nov 11 19:37 out/target/product/sailfish/userdata.img -rw-r--r-- 1 tgayet tgayet 240949220 Nov 11 19:37 out/target/product/sailfish/vendor.img

I cannot see any recovery image neither the cache image ?

Par contre il y a une nouvelle partition : system_other.img : son usage reste à définir !

Flashing images :

We can unlock the mobile but a warning message appear on boot (same message since 5x/6p)

--> Enabling the OEM UNLOCK + USB DEBUG adb reboot-bootloader fastboot devices fastboot oem unlock ou fastboot flashing unlock fastboot flash boot boot.img fastboot flash system system.img fastboot flash userdata userdata.img fastboot flash vendor vendor.img fastboot reboot

About the system-_other.img partition :

mbp-perso-thierry-gayet:Documents tgayet$ fastboot flash system_other system_other.img target reported max download size of 536870912 bytes sending 'system_other' (225123 KB)... OKAY [ 6.420s] writing 'system_other'... FAILED (remote: partition table doesn't exist) finished. total time: 6.513s

http://static.googleusercontent.com/media/source.android.com/fr//compatibility/android-cdd.pdf

Regards

Thierry GAYET Mobile : +33.663.849.589 Skype : terranova44 / tux35220 GPS : N 48° 6.375' W 1° 23.365' Address : 40 Rue de la Janaie, F-35220 Châteaubourg FRANCE Member of the GNU/Linux Fundation : https://www.linuxfoundation.org/users/thierryg35 www.nextinnovation.org

2016-11-12 8:38 GMT+01:00 Anestis Bechtsoudis notifications@github.com:

The Makefile to generate vendor.img is dynamically constructed based on the contents extracted from factory image archive. So if you edit the execute-all.sh script and add the codenames for Pixel devices at the availDevices array at the top you should have a vendor.img output.

However, someone still needs to dig deeper and see which elements are missing from /system for the two new devices (hopefully they will be the same). Plus needs to trace again which portions of the AOSP device configuration are not aligned with production releases (e.g. the dm-verity disabled at aosp fstab https://android.googlesource.com/device/google/marlin/+/android-7.1.0_r7/fstab.aosp_common) and hack around them.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/anestisb/android-prepare-vendor/issues/44#issuecomment-260107571, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcgtAs-rSTpKYbeS8XcG5s6UmIqCgD3ks5q9WzzgaJpZM4Ku-T9 .

anestisb commented 7 years ago

Have started working the Pixel support as I've managed to get my hands to sailfish device.

Also for the record since I've noticed the errors at your logs too, you need to use the fastboot host binary as compiled from AOSP tree since the SDK version is outdated. This important due to new partitions layout at Pixel devices.

thierryg commented 7 years ago

2016-12-17 19:57 GMT+01:00 Anestis Bechtsoudis notifications@github.com:

Have started working the Pixel support as I've managed to get my hands to sailfish device.

Also for the record since I've noticed the errors at your logs too, you need to use the fastboot host binary as compiled from AOSP tree since the SDK version is outdated. This important due to new partitions layout at Pixel devices.

hi, here is my last knowledge on pixel.

Will you provide a version of your script that support Pixel and Pixel XL ?

Regards

Thierry GAYET Mobile : +33.663.849.589 Skype : terranova44 / tux35220 GPS : N 48° 6.375' W 1° 23.365' Address : 40 Rue de la Janaie, F-35220 Châteaubourg FRANCE Member of the GNU/Linux Fundation : https://www.linuxfoundation.org/users/thierryg35 www.nextinnovation.org

anestisb commented 7 years ago

As you notice in the commits history I've started working the baseline for Pixel support. So far progress includes:

Generation of /vendor partition from matching factory images
Include correct fstab config that enables dm-verity for both /system & /vendor
Vendor blobs configs under /system
Updating the scripts to support functional differences between Nexus & Pixel (e.g. partitions layout)

I've made a release build for Pixel with 7.1.1_r4 tag using the output of the current master and so far no major issues occurred. I've noticed some errors in debug log so I'm chasing them now to identify the root cause. Also I need to track down which APKs need to be repaired and copied along with vendor blobs. So far the scripts are only including framework jars and not any APKs.

Along with some testing I hope to add stable support within the month. Of course additional hands / eyes for debugging and testing are more than welcome.

Now when it comes to Pixel XL, since I don't have a device yet, someone else needs to take care of testing when Pixel support is completed. In theory they should be identical with just different board name, but you never know.

thierryg commented 7 years ago

Hi, i have started to pull your work and i will test it.

Le vendor repaking works fine :

$ git clone https://github.com/anestisb/android-prepare-vendor.git && cd android-prepare-vendor/ Clonage dans 'android-prepare-vendor'... remote: Counting objects: 1169, done. remote: Compressing objects: 100% (16/16), done. remote: Total 1169 (delta 3), reused 0 (delta 0), pack-reused 1151 Réception d'objets: 100% (1169/1169), 5.92 MiB | 3.03 MiB/s, done. Résolution des deltas: 100% (769/769), done. Vérification de la connectivité... fait. android-prepare-vendor/ $ $ PATH=/home/tgayet/_aosp-git-repos/android-prepare-vendor/_tmp-jdk/jdk1.8.0_101/bin:$PATH ; ./execute-all.sh --device sailfish --buildID NMF26Q --output $(pwd) --yes [*] Setting output base to '/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q'

--{ Google Terms and Conditions Downloading of the system image and use of the device software is subject to the Google Terms of Service [1]. By continuing, you agree to the Google Terms of Service [1] and Privacy Policy [2]. Your downloading of the system image and use of the device software may also be subject to certain third-party terms of service, which can be found in Settings > About phone > Legal information, or as otherwise provided.

[1] https://www.google.com/intl/en/policies/terms/ [2] https://www.google.com/intl/en/policies/privacy/

[?] I have read and agree with the above terms and conditions - ACKNOWLEDGE [y|n]: yes [*] Downloading image from 'https://dl.google.com/dl/android/aosp/sailfish-nmf26q-factory-a84e0e4b.zip' --2016-12-19 15:05:24-- https://dl.google.com/dl/android/aosp/sailfish-nmf26q-factory-a84e0e4b.zip Résolution de dl.google.com (dl.google.com)… 216.58.213.142, 2a00:1450:4007:810::200e Connexion à dl.google.com (dl.google.com)|216.58.213.142|:443… connecté. requête HTTP transmise, en attente de la réponse… 200 OK Taille : 1891088718 (1.8G) [application/zip] Enregistre : «/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q/sailfish-nmf26q-factory-a84e0e4b.zip»

100%[==========================================================================================================================================================================================>] 1,891,088,718 3.61MB/s ds 10m 15s

2016-12-19 15:15:39 (2.93 MB/s) - «/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q/sailfish-nmf26q-factory-a84e0e4b.zip» enregistré [1891088718/1891088718]

[] Extracting '/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q/sailfish-nmf26q-factory-a84e0e4b.zip' [] Unzipping 'image-sailfish-nmf26q.zip' [] Processing with 'API-25 config-naked' configuration [] First run detected - downloading oatdump host bin & lib dependencies --2016-12-19 15:16:19-- https://onedrive.live.com/download?cid=D1FAC8CC6BE2C2B0&resid=D1FAC8CC6BE2C2B0%21503&authkey=AKDpBAzhzum6d7w Résolution de onedrive.live.com (onedrive.live.com)… 204.79.197.217 Connexion à onedrive.live.com (onedrive.live.com)|204.79.197.217|:443… connecté. requête HTTP transmise, en attente de la réponse… 302 Found Emplacement : https://7vu9xq.bl3301.livefilestore.com/y3mqIWTUEjKFUKyGjWcx1n5jfEC30acQXgTpnDop-YhOYFMnAZdIn_WhCY9LwO7dABOFFDnuHAczdwgJJbQs383xSKnJCSuR3Sni6qlsLKFExWVLc8R977uCNnr86Vju2Pju2fb28LUMg-n9g0a-gZYeEOuY4JplBnXPCzMaZR7EQ8/Linux_oatdump_bin_deps.zip?download&psid=1 [suivant] --2016-12-19 15:16:20-- https://7vu9xq.bl3301.livefilestore.com/y3mqIWTUEjKFUKyGjWcx1n5jfEC30acQXgTpnDop-YhOYFMnAZdIn_WhCY9LwO7dABOFFDnuHAczdwgJJbQs383xSKnJCSuR3Sni6qlsLKFExWVLc8R977uCNnr86Vju2Pju2fb28LUMg-n9g0a-gZYeEOuY4JplBnXPCzMaZR7EQ8/Linux_oatdump_bin_deps.zip?download&psid=1 Résolution de 7vu9xq.bl3301.livefilestore.com (7vu9xq.bl3301.livefilestore.com)… 204.79.197.213 Connexion à 7vu9xq.bl3301.livefilestore.com (7vu9xq.bl3301.livefilestore.com)|204.79.197.213|:443… connecté. requête HTTP transmise, en attente de la réponse… 200 OK Taille : 74173216 (71M) [application/zip] Enregistre : «/home/tgayet/_aosp-build-tests/android-prepare-vendor/hostTools/Linux/api-25/oatdump_deps.zip»

100%[============================================================================================================================================================================================>] 74,173,216 4.31MB/s ds 20s

2016-12-19 15:16:48 (3.61 MB/s) - «/home/tgayet/_aosp-build-tests/android-prepare-vendor/hostTools/Linux/api-25/oatdump_deps.zip» enregistré [74173216/74173216]

[] '4' bytecode archive files will be repaired [] Repairing bytecode under /system partition using oatdump method [!] '/framework/embmslibrary.jar' not pre-optimized with sanity checks passed - copying without changes [!] '/framework/rcsservice.jar' not pre-optimized with sanity checks passed - copying without changes [] System partition successfully extracted & repaired at '/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q/factory_imgs_repaired_data' [] Generating blobs for vendor/google_devices/sailfish [] Copying radio files '/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q/vendor/google_devices/sailfish' [] Copying product files & generating 'sailfish-vendor-blobs.mk' makefile [] Generating 'device-vendor.mk' [] Generating 'AndroidBoardVendor.mk' [] Bootloader:8996-012001-1610061102 [] Baseband:8996-012511-1611190200 [] Generating 'BoardConfigVendor.mk' [] Generating 'vendor-board-info.txt' [] Generating 'Android.mk' [] Gathering data from 'vendor/app' APK/JAR pre-builts [] Gathering data from 'vendor/framework' APK/JAR pre-builts [] Gathering data from 'proprietary/framework' APK/JAR pre-builts [] Processing standalone symlinks [] Generating signatures file [*] All actions completed successfully

[*] Import '/home/tgayet/_aosp-build-tests/android-prepare-vendor/sailfish/nmf26q/vendor' to AOSP root

Regards

Thierry GAYET Mobile : +33.663.849.589 Skype : terranova44 / tux35220 GPS : N 48° 6.375' W 1° 23.365' Address : 40 Rue de la Janaie, F-35220 Châteaubourg FRANCE Member of the GNU/Linux Fundation : https://www.linuxfoundation.org/users/thierryg35 www.nextinnovation.org

2016-12-19 12:17 GMT+01:00 Anestis Bechtsoudis notifications@github.com:

As you notice in the commits history I've started working the baseline for Pixel support. So far progress includes:

Generation of /vendor partition from matching factory images

Include correct fstab config that enables dm-verity for both /system & /vendor

Vendor blobs configs under /system

Updating the scripts to support functional differences between Nexus & Pixel (e.g. partitions layout)

I've made a release build for Pixel with 7.1.1_r4 tag using the output of the current master and so far no major issues occurred. I've noticed some errors in debug log so I'm chasing them now to identify the root cause. Also I need to track down which APKs need to be repaired and copied along with vendor blobs. So far the scripts are only including framework jars and not any APKs.

Along with some testing I hope to add stable support within the month. Of course additional hands / eyes for debugging and testing are more than welcome.

Now when it comes to Pixel XL, since I don't have a device yet, someone else needs to take care of testing when Pixel support is completed. In theory they should be identical with just different board name, but you never know.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/anestisb/android-prepare-vendor/issues/44#issuecomment-267940686, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcgtMGvXj2cw8vddAlp6AA24dLAFuKwks5rJmfFgaJpZM4Ku-T9 .

anestisb commented 7 years ago

Sailfish device feature checks

[x] WiFi
[x] Bluetooth
[x] 3g/4g
[x] Voice/SMS
[x] Camera
[x] Hotspot
[x] Fingerprint
[x] WiFi

Identified issues

[x] Fingerprint is broken (cannot enroll new fingerprint)

ghost commented 7 years ago

Hi anestisb and thierryg ! I just finished the aosp_sailfish-userdebug build of android-7.1.1_r6 including the pixel binaries from google_devices-sailfish-nmf26q-f17dae0a.tgz and qcom-sailfish-nmf26q-a2cce069.tgz. It took more than 5 hours with 8Gb of Ram, i7-3520M CPU @ 2.90GHz. I have used make -j4 and SERVER_NB_COMPILE=2 (in ~/.jack) and jack.server.max-service=2 (in ~/.jack-server/config.properties) and a ccache limit size of 25Gb.

At the end, I have a 44Gb build including :

find ./ -name "*.img" -exec du -h {} \;
1.5M    ./target/product/sailfish/ramdisk.img
25M ./target/product/sailfish/boot.img
231M    ./target/product/sailfish/vendor.img
832M    ./target/product/sailfish/system.img
146M    ./target/product/sailfish/userdata.img
832M    ./target/product/sailfish/obj/PACKAGING/systemimage_intermediates/system.img
220M    ./target/product/sailfish/system_other.img
6.5M    ./target/product/sailfish/ramdisk-recovery.img
84K ./target/product/sailfish/system/etc/firmware/rampatch_tlv.img

I have flashed ramdisk boot vendor system userdata with ./host/linux-x86/bin/fastboot and I have rebooted. I have the usual verify boot warning at the beginning. Instead of just the Google logo, I have now the Google logo, then the Android logo. Then I have a Warning :

Android System
There's an internal problem with your device. Contact your manufacturer for details.

Then I have the phone with 17 applications : Android Keyboard (AOSP) , Calculator , Calendar , Camera , Clock , Contacts , Email , Files , Gallery , Launcher3 , Messaging , Music , Phone , QXDM Logger version2 , Search , Settings , Sim Toolkit , Webview Shell

Wifi, Bluetooth and Camera seems to work. I have no nano sim card available however to test 3G/4G.

So , in conclusion, I have not used anything from this github project android-prepare-vendor and I have noticed anestisb comment in README.md :

Status update (17 Dec 2016)

As of 7.1 release Google has started publishing again a set of vendor blobs for supported Nexus & Pixel devices. Unfortunately the distributed blobs still miss some functionality when compiled under AOSP:

    Vendor partition is distributed in a form that does not allow to enable verified boot (dm-verity) against it
    Distributed blobs do not include APK bytecode vendor packages, only some jar files. It is still unclear to what extend device functionalities are broken.

So if I except the verify boot warning and the strange Android System warning.... does it mean that I have now a functional pixel as if I have flashed directly the Factory ROM sailfish-nmf26q-factory-a84e0e4b.zip ? Does this mean that android-prepare-vendor is not anymore relevant for Google Pixel phones ? Thanks, Cheers

ghost commented 7 years ago

Also rebuilded the android-msm-marlin-3.18-nougat-mr1 branch of the Linux kernel/msm repository. Then in the AOSP source directory : make bootimage Then in the AOSP output directory :

./host/linux-x86/bin/fastboot flashall -w -p sailfish
[...]
rebooting...

finished. total time: 64.628s

I still have the strange :

Android System
There's an internal problem with your device. Contact your manufacturer for details.

thierryg commented 7 years ago

2016-12-31 1:55 GMT+01:00 kulamfm notifications@github.com:

Android System There's an internal problem with your device. Contact your manufacturer for details.

Hi,

the following problem :

"Android System

There's an internal problem with your device. Contact your manufacturer for details."

is because we do not rebuild and update the vendor image.

Indeed, at runtime, there is a check between the system/build.prop and vendor/build.prop images ; it they do not match an error message is displayed. If you want to bypass it, just get the vendor/build.prop file and comment with a '#' the three lines, then remount the vendor partition and pull it on the same location.

Here is some detail :

Hacking the Android system popup

After rebuilding an image for the bullhead device, a system popup can be viewed :

Android System

There's an internal problem with your device. Contact your manufacturer for details.

The problem occurs because of a check that google implemented in Android 5.1 which compares /system/build.prop with the values found in /vendor/build.prop.

If they differ you get that error message. All one has to do to get rid of the error is to change the 3 values in /vendor/build.prop according to the values in /system/build.prop.

First prepare a build.prop file that comment all lines :

$ nano build.prop

ro.vendor.build.date=Wed Oct 26 00:45:01 UTC 2016

ro.vendor.build.date.utc=1477442701

ro.vendor.build.fingerprint=google/sailfish/sailfish:7.1/NDE63X/3399619:userdebug/dev-keys

Then follow the above steps :

$ adb root $ adb disable-verity $ adb reboot $ adb remount

$ adb pull /vendor/build.prop .

$ nano ./build.prop --> comment the three lines as below :

ro.vendor.build.date=Wed Oct 26 00:45:01 UTC 2016

ro.vendor.build.date.utc=1477442701

ro.vendor.build.fingerprint=google/sailfish/sailfish:7.1/NDE63X/3399619:userdebug/dev-keys

$ adb push build.prop /vendor/build.prop

$ adb reboot

This hack will have to be done one time and push through the vendor.img image !

http://forum.xda-developers.com/nexus-9/development/fix-build-prop-variety-fix-aka-contact-t3133347/page3

Regards

Thierry GAYET Mobile : +33.663.849.589 Skype : terranova44 / tux35220 GPS : N 48° 6.375' W 1° 23.365' Address : 40 Rue de la Janaie, F-35220 Châteaubourg FRANCE Member of the GNU/Linux Fundation : https://www.linuxfoundation.org/users/thierryg35 www.nextinnovation.org

thestinger commented 7 years ago

There's still clearly a need for android-prepare-vendor despite incomplete blob releases missing functionality and unable to rebuild vendor.img for verity to work. I don't think it makes sense to use this issue to talk about using Google's incomplete releases. If you don't want a fully working build with verified boot and nothing missing or mismatched from a proper build then you don't really need android-prepare-vendor. This project is the only feasible way to get proper robust builds that once signed and turned into factory image bundles and over-the-air updates are actually production ready. On the other hand, getting a mostly working build with unresolved issues doesn't require using this project.

thestinger commented 7 years ago

The only use for their releases would be avoiding some cases of needing to deal with optimized code. However, they haven't been reliability doing the driver releases. Look at the Pixel in https://developers.google.com/android/drivers. It's missing NMF26Q. There's also no guarantee that these really line up with the stuff that they're releasing. It's not a focus for them, unlike the factory images and full OTA updates they publish which are not just for people doing their own builds.

thierryg commented 7 years ago

Yes i was hoping that with the pixel series fully from google, everything would be provided. Since i have problem both with pixel (sailfish) and pixel xl (marlin), i am using the android-prepare-vendor script.

First, there is the problem with the vendor image that is not rebuild and or update the properties on it.

Second, since pixel C (tablet) no kernel source code is provided on the android git repo. This is because the the kernel is now signed and must be prepared. I am trying to rebuild a kernel version in order to have a new security chain. Indeed, the ugly warning message on boot is orange if we lock the device. This is because we must provide an OEM-key. If someone have fixe the initial boot screen, i ma interested 'cos i want to be sure if i am on the right way or not.

However, i have also seen that lot of selinux are not provided 'cos when we rebuild and flash a new build, we have lot of avc exception. I have fixed most of them on my side. I can provide them as patch if you want to. Maybe we can used the quilt tool in order to apply a set of patch (tell me if you are interested to add them to android-prepare-vendor).

Regards

Thierry GAYET Mobile : +33.663.849.589 Skype : terranova44 / tux35220 GPS : N 48° 6.375' W 1° 23.365' Address : 40 Rue de la Janaie, F-35220 Châteaubourg FRANCE Member of the GNU/Linux Fundation : https://www.linuxfoundation.org/users/thierryg35 www.nextinnovation.org

2016-12-31 14:23 GMT+01:00 Daniel Micay notifications@github.com:

The only use for their releases would be avoiding some cases of needing to deal with optimized code. However, they haven't been reliability doing the driver releases. Look at the Pixel in https://developers.google.com/ android/drivers. It's missing NMF26Q. There's also no guarantee that these really line up with the stuff that they're releasing. It's not a focus for them, unlike the factory images and full OTA updates they publish which are not just for people doing their own builds.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/anestisb/android-prepare-vendor/issues/44#issuecomment-269864849, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcgtEV8duYzhaKDczkLyAh6PCkzqy1fks5rNldcgaJpZM4Ku-T9 .

thestinger commented 7 years ago

Many SELinux audit warnings are expected and extending the policy with new permissions to silence them is incorrect. Some programs do things that are not permitted or silenced with dontaudit but it doesn't mean there is a bug. They might simply be doing something like scanning over a directory and it's totally fine if they can't access everything. There are so many reasons that audit warnings can be expected but not bugs. Stock Android has them too. Unless it only occurs on AOSP, then there's no problem. If it does only occur on AOSP, you need to understand why it's happening before you can determine the correct solution which may very well be not doing anything or just using dontaudit if it's particularly spammy.

thestinger commented 7 years ago

Also, Pixel C sources are provided.

Verified boot also works fine via AOSP using android-prepare-vendor. You need to properly sign your releases. If you want a proper building guide for production releases follow https://copperhead.co/android/docs/building. It applies to AOSP too, once you've generated the vendor directory with android-prepare-vendor. Building the kernel is a separate step for AOSP though.

ghost commented 7 years ago

Hi anestisb and thierryg !

In my previous post, I said that I had the orange warning... but I forgot to relock ./host/linux-x86/bin/fastboot oem lock

Anyway, thanks for your replies and sorry for my newbie questions, this is my first Smartphone (and so my first Android phone, AOSP build/flash...)

So this time, I used your script ./execute-all.sh -d sailfish -b nmf26q -i ../FACTORYROM/sailfish-nmf26q-factory-a84e0e4b.zip -o out --yes

And I rebuild AOSP with the new vendor directory. Can you confirm me that I dont need to redo all the build with a make clobber ? I took only 3 minutes and now I have

find ./ -name "*.img" -exec du -h {} \;
1.5M    ./target/product/sailfish/ramdisk.img
25M ./target/product/sailfish/boot.img
230M    ./target/product/sailfish/vendor.img
832M    ./target/product/sailfish/system.img
146M    ./target/product/sailfish/userdata.img
832M    ./target/product/sailfish/obj/PACKAGING/systemimage_intermediates/system.img
220M    ./target/product/sailfish/system_other.img
6.5M    ./target/product/sailfish/ramdisk-recovery.img
200K    ./target/product/sailfish/vendor/firmware/synaptics_bl71.img
84K ./target/product/sailfish/system/etc/firmware/rampatch_tlv.img

I flashed, rebooted, locked.

Now I dont have anymore the

Android System
There's an internal problem with your device. Contact your manufacturer for details.

warning.

But now I have the yellow warning https://support.google.com/nexus/answer/6185381?hl=en

About the /vendor/build.prop edition, if I understand correctly, I dont need it anymore because the purpose of android-prepare-vendor is precisely to rebuild /vendor during the build of /system ?

This project is the only feasible way to get proper robust builds that once signed and turned into factory image bundles and over-the-air updates are actually production ready. On the other hand, getting a mostly working build with unresolved issues doesn't require using this project.

=> ok thanks, its more clear now

Look at the Pixel in https://developers.google.com/android/drivers. It's missing NMF26Q.

=> I dont understand, isn't it just an error in their webpage (they forgot for sailfish to write Pixel binaries for Android 7.1.1 (NMF26Q)) but there is both the NMF26O and NMF26Q in the links (as for marlin) : https://dl.google.com/dl/android/aosp/google_devices-sailfish-nmf26o-d5e765ee.tgz https://dl.google.com/dl/android/aosp/qcom-sailfish-nmf26o-218fc720.tgz https://dl.google.com/dl/android/aosp/google_devices-sailfish-nmf26q-f17dae0a.tgz https://dl.google.com/dl/android/aosp/qcom-sailfish-nmf26q-a2cce069.tgz

but the argument

There's also no guarantee that these really line up with the stuff that they're releasing.

remains relevant...

Thanks

thierryg commented 7 years ago

Hi Daniel,

Thanks for your response abour the selinux rules.

Thanks for your link about copperhead 'cos i was looking for why ota packages didn't generate properly. Indeed, i didn't do the following command :

make generate_verity_key

make brillo_update_payload

What the last one do exactly ?

Now on my pixel/pixel xl, i can rebuild all images including vendors prepare from your script, i can generate update package, i can generate ota.

When i flash them and lock my device, i still have a warning on boot (yellow) that say that we are missing an OEM KEY :

https://source.android.com/security/images/verified_boot.png https://source.android.com/security/verifiedboot/verified-boot.html

Does the OEM key is the verity key ? Where does the OEM key need to be set ?

Is it because the gnu/linux is now signed ? Does someone have a way that describe how to sign a kernel for pixel ?

Does someone know how to solve the daisy security chain (by verity) ?

Thanks in advance ?

Regards

Thierry GAYET

2016-12-31 14:45 GMT+01:00 Daniel Micay notifications@github.com:

Also, Pixel C sources are provided.

Verified boot also works fine via AOSP using android-prepare-vendor. You need to properly sign your releases. If you want a proper building guide for production releases follow https://copperhead.co/android/docs/building. It applies to AOSP too, once you've generated the vendor directory with android-prepare-vendor. Building the kernel is a separate step for AOSP though.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/anestisb/android-prepare-vendor/issues/44#issuecomment-269865679, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcgtB_9K0MpRqddtgBw06qfNk3IcDoiks5rNlxxgaJpZM4Ku-T9 .

thierryg commented 7 years ago

About the OEM key i have found this information :

OEM key The OEM key is a fixed, tamper-protected key available to the bootloader that must be used to verify the boot image. Source : https://source.android.com/security/verifiedboot/verified-boot.html

According to the digram (https://source.android.com/ security/images/verified_boot.png), my state can be explain like this :

YELLOW : indicating the boot partition has been verified using the embedded certificate, and the signature is valid. The bootloader displays a warning and the fingerprint of the public key before allowing the boot process to continue.

Is there a way to remove it ?

Regards

Thierry GAYET Mobile : +33.663.849.589 Skype : terranova44 / tux35220 GPS : N 48° 6.375' W 1° 23.365' Address : 40 Rue de la Janaie, F-35220 Châteaubourg FRANCE Member of the GNU/Linux Fundation : https://www.linuxfoundation.org/users/thierryg35 www.nextinnovation.org

2017-01-02 11:14 GMT+01:00 Thierry GAYET thierry.gayet@gmail.com:

Hi Daniel,

Thanks for your response abour the selinux rules.

Thanks for your link about copperhead 'cos i was looking for why ota packages didn't generate properly. Indeed, i didn't do the following command :

make generate_verity_key

make brillo_update_payload

What the last one do exactly ?

Now on my pixel/pixel xl, i can rebuild all images including vendors prepare from your script, i can generate update package, i can generate ota.

When i flash them and lock my device, i still have a warning on boot (yellow) that say that we are missing an OEM KEY :

https://source.android.com/security/images/verified_boot.png https://source.android.com/security/verifiedboot/verified-boot.html

Does the OEM key is the verity key ? Where does the OEM key need to be set ?

Is it because the gnu/linux is now signed ? Does someone have a way that describe how to sign a kernel for pixel ?

Does someone know how to solve the daisy security chain (by verity) ?

Thanks in advance ?

Regards

Thierry GAYET

2016-12-31 14:45 GMT+01:00 Daniel Micay notifications@github.com:

Also, Pixel C sources are provided.

Verified boot also works fine via AOSP using android-prepare-vendor. You need to properly sign your releases. If you want a proper building guide for production releases follow https://copperhead.co/android/ docs/building. It applies to AOSP too, once you've generated the vendor directory with android-prepare-vendor. Building the kernel is a separate step for AOSP though.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/anestisb/android-prepare-vendor/issues/44#issuecomment-269865679, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcgtB_9K0MpRqddtgBw06qfNk3IcDoiks5rNlxxgaJpZM4Ku-T9 .

thestinger commented 7 years ago

You can't remove the yellow verified boot screen. It indicates that a third party key is being used and that the OS verified correctly. It will be red if verified boot fails, and decryption of the data will fail if the key is changed. It's better than it being orange though, which means boot/recovery aren't being verified by the bootloader. The only way to have the green status (no verified boot status screen) is using the key that's hard-wired into the bootloader.

The best you can do is generating proper release keys + signed factory images / updates / incremental updates. For incremental updates, you just use a similar OTA generation command specifying key, etc. to ota_from_target_files with -i old_target_files.zip added too.

The brillo_update_payload target is needed to generate the new A/B style updates used by the Pixel and Pixel XL. No need for that with earlier devices.

thestinger commented 7 years ago

Note that if you're using that release.sh, you're already signing everything including the kernel by signing recovery/boot/system/vendor, all of the included apps, the MAC policy signatures, etc. and then finally the OTA generation script also takes a key, which signs it with the release key that the recovery image will verify.

vamsirocks commented 7 years ago

Dear

I had recently successfully build the latest source code with build id NMF26Q and flashed into Google pixel device , but i am unable to get mobile data connectivity. I had checked the APN and other settings , everything seems to be perfect still mobile data connection not working. I am able to get mobile signal network and able to call and send sms too, but no internet. So please kindly provide any solution or whether i had missed out any build property during build. I had taken proprietary binaries too provided by google for google pixel for corresponding build. Based on inclusion of proprietary binaries i got vendor.img too but no mobile data please help me.

Please provide me solution earliest possible

Thank you Vamsi Donkada

anestisb commented 7 years ago

@vamsirocks ok will have a look at it. Which is your data plan carrier? From time to time I come across with US carriers specific bugs that is not easy to reproduce so we'll probably have to work it together.

vamsirocks commented 7 years ago

India carrier Airtel. AOSP claimed that there exists some space issues in APN types in previou builds which are fixed in build NMF26Q . So I am using build NMF26Q still issue of data connection exists. Fingerprint issue also there but I had resolved successfully

peterknowles commented 7 years ago

@vamsirocks , could you please explain what you mean about resolving the fingerprint issue? That isn't something that I've been able to get working (on a Marlin device). I'm simply not able to register any fingerprints at all. Was wondering if you got it working and how you achieved that?

Azlun commented 7 years ago

@thierryg I guess userdebug build will always have such problem, I am going to do some test, can you share your Sailfish and Marlin userdebug images to me by somewhere, then I can test and feedback

vamsirocks commented 7 years ago

@peterknowles . at the end of the device-marlin.mk file(~/AOSP/device/google/marlin/device-marlin.mk) file add following lines properly and rebuild complete AOSP. Which will resolve issue

#fingerpint

PRODUCT_PROPERTY_OVERRIDES += \
   ro.hardware.fingerprint=fpc

anestisb commented 7 years ago

Thanks @vamsirocks for the info. I'll verify it fixes the Marlin/Sailfish fingerprint issue and push it by default at vendor generation makefiles.

Unfortunately I'm quite busy at the moment, so no progress has been made at chasing the Marlin support issues. Have you noticed any other functionality issues?

peterknowles commented 7 years ago

Thanks a lot @vamsirocks

That worked perfectly for resolving the fingerprint reader on my Marlin device. The only remaining problem that I appear to have is the exact same one that you do (i.e. no cellular data). The APN is definitely correct (copied over exactly from my working Angler device).

I've tried with NMF26V and NMF26U and it stubbornly refuses to work with either. Like you, I get cellular signal and SMS, but data just isn't working at all.

thestinger commented 7 years ago

I can confirm that setting that property for the fingerprint reader fixes it, and that property is set on stock Android.

thestinger commented 7 years ago

An issue that's happening here is that the Flashlight icon in Quick Settings is greyed out / disabled. I'm not sure if that's an AOSP / android-prepare-vendor issue or a CopperheadOS issue yet.

anestisb commented 7 years ago

Thanks everyone for the reported issues. Everything mentioned in this thread has been fixed, including an auto-generated PRODUCT_PROPERTY_OVERRIDE for the fingerprint fix that doesn't require editing the AOSP sources.

I've extensively tested the latest master against sailfish release builds and so far I haven't notice any issues. Unfortunately I don't have a Pixel XL device to test marlin configs too.

If you want to build a ROM with Verizon or Sprint carriers support, use the gplay device configs (-g flag) since the APKs/JARs matching these specific vendors are not included in naked config profiles.

Please open a new issue in case you notice any broken or missing functionality that possibly roots to the vendor files.

thestinger commented 7 years ago

Do you run into #58?

anestisb commented 7 years ago

@thestinger flashlight seems to work fine at the moment (master run against NOF26V build)when accessed from the quick settings.

Does the problem occur after an OTA update, or is constantly broken for Copperhead marlin builds?

thestinger commented 7 years ago

Maybe if you apply an OTA update, boot into the OS and then reboot. It might be a Copperhead issue though, or some local issue with my build environment. I could try building AOSP + android-prepare-vendor alone.

vamsirocks commented 7 years ago

Is mobile data issue resolved. Please help me in resolving mobile data issue @anestisb

anestisb commented 7 years ago

@vamsirocks it's hard for me to confirm issues for carriers I don't have access to. Have you tried building AOSPO with the the latest master? Do you still have issues? If so please open a new issue with your setup details (device, build num, carrier APN name) and I'll have a look at it.

vamsirocks commented 7 years ago

Thank you very much . Issue resolved after following your vendor generation procedure. Thank you very very much @anestisb

alengh commented 5 years ago

Any help with OEM unlock Disabled ? The Build is signed and did the mistake of re-lock the bootloader :(

Because i did signe the Build i am no longer able to (ADB Sideload) to update with OTA or Stock zip

anestisb / android-prepare-vendor

Support for google Pixel (sailfish) and Pixel XL (marlin) #44

Sailfish device feature checks

Identified issues

ro.vendor.build.date=Wed Oct 26 00:45:01 UTC 2016

ro.vendor.build.date.utc=1477442701

ro.vendor.build.fingerprint=google/sailfish/sailfish:7.1/NDE63X/3399619:userdebug/dev-keys

ro.vendor.build.date=Wed Oct 26 00:45:01 UTC 2016

ro.vendor.build.date.utc=1477442701

ro.vendor.build.fingerprint=google/sailfish/sailfish:7.1/NDE63X/3399619:userdebug/dev-keys