Closed Rudd-O closed 5 years ago
@Rudd-O - you didn't include the majority of the details requested (version, command run, email output, etc), which are included in the template to help debug issues. I also have to ask if this is your modified version or standard latest version v9.0.6+.
This is modified, the build process is identical. I can show you my script, give you a diff with the build_template.go file, everything. Will disclose everything now, save for the massive 80 MB log.
My script:
#!/bin/bash
if [ $# -ne 1 ]; then
echo "Need to specify device name as argument"
exit 1
fi
# check if supported device
DEVICE=$1
if [ "$DEVICE" == 'sailfish' ] || [ "$DEVICE" == 'marlin' ] || [ "$DEVICE" == 'walleye' ] || [ "$DEVICE" == 'taimen' ]; then
echo "Supported device $DEVICE - continuing build"
else
echo "Unsupported device $DEVICE"
exit 1
fi
# set region
REGION=none
export AWS_DEFAULT_REGION=${REGION}
# stack name
STACK_NAME=rattlesnakeos
# version of stack running
STACK_VERSION=
# prevent default action of shutting down on exit
PREVENT_SHUTDOWN=
# force build even if no new versions exist of components
FORCE_BUILD=false
# skip chromium build if there is an existing build
SKIP_CHROMIUM_BUILD=false
# pin to specific version of android
ANDROID_VERSION="9.0"
# build type (user or userdebug)
BUILD_TYPE="user" # replaced
# build channel (stable or beta)
BUILD_CHANNEL="stable"
# user customizable things
REPO_PATCHES=
REPO_PREBUILTS=
HOSTS_FILE=
# get current instance details for reference
INSTANCE_TYPE=none
INSTANCE_REGION=none
INSTANCE_IP=none
# aws settings
AWS_KEYS_BUCKET="${STACK_NAME}-keys"
AWS_RELEASE_BUCKET="${STACK_NAME}-release"
AWS_LOGS_BUCKET="${STACK_NAME}-logs"
AWS_SNS_ARN=none
# build settings
BUILD_TARGET="release aosp_${DEVICE} ${BUILD_TYPE}"
RELEASE_URL="https://${AWS_RELEASE_BUCKET}.s3.amazonaws.com"
RELEASE_CHANNEL="${DEVICE}-${BUILD_CHANNEL}"
CHROME_CHANNEL="stable"
BUILD_DATE=$(date +%Y.%m.%d.%H)
BUILD_TIMESTAMP=$(date +%s)
BUILD_DIR="$HOME/rattlesnake-os"
CERTIFICATE_SUBJECT='/CN=RattlesnakeOS'
OFFICIAL_FDROID_KEY="43238d512c1e5eb2d6569f4a3afbf5523418b82e0a3ed1552770abb9a9c9ccab"
MARLIN_KERNEL_SOURCE_DIR="${BUILD_DIR}/kernel/google/marlin"
SECONDS=0
# urls
ANDROID_SDK_URL="https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip"
MANIFEST_URL="https://android.googlesource.com/platform/manifest"
CHROME_URL_LATEST="https://omahaproxy.appspot.com/all.json"
STACK_URL_LATEST="https://api.github.com/repos/dan-v/rattlesnakeos-stack/releases/latest"
FDROID_CLIENT_URL_LATEST="https://gitlab.com/api/v4/projects/36189/repository/tags"
FDROID_PRIV_EXT_URL_LATEST="https://gitlab.com/api/v4/projects/1481578/repository/tags"
KERNEL_SOURCE_URL="https://android.googlesource.com/kernel/msm"
STACK_UPDATE_MESSAGE=
LATEST_STACK_VERSION=
LATEST_CHROMIUM=
FDROID_CLIENT_VERSION=
FDROID_PRIV_EXT_VERSION=
AOSP_BUILD=
AOSP_BRANCH=
get_latest_versions() {
sudo apt-get -y install jq
# check if running latest stack
LATEST_STACK_VERSION=$(curl -s "$STACK_URL_LATEST" | jq -r '.name' || true)
if [ "$LATEST_STACK_VERSION" == "$STACK_VERSION" ]; then
echo "Running the latest rattlesnakeos-stack version $LATEST_STACK_VERSION"
else
STACK_UPDATE_MESSAGE="WARNING: you should upgrade to the latest version: ${LATEST_STACK_VERSION}"
fi
# check for latest stable chromium version
LATEST_CHROMIUM=$(curl -s "$CHROME_URL_LATEST" | jq -r '.[] | select(.os == "android") | .versions[] | select(.channel == "'$CHROME_CHANNEL'") | .current_version' || true)
if [ -z "$LATEST_CHROMIUM" ]; then
aws_notify_simple "ERROR: Unable to get latest Chromium version details. Stopping build."
exit 1
fi
# fdroid - get latest non alpha tags from gitlab
FDROID_CLIENT_VERSION=$(curl -s "$FDROID_CLIENT_URL_LATEST" | jq -r '[.[] | select(.name | test("^[0-9]+\\.[0-9]+")) | select(.name | contains("alpha") | not) | select(.name | contains("ota") | not)][0] | .name')
if [ -z "$FDROID_CLIENT_VERSION" ]; then
aws_notify_simple "ERROR: Unable to get latest F-Droid version details. Stopping build."
exit 1
fi
FDROID_PRIV_EXT_VERSION=$(curl -s "$FDROID_PRIV_EXT_URL_LATEST" | jq -r '[.[] | select(.name | test("^[0-9]+\\.[0-9]+")) | select(.name | contains("alpha") | not) | select(.name | contains("ota") | not)][0] | .name')
if [ -z "$FDROID_PRIV_EXT_VERSION" ]; then
aws_notify_simple "ERROR: Unable to get latest F-Droid privilege extension version details. Stopping build."
exit 1
fi
# attempt to automatically pick latest build version and branch. note this is likely to break with any page redesign. should also add some validation here.
AOSP_BUILD=$(curl -s https://developers.google.com/android/images | grep -A1 "${DEVICE}" | egrep '[a-zA-Z]+ [0-9]{4}\)' | grep "${ANDROID_VERSION}" | tail -1 | cut -d"(" -f2 | cut -d"," -f1)
if [ -z "$AOSP_BUILD" ]; then
aws_notify_simple "ERROR: Unable to get latest AOSP build information. Stopping build. This lookup is pretty fragile and can break on any page redesign of https://developers.google.com/android/images"
exit 1
fi
AOSP_BRANCH=$(curl -s https://source.android.com/setup/start/build-numbers | grep -A1 "${AOSP_BUILD}" | tail -1 | cut -f2 -d">"|cut -f1 -d"<")
if [ -z "$AOSP_BRANCH" ]; then
# TODO: temporary workaround until build-numbers are updated on website
if [ "$AOSP_BUILD" == "PPR2.181005.003" ]; then
AOSP_BRANCH="android-9.0.0_r10"
fi
if [ -z "$AOSP_BRANCH" ]; then
aws_notify_simple "ERROR: Unable to get latest AOSP branch information. Stopping build. This can happen if https://source.android.com/setup/start/build-numbers hasn't been updated yet with newly released factory images."
exit 1
fi
fi
}
check_for_new_versions() {
echo "Checking if any new versions of software exist"
needs_update=false
# check stack version
existing_stack_version=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/rattlesnakeos-stack/revision" - || true)
if [ "$existing_stack_version" == "$STACK_VERSION" ]; then
echo "Stack version ($existing_stack_version) is up to date"
else
echo "Last successful build (if there was one) is not with latest stack version ${STACK_VERSION}"
needs_update=true
fi
# check aosp
existing_aosp_build=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-vendor" - || true)
if [ "$existing_aosp_build" == "$AOSP_BUILD" ]; then
echo "AOSP build ($existing_aosp_build) is up to date"
else
echo "AOSP needs to be updated to ${AOSP_BUILD}"
needs_update=true
fi
# check chromium
if [ "$SKIP_CHROMIUM_BUILD" = false ]; then
existing_chromium=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/revision" - || true)
if [ "$existing_chromium" == "$LATEST_CHROMIUM" ]; then
echo "Chromium build ($existing_chromium) is up to date"
else
echo "Chromium needs to be updated to ${LATEST_CHROMIUM}"
needs_update=true
fi
else
echo "Skipping Chromium version check as SKIP_CHROMIUM_BUILD=true"
fi
# check fdroid
existing_fdroid_client=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/fdroid/revision" - || true)
if [ "$existing_fdroid_client" == "$FDROID_CLIENT_VERSION" ]; then
echo "F-Droid build ($existing_fdroid_client) is up to date"
else
echo "F-Droid needs to be updated to ${FDROID_CLIENT_VERSION}"
needs_update=true
fi
# check fdroid priv extension
existing_fdroid_priv_version=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/fdroid-priv/revision" - || true)
if [ "$existing_fdroid_priv_version" == "$FDROID_PRIV_EXT_VERSION" ]; then
echo "F-Droid privilege extension build ($existing_fdroid_priv_version) is up to date"
else
echo "F-Droid privilege extensions needs to be updated to ${FDROID_PRIV_EXT_VERSION}"
needs_update=true
fi
if [ "$needs_update" = true ]; then
aws_notify "New build is required"
else
if [ "$FORCE_BUILD" = true ]; then
aws_notify "No build is required, but FORCE_BUILD=true"
else
aws_notify "RattlesnakeOS build not required as all components are already up to date."
exit 0
fi
fi
}
full_run() {
get_latest_versions
check_for_new_versions
aws_notify "RattlesnakeOS Build STARTED"
setup_env
check_chromium
fetch_aosp_source
setup_vendor
aws_import_keys
apply_patches
# only marlin and sailfish need kernel rebuilt so that verity_key is included
if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
rebuild_marlin_kernel
fi
build_aosp
aws_release
checkpoint_versions
aws_notify "RattlesnakeOS Build SUCCESS"
}
setup_env() {
echo "=================================="
echo "Running setup_env"
echo "=================================="
# setup build dir
mkdir -p "$BUILD_DIR"
# install packages
sudo apt-get update
sudo apt-get --assume-yes install openjdk-8-jdk git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip python-networkx liblz4-tool
sudo apt-get --assume-yes build-dep "$(apt-cache search linux-image-* | awk ' { print $1 } ' | sort | egrep -v -- '(-dbg|-rt|-pae)' | grep ^linux-image-[0-9][.] | tail -1)"
sudo apt-get --assume-yes install repo gperf jq fuseext2
# setup android sdk (required for fdroid build)
mkdir -p ${HOME}/sdk
pushd ${HOME}/sdk
if [ ! -f sdk-tools.zip ] ; then
wget ${ANDROID_SDK_URL} -O sdk-tools.zip
fi
unzip -o sdk-tools.zip || {
echo unzip failed, retrying download
rm -f sdk-tools.zip
wget ${ANDROID_SDK_URL} -O sdk-tools.zip
unzip -o sdk-tools.zip
}
yes | ./tools/bin/sdkmanager --licenses
./tools/android update sdk -u --use-sdk-wrapper
# setup git
git config --get --global user.name || git config --global user.name 'unknown'
git config --get --global user.email || git config --global user.email 'unknown@localhost'
git config --global color.ui true
}
check_chromium() {
echo "=================================="
echo "Running check_chromium"
echo "=================================="
current=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/revision" - || true)
echo "Chromium current: $current"
if [ "$SKIP_CHROMIUM_BUILD" = true ]; then
if [ -z "$current" ]; then
echo "Can't skip Chromium build as requested as Chromium hasn't been built yet previously"
else
echo "Skipping Chromium build as requested"
aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
return
fi
fi
mkdir -p $HOME/chromium
cd $HOME/chromium
echo "Chromium latest: $LATEST_CHROMIUM"
if [ "$LATEST_CHROMIUM" == "$current" ]; then
echo "Chromium latest ($LATEST_CHROMIUM) matches current ($current) - just copying s3 chromium artifact"
aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
else
echo "Building chromium $LATEST_CHROMIUM"
build_chromium $LATEST_CHROMIUM
fi
}
build_chromium() {
echo "=================================="
echo "Running build_chromium"
echo "=================================="
CHROMIUM_REVISION=$1
DEFAULT_VERSION=$(echo $CHROMIUM_REVISION | awk -F"." '{ printf "%s%03d52\n",$3,$4}')
# depot tools setup
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git $HOME/depot_tools || true
export PATH="$PATH:$HOME/depot_tools"
# fetch chromium
mkdir -p $HOME/chromium
cd $HOME/chromium
test -f .gclient || fetch --nohooks android
cd src
# checkout specific revision
git checkout "$CHROMIUM_REVISION" -f
# install dependencies
echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | sudo debconf-set-selections
sudo ./build/install-build-deps-android.sh
# run gclient sync (runhooks will run as part of this)
for gitdir in $( find -name .git ) ; do
pushd $gitdir/.. || continue
git clean -dff || { popd ; return $? ; }
git reset --hard || { popd ; return $? ; }
popd || return $?
done
yes | gclient sync --with_branch_heads --jobs 32 -RDf
# cleanup any files in tree not part of this revision
git clean -dff
# reset any modifications
git checkout -- .
mkdir -p "$HOME"/chromium-out
cat <<EOF > "$HOME"/chromium-out/args.gn
target_os = "android"
target_cpu = "arm64"
is_debug = false
is_official_build = true
is_component_build = false
symbol_level = 0
ffmpeg_branding = "Chrome"
proprietary_codecs = true
android_channel = "stable"
android_default_version_name = "$CHROMIUM_REVISION"
android_default_version_code = "$DEFAULT_VERSION"
EOF
gn gen "$HOME"/chromium-out
autoninja -C "$HOME"/chromium-out/ monochrome_public_apk
aws s3 cp "$HOME"/chromium-out/apks/MonochromePublic.apk "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk"
echo "${CHROMIUM_REVISION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/chromium/revision"
}
fetch_aosp_source() {
echo "=================================="
echo "Running fetch_aosp_source"
echo "=================================="
pushd "${BUILD_DIR}"
repo init --manifest-url "$MANIFEST_URL" --manifest-branch "$AOSP_BRANCH" --depth 1 || true
for gitdir in $(find -name .git) ; do
pushd "$gitdir/.." || continue
git status || { popd ; return $? ; }
git clean -dff || { popd ; return $? ; }
git reset --hard || { popd ; return $? ; }
popd || return $?
done
# make modifications to default AOSP
# Since we just git cleaned everything, we will have to re-copy
# the MonochromePublic.apk file once again.
mkdir -p ${BUILD_DIR}/external/chromium/prebuilt/arm64
aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
if ! grep -q "RattlesnakeOS" .repo/manifest.xml; then
awk -i inplace \
-v ANDROID_VERSION="$ANDROID_VERSION" \
-v FDROID_CLIENT_VERSION="$FDROID_CLIENT_VERSION" \
-v FDROID_PRIV_EXT_VERSION="$FDROID_PRIV_EXT_VERSION" \
'1;/<repo-hooks in-project=/{
print " ";
print " <remote name=\"github\" fetch=\"https://github.com/RattlesnakeOS/\" revision=\"" ANDROID_VERSION "\" />";
print " <remote name=\"fdroid\" fetch=\"https://gitlab.com/fdroid/\" />";
print " <remote name=\"prepare-vendor\" fetch=\"https://github.com/anestisb/\" revision=\"master\" />";
print " ";
print " <project path=\"script\" name=\"script\" remote=\"github\" />";
print " <project path=\"external/chromium\" name=\"platform_external_chromium\" remote=\"github\" />";
print " <project path=\"packages/apps/Updater\" name=\"platform_packages_apps_Updater\" remote=\"github\" />";
print " <project path=\"packages/apps/F-Droid\" name=\"fdroidclient\" remote=\"fdroid\" revision=\"refs/tags/" FDROID_CLIENT_VERSION "\" />";
print " <project path=\"packages/apps/F-DroidPrivilegedExtension\" name=\"privileged-extension\" remote=\"fdroid\" revision=\"refs/tags/" FDROID_PRIV_EXT_VERSION "\" />";
print " <project path=\"vendor/android-prepare-vendor\" name=\"android-prepare-vendor\" remote=\"prepare-vendor\" />"}' .repo/manifest.xml
else
echo "Skipping modification of .repo/manifest.xml as they have already been made"
fi
# remove things from manifest
sed -i '/chromium-webview/d' .repo/manifest.xml
sed -i '/packages\/apps\/Browser2/d' .repo/manifest.xml
sed -i '/packages\/apps\/Calendar/d' .repo/manifest.xml
sed -i '/packages\/apps\/QuickSearchBox/d' .repo/manifest.xml
# sync with retries
for i in {1..10}; do
repo sync -c --no-tags --no-clone-bundle --jobs 32 && break
done
# remove webview
rm -rf platform/external/chromium-webview
sed -i '/webview \\/d' build/make/target/product/core_minimal.mk
# remove Browser2
sed -i '/Browser2/d' build/make/target/product/core.mk
# remove Calendar
sed -i '/Calendar \\/d' build/make/target/product/core.mk
# remove QuickSearchBox
sed -i '/QuickSearchBox/d' build/make/target/product/core.mk
}
setup_vendor() {
echo "=================================="
echo "Running setup_vendor"
echo "=================================="
pushd "${BUILD_DIR}/vendor/android-prepare-vendor"
sed -i.bkp 's/ USE_DEBUGFS=true/ USE_DEBUGFS=false/; s/ # SYS_TOOLS/ SYS_TOOLS/; s/ # _UMOUNT=/ _UMOUNT=/' execute-all.sh
# get vendor files
yes | "${BUILD_DIR}/vendor/android-prepare-vendor/execute-all.sh" --fuse-ext2 --device "${DEVICE}" --buildID "${AOSP_BUILD}" --output "${BUILD_DIR}/vendor/android-prepare-vendor"
aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-vendor" --acl public-read <<< "${AOSP_BUILD}" || true
# copy vendor files to build tree
mkdir --parents "${BUILD_DIR}/vendor/google_devices" || true
rm --recursive --force "${BUILD_DIR}/vendor/google_devices/$DEVICE" || true
mv "${BUILD_DIR}/vendor/android-prepare-vendor/${DEVICE}/$(tr '[:upper:]' '[:lower:]' <<< "${AOSP_BUILD}")/vendor/google_devices/${DEVICE}" "${BUILD_DIR}/vendor/google_devices"
# smaller devices need big brother vendor files
if [ "$DEVICE" == 'sailfish' ]; then
rm --recursive --force "${BUILD_DIR}/vendor/google_devices/marlin" || true
mv "${BUILD_DIR}/vendor/android-prepare-vendor/sailfish/$(tr '[:upper:]' '[:lower:]' <<< "${AOSP_BUILD}")/vendor/google_devices/marlin" "${BUILD_DIR}/vendor/google_devices"
fi
if [ "$DEVICE" == 'walleye' ]; then
rm --recursive --force "${BUILD_DIR}/vendor/google_devices/muskie" || true
mv "${BUILD_DIR}/vendor/android-prepare-vendor/walleye/$(tr '[:upper:]' '[:lower:]' <<< "${AOSP_BUILD}")/vendor/google_devices/muskie" "${BUILD_DIR}/vendor/google_devices"
fi
popd
}
aws_import_keys() {
echo "=================================="
echo "Running aws_import_keys"
echo "=================================="
if [ "$(aws s3 ls "s3://${AWS_KEYS_BUCKET}/${DEVICE}" | wc -l)" == '0' ]; then
aws_gen_keys
else
echo "Keys already exist for ${DEVICE} - grabbing them from S3"
mkdir -p "${BUILD_DIR}/keys"
aws s3 sync "s3://${AWS_KEYS_BUCKET}" "${BUILD_DIR}/keys"
fi
}
apply_patches() {
echo "=================================="
echo "Running apply_patches"
echo "=================================="
patch_custom
patch_apps
patch_base_config
patch_device_config
patch_chromium_webview
patch_updater
patch_fdroid
patch_priv_ext
patch_launcher
}
patch_custom() {
# allow custom patches to be applied to AOSP build tree
patches_dir="$HOME/patches"
if [ -z "$REPO_PATCHES" ]; then
echo "No custom patches requested"
else
echo "Cloning custom patches $REPO_PATCHES to ${patches_dir}"
git clone $REPO_PATCHES ${patches_dir}
pushd $BUILD_DIR
while read patch; do
echo "Applying patch $patch"
patch -p1 < ${patches_dir}/$patch
done < ${patches_dir}/manifest
fi
# allow prebuilt applications to be added to build tree
prebuilt_dir="$BUILD_DIR/packages/apps/Custom"
if [ -z "$REPO_PREBUILTS" ]; then
echo "No custom apks requested"
else
echo "Putting custom prebuilts from $REPO_PREBUILTS in build tree location ${prebuilt_dir}"
git clone $REPO_PREBUILTS ${prebuilt_dir}
for file in ${prebuilt_dir}/*/ ; do
package_name=$(awk -F"=" '/LOCAL_MODULE /{print $2}' $file/Android.mk)
sed -i "\$aPRODUCT_PACKAGES += ${package_name}" ${BUILD_DIR}/build/make/target/product/core.mk
done
fi
# allow custom hosts file
hosts_file_location="$BUILD_DIR/system/core/rootdir/etc/hosts"
if [ -z "$HOSTS_FILE" ]; then
echo "No custom hosts file requested"
else
echo "Replacing hosts file with $HOSTS_FILE"
wget -O $hosts_file_location "$HOSTS_FILE"
fi
}
patch_base_config() {
# enable swipe up gesture functionality as option
sed -i 's@<bool name="config_swipe_up_gesture_setting_available">false</bool>@<bool name="config_swipe_up_gesture_setting_available">true</bool>@' ${BUILD_DIR}/frameworks/base/core/res/res/values/config.xml
}
patch_device_config() {
# set proper model names
sed -i 's@PRODUCT_MODEL := AOSP on msm8996@PRODUCT_MODEL := Pixel XL@' ${BUILD_DIR}/device/google/marlin/aosp_marlin.mk
sed -i 's@PRODUCT_MANUFACTURER := google@PRODUCT_MANUFACTURER := Google@' ${BUILD_DIR}/device/google/marlin/aosp_marlin.mk
sed -i 's@PRODUCT_MODEL := AOSP on msm8996@PRODUCT_MODEL := Pixel@' ${BUILD_DIR}/device/google/marlin/aosp_sailfish.mk
sed -i 's@PRODUCT_MANUFACTURER := google@PRODUCT_MANUFACTURER := Google@' ${BUILD_DIR}/device/google/marlin/aosp_sailfish.mk
sed -i 's@PRODUCT_MODEL := AOSP on taimen@PRODUCT_MODEL := Pixel 2 XL@' ${BUILD_DIR}/device/google/taimen/aosp_taimen.mk
sed -i 's@PRODUCT_MODEL := AOSP on walleye@PRODUCT_MODEL := Pixel 2@' ${BUILD_DIR}/device/google/muskie/aosp_walleye.mk
}
patch_chromium_webview() {
cat <<EOF > ${BUILD_DIR}/frameworks/base/core/res/res/xml/config_webview_packages.xml
<?xml version="1.0" encoding="utf-8"?>
<webviewproviders>
<webviewprovider description="Chromium" packageName="org.chromium.chrome" availableByDefault="true">
</webviewprovider>
</webviewproviders>
EOF
}
patch_fdroid() {
echo "sdk.dir=${HOME}/sdk" > ${BUILD_DIR}/packages/apps/F-Droid/local.properties
echo "sdk.dir=${HOME}/sdk" > ${BUILD_DIR}/packages/apps/F-Droid/app/local.properties
sed -i 's/gradle assembleRelease/..\/gradlew assembleRelease/' ${BUILD_DIR}/packages/apps/F-Droid/Android.mk
sed -i 's@fdroid_apk := build/outputs/apk/$(fdroid_dir)-release-unsigned.apk@fdroid_apk := build/outputs/apk/full/release/app-full-release-unsigned.apk@' ${BUILD_DIR}/packages/apps/F-Droid/Android.mk
}
patch_apps() {
sed -i.original "\$aPRODUCT_PACKAGES += Updater" ${BUILD_DIR}/build/make/target/product/core.mk
sed -i.original "\$aPRODUCT_PACKAGES += F-DroidPrivilegedExtension" ${BUILD_DIR}/build/make/target/product/core.mk
sed -i.original "\$aPRODUCT_PACKAGES += F-Droid" ${BUILD_DIR}/build/make/target/product/core.mk
sed -i.original "\$aPRODUCT_PACKAGES += chromium" ${BUILD_DIR}/build/make/target/product/core.mk
}
patch_updater() {
pushd "$BUILD_DIR"/packages/apps/Updater/res/values
sed --in-place --expression "s@s3bucket@${RELEASE_URL}/@g" config.xml
}
fdpe_hash() {
keytool -list -printcert -file "$1" | grep 'SHA256:' | tr --delete ':' | cut --delimiter ' ' --fields 3
}
patch_priv_ext() {
unofficial_sailfish_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/sailfish/releasekey.x509.pem")
unofficial_sailfish_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/sailfish/platform.x509.pem")
unofficial_marlin_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/marlin/releasekey.x509.pem")
unofficial_marlin_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/marlin/platform.x509.pem")
unofficial_taimen_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/taimen/releasekey.x509.pem")
unofficial_taimen_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/taimen/platform.x509.pem")
unofficial_walleye_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/walleye/releasekey.x509.pem")
unofficial_walleye_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/walleye/platform.x509.pem")
if [ "$DEVICE" == 'marlin' ]; then
sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_marlin_releasekey_hash}'"),\n new Pair<>("org.fdroid.fdroid", "'${unofficial_marlin_platform_hash}'")/' \
"${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
fi
if [ "$DEVICE" == 'sailfish' ]; then
sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_sailfish_releasekey_hash}'"),\n new Pair<>("org.fdroid.fdroid", "'${unofficial_sailfish_platform_hash}'")/' \
"${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
fi
if [ "$DEVICE" == 'taimen' ]; then
sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_taimen_releasekey_hash}'"),\n new Pair<>("org.fdroid.fdroid", "'${unofficial_taimen_platform_hash}'")/' \
"${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
fi
if [ "$DEVICE" == 'walleye' ]; then
sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_walleye_releasekey_hash}'"),\n new Pair<>("org.fdroid.fdroid", "'${unofficial_walleye_platform_hash}'")/' \
"${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
fi
}
patch_launcher() {
# disable QuickSearchBox widget on home screen
sed -i.original "s/QSB_ON_FIRST_SCREEN = true;/QSB_ON_FIRST_SCREEN = false;/" "${BUILD_DIR}/packages/apps/Launcher3/src/com/android/launcher3/config/BaseFlags.java"
# fix compile error with uninitialized variable
sed -i.original "s/boolean createEmptyRowOnFirstScreen;/boolean createEmptyRowOnFirstScreen = false;/" "${BUILD_DIR}/packages/apps/Launcher3/src/com/android/launcher3/provider/ImportDataTask.java"
}
rebuild_marlin_kernel() {
echo "=================================="
echo "Running rebuild_marlin_kernel"
echo "=================================="
# checkout kernel source on proper commit
mkdir -p "${MARLIN_KERNEL_SOURCE_DIR}"
if test -d "${MARLIN_KERNEL_SOURCE_DIR}"/.git ; then
pushd "${MARLIN_KERNEL_SOURCE_DIR}"
sed -i 's|url = .*|url = '"${KERNEL_SOURCE_URL}"'|' .git/config
git fetch
popd
else
git clone "${KERNEL_SOURCE_URL}" "${MARLIN_KERNEL_SOURCE_DIR}"
fi
# TODO: make this a bit more robust
kernel_commit_id=$(lz4cat "${BUILD_DIR}/device/google/marlin-kernel/Image.lz4-dtb" | grep -a 'Linux version' | cut -d ' ' -f3 | cut -d'-' -f2 | sed 's/^g//g')
cd "${MARLIN_KERNEL_SOURCE_DIR}"
echo "Checking out kernel commit ${kernel_commit_id}"
git checkout ${kernel_commit_id}
# run in another shell to avoid it mucking with environment variables for normal AOSP build
bash -c "\
cd ${BUILD_DIR};
. build/envsetup.sh;
make -j$(nproc --all) dtc mkdtimg;
export PATH=${BUILD_DIR}/out/host/linux-x86/bin:${PATH};
ln --verbose --symbolic ${BUILD_DIR}/keys/${DEVICE}/verity_user.der.x509 ${MARLIN_KERNEL_SOURCE_DIR}/verity_user.der.x509;
cd ${MARLIN_KERNEL_SOURCE_DIR};
make -j$(nproc --all) ARCH=arm64 marlin_defconfig;
make -j$(nproc --all) ARCH=arm64 CONFIG_COMPAT_VDSO=n CROSS_COMPILE=${BUILD_DIR}/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-;
cp -f arch/arm64/boot/Image.lz4-dtb ${BUILD_DIR}/device/google/marlin-kernel/;
rm -rf ${BUILD_DIR}/out/build_*;
"
}
build_aosp() {
echo "=================================="
echo "Running build_aosp"
echo "=================================="
pushd "$BUILD_DIR"
source "${BUILD_DIR}/script/setup.sh"
choosecombo $BUILD_TARGET
make -j $(nproc) target-files-package
make -j $(nproc) brillo_update_payload
bash -x "${BUILD_DIR}/script/release.sh" "$DEVICE"
}
aws_release() {
echo "=================================="
echo "Running aws_release"
echo "=================================="
pushd "${BUILD_DIR}/out"
build_date="$(< build_number.txt)"
build_timestamp="$(unzip -p "release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" META-INF/com/android/metadata | grep 'post-timestamp' | cut --delimiter "=" --fields 2)"
# copy ota file to s3, update file metadata used by updater app, and remove old ota files
read -r old_metadata <<< "$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}" -)"
old_date="$(cut -d ' ' -f 1 <<< "${old_metadata}")"
(
aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" "s3://${AWS_RELEASE_BUCKET}" --acl public-read &&
echo "${build_date} ${build_timestamp} ${AOSP_BUILD}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}" --acl public-read &&
echo "${BUILD_TIMESTAMP}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}-true-timestamp" --acl public-read
) && ( aws s3 rm "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-ota_update-${old_date}.zip" || true )
if [ "$(aws s3 ls "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-factory-latest.tar.xz" | wc -l)" == '0' ]; then
aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-factory-${build_date}.tar.xz" "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-factory-latest.tar.xz"
fi
# cleanup old target files if some exist
if [ "$(aws s3 ls "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target" | wc -l)" != '0' ]; then
cleanup_target_files
fi
# copy new target file to s3
aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-target_files-${build_date}.zip" "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target/${DEVICE}-target-files-${build_date}.zip"
}
checkpoint_versions() {
# checkpoint stack version
echo "${STACK_VERSION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/rattlesnakeos-stack/revision"
# checkpoint f-droid
echo "${FDROID_PRIV_EXT_VERSION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/fdroid-priv/revision"
echo "${FDROID_CLIENT_VERSION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/fdroid/revision"
}
cleanup_target_files() {
echo "=================================="
echo "Running cleanup_target_files"
echo "=================================="
aws s3 sync "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target" "${BUILD_DIR}/${DEVICE}-target"
pushd "${BUILD_DIR}/out"
current_date="$(< build_number.txt)"
pushd "${BUILD_DIR}/${DEVICE}-target"
for target_file in ${DEVICE}-target-files-*.zip ; do
old_date=$(echo "$target_file" | cut --delimiter "-" --fields 4 | cut --delimiter "." --fields 5 --complement)
aws s3 rm "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target/${DEVICE}-target-files-${old_date}.zip" || true
done
}
aws_notify_simple() {
aws sns publish --region ${REGION} --topic-arn "$AWS_SNS_ARN" --message "$1"
}
aws_notify() {
LOGOUTPUT=
if [ ! -z "$2" ]; then
LOGOUTPUT=$(tail -c 20000 /var/log/cloud-init-output.log)
fi
ELAPSED="$(($SECONDS / 3600))hrs $((($SECONDS / 60) % 60))min $(($SECONDS % 60))sec"
aws sns publish --region ${REGION} --topic-arn "$AWS_SNS_ARN" \
--message="$(printf "$1\n Device: %s\n Release Channel: %s\n Build Date: %s\n Elapsed Time: %s\n AOSP Build: %s\n AOSP Branch: %s\n Chromium Version: %s\n F-Droid Version: %s\n F-Droid Priv Extension Version: %s\n%s" \
"${DEVICE}" "${RELEASE_CHANNEL}" "${BUILD_DATE}" "${ELAPSED}" "${AOSP_BUILD}" "${AOSP_BRANCH}" "${LATEST_CHROMIUM}" "${FDROID_CLIENT_VERSION}" "${FDROID_PRIV_EXT_VERSION}" "${LOGOUTPUT}")" || true
}
aws_logging() {
df -h
du -chs "${BUILD_DIR}" || true
uptime
aws s3 cp /var/log/cloud-init-output.log "s3://${AWS_LOGS_BUCKET}/${DEVICE}/$(date +%s)"
}
aws_gen_keys() {
gen_keys
aws s3 sync "${BUILD_DIR}/keys" "s3://${AWS_KEYS_BUCKET}"
}
gen_keys() {
echo "=================================="
echo "Running gen_keys"
echo "=================================="
mkdir --parents "${BUILD_DIR}/keys/${DEVICE}"
pushd "${BUILD_DIR}/keys/${DEVICE}"
for key in {releasekey,platform,shared,media,verity} ; do
# make_key exits with unsuccessful code 1 instead of 0, need ! to negate
! "${BUILD_DIR}/development/tools/make_key" "$key" "$CERTIFICATE_SUBJECT"
done
if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
gen_verity_key "${DEVICE}"
fi
if [ "${DEVICE}" == "walleye" ] || [ "${DEVICE}" == "taimen" ]; then
gen_avb_key "${DEVICE}"
fi
}
gen_avb_key() {
echo "=================================="
echo "Running gen_avb_key"
echo "=================================="
pushd "$BUILD_DIR"
openssl genrsa -out "${BUILD_DIR}/keys/$1/avb.pem" 2048
${BUILD_DIR}/external/avb/avbtool extract_public_key --key "${BUILD_DIR}/keys/$1/avb.pem" --output "${BUILD_DIR}/keys/$1/avb_pkmd.bin"
}
gen_verity_key() {
echo "=================================="
echo "Running gen_verity_key"
echo "=================================="
pushd "$BUILD_DIR"
make -j 20 generate_verity_key
"${BUILD_DIR}/out/host/linux-x86/bin/generate_verity_key" -convert "${BUILD_DIR}/keys/$1/verity.x509.pem" "${BUILD_DIR}/keys/$1/verity_key"
make clobber
openssl x509 -outform der -in "${BUILD_DIR}/keys/$1/verity.x509.pem" -out "${BUILD_DIR}/keys/$1/verity_user.der.x509"
}
cleanup() {
rv=$?
aws_logging
if [ $rv -ne 0 ]; then
aws_notify "RattlesnakeOS Build FAILED" 1
fi
if [ "${PREVENT_SHUTDOWN}" = true ]; then
echo "Skipping shutdown"
else
sudo shutdown -h now
fi
}
trap cleanup 0
set -e
aws() {
func="$1"
cmd="$2"
in="$3"
out="$4"
if [ "$func" == "sns" ]
then
if [[ $7 == --message=* ]]
then
echo "${7#--message=}" | sed 's/^/aws_notify: /' >&2
else
echo "$8" | sed 's/^/aws_notify: /' >&2
fi
elif [ "$func" == "s3" ]
then
if [ "$cmd" == "cp" ]
then
if [[ $in == s3://* ]]
then
in="${in#s3://}"
in="$HOME/s3/$in"
fi
if [[ $out == s3://* ]]
then
out="${out#s3://}"
out="$HOME/s3/$out"
fi
if [ "$in" == "-" ]
then
mkdir -p $( dirname "$out" )
cat - > "$out"
elif [ "$out" == "-" ]
then
cat "$in"
else
mkdir -p $( dirname "$out" )
cp -f "$in" "$out"
fi
elif [ "$cmd" == "ls" ]
then
if [[ $in == s3://* ]]
then
in="${in#s3://}"
in="$HOME/s3/$in"
fi
ls -1 "$in"
elif [ "$cmd" == "rm" ]
then
rm -f -- "$in"
elif [ "$cmd" == "sync" ]
then
if [[ $in == s3://* ]]
then
in="${in#s3://}"
in="$HOME/s3/$in"
fi
if [[ $out == s3://* ]]
then
out="${out#s3://}"
out="$HOME/s3/$out"
fi
rsync -a --delete -- "$in/" "$out/"
fi
fi
}
gen_keys() {
echo "This program needs the keys already present in s3://${AWS_KEYS_BUCKET}/${DEVICE}" >&2
false
}
aws_logging()
{
return
}
cleanup() {
rv=$?
if [ $rv -ne 0 ]
then
aws_notify "RattlesnakeOS Build FAILED"
fi
exit $rv
}
persist_latest_versions() {
rm -rf env*.save
mkdir -p s3/interstage
cat > s3/interstage/env.$BUILD_NUMBER.save <<EOF
STACK_UPDATE_MESSAGE="$STACK_UPDATE_MESSAGE"
LATEST_STACK_VERSION="$LATEST_STACK_VERSION"
LATEST_CHROMIUM="$LATEST_CHROMIUM"
FDROID_CLIENT_VERSION="$FDROID_CLIENT_VERSION"
FDROID_PRIV_EXT_VERSION="$FDROID_PRIV_EXT_VERSION"
AOSP_BUILD="$AOSP_BUILD"
AOSP_BRANCH="$AOSP_BRANCH"
EOF
}
reload_latest_versions() {
source s3/interstage/env.$BUILD_NUMBER.save
}
if [ "$ONLY_REPORT" == "true" ]
then
full_run() {
get_latest_versions
persist_latest_versions
check_for_new_versions
}
else
full_run() {
if [ "$STAGE" != "" ] ; then
reload_latest_versions
if [ "$STAGE" == "rebuild_marlin_kernel" ] ; then
if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
"$STAGE"
fi
else
"$STAGE"
fi
else
get_latest_versions
check_for_new_versions
aws_notify "RattlesnakeOS Build STARTED"
setup_env
check_chromium
fetch_aosp_source
setup_vendor
aws_import_keys
apply_patches
# only marlin and sailfish need kernel rebuilt so that verity_key is included
if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
rebuild_marlin_kernel
fi
build_aosp
aws_release
checkpoint_versions
aws_notify "RattlesnakeOS Build SUCCESS"
fi
}
fi
RELEASE_URL=http://example.com/
full_run
Diff from your script to mine:
--- templates/build_template.go 2018-10-08 21:43:08.250800408 +0000
+++ stack-builder 2018-10-09 19:42:24.964696477 +0000
@@ -1,6 +1,4 @@
-package templates
-const BuildTemplate = `
#!/bin/bash
if [ $# -ne 1 ]; then
@@ -18,48 +16,48 @@
fi
# set region
-REGION=<% .Region %>
+REGION=none
export AWS_DEFAULT_REGION=${REGION}
# stack name
-STACK_NAME=<% .Name %>
+STACK_NAME=rattlesnakeos
# version of stack running
-STACK_VERSION=<% .Version %>
+STACK_VERSION=
# prevent default action of shutting down on exit
-PREVENT_SHUTDOWN=<% .PreventShutdown %>
+PREVENT_SHUTDOWN=
# force build even if no new versions exist of components
-FORCE_BUILD=<% .Force %>
+FORCE_BUILD=false
# skip chromium build if there is an existing build
-SKIP_CHROMIUM_BUILD=<% .SkipChromiumBuild %>
+SKIP_CHROMIUM_BUILD=false
# pin to specific version of android
ANDROID_VERSION="9.0"
# build type (user or userdebug)
-BUILD_TYPE="user"
+BUILD_TYPE="user" # replaced
# build channel (stable or beta)
BUILD_CHANNEL="stable"
# user customizable things
-REPO_PATCHES=<% .RepoPatches %>
-REPO_PREBUILTS=<% .RepoPrebuilts %>
-HOSTS_FILE=<% .HostsFile %>
+REPO_PATCHES=
+REPO_PREBUILTS=
+HOSTS_FILE=
# get current instance details for reference
-INSTANCE_TYPE=$(curl -s http://169.254.169.254/latest/meta-data/instance-type)
-INSTANCE_REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | awk -F\" '/region/ {print $4}')
-INSTANCE_IP=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)
+INSTANCE_TYPE=none
+INSTANCE_REGION=none
+INSTANCE_IP=none
# aws settings
AWS_KEYS_BUCKET="${STACK_NAME}-keys"
AWS_RELEASE_BUCKET="${STACK_NAME}-release"
AWS_LOGS_BUCKET="${STACK_NAME}-logs"
-AWS_SNS_ARN=$(aws --region ${REGION} sns list-topics --query 'Topics[0].TopicArn' --output text | cut -d":" -f1,2,3,4,5)":${STACK_NAME}"
+AWS_SNS_ARN=none
# build settings
BUILD_TARGET="release aosp_${DEVICE} ${BUILD_TYPE}"
@@ -193,10 +191,10 @@
fi
if [ "$needs_update" = true ]; then
- echo "New build is required"
+ aws_notify "New build is required"
else
if [ "$FORCE_BUILD" = true ]; then
- echo "No build is required, but FORCE_BUILD=true"
+ aws_notify "No build is required, but FORCE_BUILD=true"
else
aws_notify "RattlesnakeOS build not required as all components are already up to date."
exit 0
@@ -235,14 +233,21 @@
# install packages
sudo apt-get update
sudo apt-get --assume-yes install openjdk-8-jdk git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip python-networkx liblz4-tool
- sudo apt-get --assume-yes build-dep "linux-image-$(uname --kernel-release)"
+ sudo apt-get --assume-yes build-dep "$(apt-cache search linux-image-* | awk ' { print $1 } ' | sort | egrep -v -- '(-dbg|-rt|-pae)' | grep ^linux-image-[0-9][.] | tail -1)"
sudo apt-get --assume-yes install repo gperf jq fuseext2
# setup android sdk (required for fdroid build)
mkdir -p ${HOME}/sdk
pushd ${HOME}/sdk
- wget ${ANDROID_SDK_URL} -O sdk-tools.zip
- unzip sdk-tools.zip
+ if [ ! -f sdk-tools.zip ] ; then
+ wget ${ANDROID_SDK_URL} -O sdk-tools.zip
+fi
+ unzip -o sdk-tools.zip || {
+ echo unzip failed, retrying download
+ rm -f sdk-tools.zip
+ wget ${ANDROID_SDK_URL} -O sdk-tools.zip
+ unzip -o sdk-tools.zip
+ }
yes | ./tools/bin/sdkmanager --licenses
./tools/android update sdk -u --use-sdk-wrapper
@@ -280,7 +285,7 @@
echo "Building chromium $LATEST_CHROMIUM"
build_chromium $LATEST_CHROMIUM
fi
- rm -rf $HOME/chromium
+
}
build_chromium() {
@@ -297,7 +302,7 @@
# fetch chromium
mkdir -p $HOME/chromium
cd $HOME/chromium
- fetch --nohooks android
+ test -f .gclient || fetch --nohooks android
cd src
# checkout specific revision
@@ -308,6 +313,12 @@
sudo ./build/install-build-deps-android.sh
# run gclient sync (runhooks will run as part of this)
+ for gitdir in $( find -name .git ) ; do
+ pushd $gitdir/.. || continue
+ git clean -dff || { popd ; return $? ; }
+ git reset --hard || { popd ; return $? ; }
+ popd || return $?
+ done
yes | gclient sync --with_branch_heads --jobs 32 -RDf
# cleanup any files in tree not part of this revision
@@ -316,8 +327,8 @@
# reset any modifications
git checkout -- .
- mkdir -p out/Default
- cat <<EOF > out/Default/args.gn
+ mkdir -p "$HOME"/chromium-out
+ cat <<EOF > "$HOME"/chromium-out/args.gn
target_os = "android"
target_cpu = "arm64"
is_debug = false
@@ -334,12 +345,12 @@
android_default_version_code = "$DEFAULT_VERSION"
EOF
- gn gen out/Default
- autoninja -C out/Default/ monochrome_public_apk
+ gn gen "$HOME"/chromium-out
+ autoninja -C "$HOME"/chromium-out/ monochrome_public_apk
- mkdir -p ${BUILD_DIR}/external/chromium/prebuilt/arm64
- cp out/Default/apks/MonochromePublic.apk ${BUILD_DIR}/external/chromium/prebuilt/arm64/
- aws s3 cp "${BUILD_DIR}/external/chromium/prebuilt/arm64/MonochromePublic.apk" "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk"
+
+ aws s3 cp "$HOME"/chromium-out/apks/MonochromePublic.apk "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk"
+
echo "${CHROMIUM_REVISION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/chromium/revision"
}
@@ -349,8 +360,20 @@
echo "=================================="
pushd "${BUILD_DIR}"
repo init --manifest-url "$MANIFEST_URL" --manifest-branch "$AOSP_BRANCH" --depth 1 || true
+ for gitdir in $(find -name .git) ; do
+ pushd "$gitdir/.." || continue
+ git status || { popd ; return $? ; }
+ git clean -dff || { popd ; return $? ; }
+ git reset --hard || { popd ; return $? ; }
+ popd || return $?
+ done
+
# make modifications to default AOSP
+ # Since we just git cleaned everything, we will have to re-copy
+ # the MonochromePublic.apk file once again.
+ mkdir -p ${BUILD_DIR}/external/chromium/prebuilt/arm64
+ aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
if ! grep -q "RattlesnakeOS" .repo/manifest.xml; then
awk -i inplace \
-v ANDROID_VERSION="$ANDROID_VERSION" \
@@ -581,7 +604,14 @@
echo "=================================="
# checkout kernel source on proper commit
mkdir -p "${MARLIN_KERNEL_SOURCE_DIR}"
- git clone "${KERNEL_SOURCE_URL}" "${MARLIN_KERNEL_SOURCE_DIR}"
+ if test -d "${MARLIN_KERNEL_SOURCE_DIR}"/.git ; then
+ pushd "${MARLIN_KERNEL_SOURCE_DIR}"
+ sed -i 's|url = .*|url = '"${KERNEL_SOURCE_URL}"'|' .git/config
+ git fetch
+ popd
+ else
+ git clone "${KERNEL_SOURCE_URL}" "${MARLIN_KERNEL_SOURCE_DIR}"
+ fi
# TODO: make this a bit more robust
kernel_commit_id=$(lz4cat "${BUILD_DIR}/device/google/marlin-kernel/Image.lz4-dtb" | grep -a 'Linux version' | cut -d ' ' -f3 | cut -d'-' -f2 | sed 's/^g//g')
cd "${MARLIN_KERNEL_SOURCE_DIR}"
@@ -614,7 +644,7 @@
make -j $(nproc) target-files-package
make -j $(nproc) brillo_update_payload
- "${BUILD_DIR}/script/release.sh" "$DEVICE"
+ bash -x "${BUILD_DIR}/script/release.sh" "$DEVICE"
}
aws_release() {
@@ -626,7 +656,7 @@
build_timestamp="$(unzip -p "release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" META-INF/com/android/metadata | grep 'post-timestamp' | cut --delimiter "=" --fields 2)"
# copy ota file to s3, update file metadata used by updater app, and remove old ota files
- read -r old_metadata <<< "$(wget -O - "${RELEASE_URL}/${RELEASE_CHANNEL}")"
+ read -r old_metadata <<< "$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}" -)"
old_date="$(cut -d ' ' -f 1 <<< "${old_metadata}")"
(
aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" "s3://${AWS_RELEASE_BUCKET}" --acl public-read &&
@@ -681,8 +711,8 @@
fi
ELAPSED="$(($SECONDS / 3600))hrs $((($SECONDS / 60) % 60))min $(($SECONDS % 60))sec"
aws sns publish --region ${REGION} --topic-arn "$AWS_SNS_ARN" \
- --message="$(printf "$1\n Device: %s\n Stack Name: %s\n Stack Version: %s %s\n Stack Region: %s\n Release Channel: %s\n Instance Type: %s\n Instance Region: %s\n Instance IP: %s\n Build Date: %s\n Elapsed Time: %s\n AOSP Build: %s\n AOSP Branch: %s\n Chromium Version: %s\n F-Droid Version: %s\n F-Droid Priv Extension Version: %s\n%s" \
- "${DEVICE}" "${STACK_NAME}" "${STACK_VERSION}" "${STACK_UPDATE_MESSAGE}" "${REGION}" "${RELEASE_CHANNEL}" "${INSTANCE_TYPE}" "${INSTANCE_REGION}" "${INSTANCE_IP}" "${BUILD_DATE}" "${ELAPSED}" "${AOSP_BUILD}" "${AOSP_BRANCH}" "${LATEST_CHROMIUM}" "${FDROID_CLIENT_VERSION}" "${FDROID_PRIV_EXT_VERSION}" "${LOGOUTPUT}")" || true
+ --message="$(printf "$1\n Device: %s\n Release Channel: %s\n Build Date: %s\n Elapsed Time: %s\n AOSP Build: %s\n AOSP Branch: %s\n Chromium Version: %s\n F-Droid Version: %s\n F-Droid Priv Extension Version: %s\n%s" \
+ "${DEVICE}" "${RELEASE_CHANNEL}" "${BUILD_DATE}" "${ELAPSED}" "${AOSP_BUILD}" "${AOSP_BRANCH}" "${LATEST_CHROMIUM}" "${FDROID_CLIENT_VERSION}" "${FDROID_PRIV_EXT_VERSION}" "${LOGOUTPUT}")" || true
}
aws_logging() {
@@ -755,5 +785,142 @@
set -e
+aws() {
+ func="$1"
+ cmd="$2"
+ in="$3"
+ out="$4"
+ if [ "$func" == "sns" ]
+ then
+ if [[ $7 == --message=* ]]
+ then
+ echo "${7#--message=}" | sed 's/^/aws_notify: /' >&2
+ else
+ echo "$8" | sed 's/^/aws_notify: /' >&2
+ fi
+ elif [ "$func" == "s3" ]
+ then
+ if [ "$cmd" == "cp" ]
+ then
+ if [[ $in == s3://* ]]
+ then
+ in="${in#s3://}"
+ in="$HOME/s3/$in"
+ fi
+ if [[ $out == s3://* ]]
+ then
+ out="${out#s3://}"
+ out="$HOME/s3/$out"
+ fi
+ if [ "$in" == "-" ]
+ then
+ mkdir -p $( dirname "$out" )
+ cat - > "$out"
+ elif [ "$out" == "-" ]
+ then
+ cat "$in"
+ else
+ mkdir -p $( dirname "$out" )
+ cp -f "$in" "$out"
+ fi
+ elif [ "$cmd" == "ls" ]
+ then
+ if [[ $in == s3://* ]]
+ then
+ in="${in#s3://}"
+ in="$HOME/s3/$in"
+ fi
+ ls -1 "$in"
+ elif [ "$cmd" == "rm" ]
+ then
+ rm -f -- "$in"
+ elif [ "$cmd" == "sync" ]
+ then
+ if [[ $in == s3://* ]]
+ then
+ in="${in#s3://}"
+ in="$HOME/s3/$in"
+ fi
+ if [[ $out == s3://* ]]
+ then
+ out="${out#s3://}"
+ out="$HOME/s3/$out"
+ fi
+ rsync -a --delete -- "$in/" "$out/"
+ fi
+ fi
+}
+gen_keys() {
+ echo "This program needs the keys already present in s3://${AWS_KEYS_BUCKET}/${DEVICE}" >&2
+ false
+}
+aws_logging()
+{
+ return
+}
+cleanup() {
+ rv=$?
+ if [ $rv -ne 0 ]
+ then
+ aws_notify "RattlesnakeOS Build FAILED"
+ fi
+ exit $rv
+}
+persist_latest_versions() {
+ rm -rf env*.save
+ mkdir -p s3/interstage
+ cat > s3/interstage/env.$BUILD_NUMBER.save <<EOF
+STACK_UPDATE_MESSAGE="$STACK_UPDATE_MESSAGE"
+LATEST_STACK_VERSION="$LATEST_STACK_VERSION"
+LATEST_CHROMIUM="$LATEST_CHROMIUM"
+FDROID_CLIENT_VERSION="$FDROID_CLIENT_VERSION"
+FDROID_PRIV_EXT_VERSION="$FDROID_PRIV_EXT_VERSION"
+AOSP_BUILD="$AOSP_BUILD"
+AOSP_BRANCH="$AOSP_BRANCH"
+EOF
+}
+reload_latest_versions() {
+ source s3/interstage/env.$BUILD_NUMBER.save
+}
+if [ "$ONLY_REPORT" == "true" ]
+then
+full_run() {
+ get_latest_versions
+ persist_latest_versions
+ check_for_new_versions
+}
+else
+full_run() {
+ if [ "$STAGE" != "" ] ; then
+ reload_latest_versions
+ if [ "$STAGE" == "rebuild_marlin_kernel" ] ; then
+ if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
+ "$STAGE"
+ fi
+ else
+ "$STAGE"
+ fi
+ else
+ get_latest_versions
+ check_for_new_versions
+ aws_notify "RattlesnakeOS Build STARTED"
+ setup_env
+ check_chromium
+ fetch_aosp_source
+ setup_vendor
+ aws_import_keys
+ apply_patches
+ # only marlin and sailfish need kernel rebuilt so that verity_key is included
+ if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
+ rebuild_marlin_kernel
+ fi
+ build_aosp
+ aws_release
+ checkpoint_versions
+ aws_notify "RattlesnakeOS Build SUCCESS"
+ fi
+}
+fi
+
+RELEASE_URL=http://example.com/
full_run
-`
I stress this was built with a 100% clean workspace directory:
sudo rm -rf rattlesnake-os rattlesnakeos-stack rattlesnakeos-stack@tmp s3 sdk tmp .android .cache .gitconfig .gsutil .repo_.gitconfig.json .repoconfig .vpython-root .vpython_cipd_cache .wget-hsts
Version: > git checkout -f d45d3a49b0989497841390d593518b3611e6710e
I'm sure it's very similar to the original, but there is likely some difference causing issues here as I've done a successful flash as of yesterday for marlin. I don't at the moment have time to dig in the differences between the two build scripts, but if I have some free cycles I'll take a peak.
Usually you get into a boot loop state without loading the boot animation screen when there is an issue with the kernel. Unfortunately that also means you can't get really any output through ADB to troubleshoot unless you have a debug cable. I'd double check you are properly building the kernel and also including the appropriate verity key with the kernel.
Hopefully you didn't lock the bootloader before doing any of this experimentation as this means you can easily get out of the situation. If you have the bootloader locked, you are basically in a situation where you now need to produce a properly booting image signed with the same key to flash on your device.
Yes, absolutely no boot animation screen.
Fortunately for me, my phone is both unlocked and OEM unlocked right now -- it's sitting next to my keyboard, idling in the recovery screen.
Perhaps you can furnish me with a build logfile of one of your builds, so that I can diff it with mine, and see if there's anything different?
In the meantime, here's the kernel stage build log. consoleText kernel build.log I see nothing that stands out in it.
Man, I see no meaningful differences in our kernel build. About the only things different are the order of compilation of the .o
files. Will continu to investigate.
I need to get a console-ramoops dump of the machine while it is running. How do I build a recovery image that will not actually bootloop back to recovery when something goes wrong?
I believe I have found the bug. My script is more aggressive than yours in cleaning junk from source directories. Since, in my edition, the step fetch_aosp_source
resets the device/google/marlin-kernel
repository to its original state, I have to suppress the copy of the kernel in rebuild_marlin_kernel
and move it to fetch_aosp_source
(where I also integrate the Chromium APK).
In pseudocode:
rebuild_marlin_kernel
builds kernelrebuild_marlin_kernel
hands over the kernel to build_aosp
phase by copying it to $BUILD_DIR/device/google/marlin-kernel
fetch_aosp
does a git clean -fxd
of everything that is a git repo -- this resets the Image.lz4-dtb
kernel image to the stock one shipping on the repofetch_aosp
patches stuff after cleaning everythingbuild_aosp
builds stuff -- this builds the incorrect kernel image into the final build productsrelease
packages everything up neatlyThe reason we did not catch this, is that (unlike the Chromium prebuilt APKs git repo) the kernel directory actually has a file Image.lz4-dtb
, so there is never a lack of kernel that would trigger a build failure.
There are too many side effects not tracked by version control in this pipeline. We should collaborate to reduce them. In all frankness, something like a Makefile to tie up the three main phases (as well as the patch phase) together sounds precisely like what we need here. I can understand patching the source in a shell script because there's no alternative, but putting build products in the source tree is usually a recipe for defective builds.
My edition of the program also does out-of-tree kernel builds now. This seriously improves the certainty that the kernel is being built pristinely and only with whatever patches your program may contain, never confused by .gitignore
.
(Will report soon.)
Yeah, I've verified my code actually builds in the new kernel, but to no avail. As a last ditch attempt, I will attempt another full clean build with the code updates.
Wait. After my latest rebuild, I found this in the ramoops log:
[ 1.878308] c2 1 apcc_l3_mem_acc_corner: disabling
[ 1.878312] c2 1 gfx_mem_acc_corner: disabling
[ 1.878317] c2 1 kryo0-retention: disabling
[ 1.878322] c2 1 kryo1-retention: disabling
[ 1.878376] c2 1 gdsc_mmagic_bimc: disabling
[ 1.878381] c2 1 rome_vreg: disabling
[ 1.878437] c2 1 usb_otg_vreg: disabling
[ 1.878445] c2 1 regulator_proxy_consumer_remove_all: removing regulator proxy consumer requests
[ 1.878772] c2 1 clock_late_init: Removing enables held for handed-off clocks
[ 1.879367] c1 300 [AUD] snd_soc_msm8996: uart_gpios_init done
[ 1.879379] c1 300 [AUD] snd_soc_msm8996: aud,uart-disable-mbhc is not set
[ 1.879390] c1 300 msm8996-asoc-snd soc:sound-9335: property qcom,tfa9888-reset-gpio not detected in node /soc/sound-9335
[ 1.879509] c1 300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-dout no 67
[ 1.879545] c1 300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-din no 68
[ 1.879575] c1 300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-clock no 65
[ 1.879601] c1 300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-sync no 66
[ 1.879629] c1 300 [AUD] snd_soc_msm8996: htc_msm8996_parse_gpio: gpio rcv-amp-gpio-en no 31
[ 1.881506] c1 300 msm8996-asoc-snd soc:sound-9335: ASoC: platform (null) not registered
[ 1.882916] c3 1 ALSA device list:
[ 1.882922] c3 1 No soundcards found.
[ 1.882974] c3 1 Warning: unable to open an initial console.
[ 1.882995] c3 1 dm: attempting early device configuration.
[ 1.883485] c3 1 device-mapper: init: adding target '0 1 android-verity 259:17 '
[ 1.883498] c3 1 device-mapper: android-verity: key:id:5d0936bec3848ab90c5dfaa8c80ab1f4f9dd5068 dev:259:17
[ 1.887540] c3 1 device-mapper: android-verity: bio magic_number:2952900609 protocol_version:0 table_length:266
[ 1.887551] c3 1 device-mapper: android-verity: verity_table: 1 /dev/block/platform/soc/624000.ufshc/by-name/system /dev/block/platform/soc/624000.ufshc/by-name/system 4096 4096 516099 516099 sha256 a1beaf398c2d163e3ce048fa3a69dc25948d9227371ba5b44df712611ac061ad aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7
[ 1.887570] c3 1 device-mapper: android-verity: keyring: key not found
[ 1.887575] c3 1 device-mapper: android-verity: Signature verification failed
[ 1.887585] c3 1 device-mapper: android-verity: triggering restart
[ 1.887603] c3 1 htc_reboot_block reboot_block: reason=77665508 msg=dm-verity device corrupted save to /misc+00003800
[ 1.887624] c3 1 htc_reboot_block reboot_block: unable to open file: /dev/block/platform/soc/624000.ufshc/by-name/misc
[ 1.887629] c3 1 [BATT] reboot_consistent_command_call: save batt cycle data
[ 1.887633] c3 1 [BATT] reboot_consistent_command_call: save consistent data
[ 1.887638] c3 1 [BATT] change_level_by_consistent_and_store_into_emmc: Battery data not ready, don't save consistent data.
[ 1.887643] c3 1 at 1887627895 (1970-02-01 07:58:06.530014065 UTC)
[ 1.887653] c2 58 [TP]: [ERR]:fwu_start_reflash: Firmware image synaptics_bl71.img not available
[ 1.887659] c2 58 [TP]:fwu_start_reflash: End of reflash process
[ 1.887664] c2 58 [TP]:synaptics_config_updater
[ 1.888246] c3 1 sd 0:0:0:4: [sde] Synchronizing SCSI cache
[ 1.888621] c2 58 [TP]:synaptics_config_updater: Device config ID = 4d31763200180000000000000000000000000000000000000000000000000000
[ 1.888626] c2 58 [TP]:synaptics_config_updater: Start of write config process
[ 1.888971] c3 1 sd 0:0:0:3: [sdd] Synchronizing SCSI cache
[ 1.889663] c3 1 sd 0:0:0:2: [sdc] Synchronizing SCSI cache
[ 1.890438] c3 1 sd 0:0:0:1: [sdb] Synchronizing SCSI cache
[ 1.891132] c3 1 sd 0:0:0:0: [sda] Synchronizing SCSI cache
[ 1.892031] c3 1 mdss_fb_release_all: try to close unopened fb 2! from pid:1 name:swapper/0
[ 1.892055] c3 1 mdss_fb_release_all: try to close unopened fb 1! from pid:1 name:swapper/0
[ 1.892073] c3 1 mdss_fb_release_all: try to close unopened fb 0! from pid:1 name:swapper/0
[ 1.892380] c3 1 [NFC] pn551_power_off_sequence ++
[ 1.892403] c3 1 [NFC] pn551_power_off_sequence : NFC_I2C_SCL set 0 0
[ 1.893417] c3 1 [NFC] pn551_power_off_sequence : NFC_I2C_SDA set 0 0
[ 1.941394] c2 58 [TP]: fwu_get_tw_vendor_v7: tw_vendor = 0x2
[ 1.941406] c2 58 [TP]:synaptics_config_updater: Image config ID = 4d31763200180000000000000000000000000000000000000000000000000000
[ 1.941411] c2 58 [TP]:synaptics_config_updater: config_size = 2048
[ 1.941417] c2 58 [TP]: synaptics_config_updater: crc_cksum = 66A6880E
[ 1.943071] c2 58 [TP]: crc_comparison_v7: config_crc = 66A6880E, flash_crc = 66A6880E
[ 1.943076] c2 58 [TP]: synaptics_config_updater: No need to update
[ 1.943088] c2 58 [TP]:synaptics_rmi4_reset_device from synaptics_config_updater+0x128/0xd38, rebuild = 0
[ 1.943424] c3 1 [NFC] pn551_power_off_sequence --
[ 1.972125] c2 58 [TP]:synaptics_rmi4_f12_init: Function 12 max x = 1439 max y = 2559 Rx: 17 Tx: 30
[ 1.972788] c2 58 [TP]synaptics_rmi4_f12_init:Wakeup Gesture range (0,0) -> (1439,2559)
[ 1.973050] c2 58 [TP]:synaptics_rmi4_f12_init report data init done
[ 1.981176] c2 58 [TP]:synaptics_rmi4_query_device: chip_id:3708, firmware_id:2433782
[ 1.982108] c2 58 [TP]:synaptics_rmi4_query_device: config_version: 4d31763200180000000000000000000000000000000000000000000000000000
[ 1.988152] c2 58 [TP]:synaptics_config_updater: End of write config process
[ 1.988157] c2 58 [TP]: synaptics_config_updater end
[ 2.236747] c1 376 SMBCHG: smbchg_parallel_usb_en_work: AICL at 1700
[ 2.236783] c1 376 SMBCHG: smbchg_is_parallel_usb_ok: Not in fast charge, skipping
[ 2.236790] c1 376 SMBCHG: smbchg_relax: relaxing: 0x00 (bit 1)
[ 2.740081] c1 376 SMBCHG: smbchg_parallel_usb_en_work: AICL at 1700
[ 2.740117] c1 376 SMBCHG: smbchg_is_parallel_usb_ok: Not in fast charge, skipping
[ 3.086905] c3 1 SMBCHG: usb_suspend: 5 voting for 0 - off
[ 3.087101] c0 4 of_batterydata_get_best_profile: Find batterydata path: qcom,m1-id1-batterydata, id_ohm=33320, raw_min=0, raw_max=16000.
[ 3.087144] c0 4 of_batterydata_get_best_profile: Find batterydata path: qcom,m1-id2-batterydata, id_ohm=33320, raw_min=27501, raw_max=40000.
[ 3.087158] c0 4 FG: fg_batt_profile_init: batt_full_charge_criteria_ma = 345
[ 3.087164] c0 4 FG: fg_batt_profile_init: batt_fcc_ma = 3400
[ 3.087170] c0 4 FG: fg_batt_profile_init: batt_capacity_mah = 3450
[ 3.087175] c0 4 FG: fg_batt_profile_init: fcc_half_capacity_ma = 1600
[ 3.087183] c0 4 FG: fg_batt_profile_init: failed to read profile integrity rc=-22
[ 3.087192] c0 4 [BATT] Probe process: (1, 3)
[ 3.087207] c0 4 [BATT] htc_battery_probe_process: catch name Unknown Battery, set batt id=255, full_ma=345, fcc_ma=3400, capacity=3450, half_capacity_ma:1600, pd_is_limited:0, batt_eoc_current_ma:100
[ 3.087212] c0 4 [BATT] Probe process done.
[ 3.087217] c0 4 FG: batt_profile_init: failed to initialize profile
[ 3.087230] c0 4 SMBCHG: battchg_suspend: 1 voting for 0 - on
[ 3.087267] c0 4 SMBCHG: smbchg_external_power_changed: usb type = 4, current_limit = 0, rc = 0
[ 3.087271] c0 4 SMBCHG: smbchg_stay_awake: staying awake: 0x02 (bit 2)
[ 3.087274] c0 4 SMBCHG: smbchg_vfloat_adjust_check: Starting vfloat adjustments
[ 3.087370] c0 4 SMBCHG: smbchg_relax: relaxing: 0x00 (bit 2)
[ 3.087598] c3 1 SMB1351 smb1351_parallel_set_property: SMB1351 OTG Boost is disabled
[ 3.087604] c3 1 SMBCHG: smbchg_external_otg_regulator_disable: Disabling external OTG Boost
[ 3.087608] c3 1 SMB1351 smb1351_parallel_set_chg_present: set slave present = 1 -> 0
[ 3.087613] c3 1 SMB1351 smb1351_usb_suspend: reason = 16 requested_suspend = 1 suspended_status = 4 -> 20
[ 3.087908] c3 1 SMBCHG: is_hvdcp_present: HVDCP not support, return false
[ 3.088339] c0 1 reboot: Restarting system with command 'dm-verity device corrupted'
[ 3.088368] c0 1 Kernel command line: console=ttyHSL0,115200,n8 androidboot.console=ttyHSL0 androidboot.hardware=marlin user_debug=31 ehci-hcd.park=3 lpm_levels.sleep_disabled=1 cma=32M@0-0xffffffff loop.max_part=7 buildvariant=userdebug veritykeyid=id:5d0936bec3848ab90c5dfaa8c80ab1f4f9dd5068 androidboot.bootdevice=624000.ufshc androidboot.verifiedbootstate=orange androidboot.veritymode=enforcing androidboot.keymaster=1 androidboot.serialno=HT74C0201441 androidboot.baseband=msm mdss_mdp.panel=1:dsi:0:qcom,mdss_dsi_samsung_s6e3ha3_wqhd_cmd:1:qcom,mdss_dsi_samsung_s6e3ha3_wqhd_cmd:cfg:split_dsi androidboot.slot_suffix=_a fpsimd.fpsimd_settings=0 app_setting.use_app_setting=0 kernelflag=0x00000000 debugflag=0x00000000 androidboot.hardware.revision=PVT radioflag=0x00000008 radioflagex1=0x00000000 radioflagex2=0x00000000 cpumask=0x00000000 androidboot.hardware.ddr=4096MB,Hynix,LPDDR4 androidboot.ddrinfo=00000006 androidboot.ddrsize=4GB androidboot.hardware.color=GRA00 androidboot.hardware.ufs=128[ 3.088399] c0 1 Going down for restart now
[ 3.088441] c0 1 msm_restart_prepare: restart by command: [dm-verity device corrupted]
[ 3.089028] c0 1 qcom,qpnp-power-on qpnp-power-on-11: PMIC@SID2: configuring PON for reset
[ 3.089629] c0 1 set restart reason = 0x77665508
[ 3.089635] c0 1 set restart msg = 'DM verity device corrupted'
Is this why the thing restarts now?
Indeed that looks like the issue.
Goddammit. I'm positive this is another problem with my code.
Please stand by 16 hours for more info...
(I seriously think I need more RAM and SSDs on that box.)
Hopefully you get it figured out. 16 hours is a painful turn around time :( On the c5.9xlarge instances I can do a full rebuild in 1 hour now which is pretty nice.
I might just dispatch test builds directly on Amazon AWS, as my build server is already fully-equipped to dispatch such builds -- just a bit of tweaking on the Jenkins pipeline and that should do it.
(The problem is I'm not willing to flash them on my phone. I'm gonna have to get another one just to test on it.)
Makes sense, probably good for testing at least for your threat model. My primary concern with cloud builds is just the private keys in S3. It would be neat to come up with a next level of security for cloud builds where: 1) maybe the keys are all stored with a passphrase, and you get an email when keys need to be decrypted during build, you login manually and decrypt and it stays decrypted just long enough to execute the required task. 2) or maybe just the build happens in the cloud and all signing could be done offline (this would work better with pixel 2 devices as the kernel doesn't have to be built with verity key)
Hm, key theft is a problem, but not the only one -- machines not under your control can inject malicious code more easily than those under your control.
It's probably obvious that I'm not a big fan of "other people's computers". What I should probably clarify is that I am aware that a machine under my control is running other people's "mystery meat" in order to build this thing -- but I have already assumed and contained that specific risk while minimizing the total TCB.
Yeah that's a bit beyond my threat model for sure, but I get it.
FWIW I ran into this issue (that being a kernel boot loop) after uploading my previous CopperheadOS keys to the S3 bucket and doing a build. The solution was to have the build script generate the keys - I'm not sure if this is related but I had the same symptoms. It didn't make sense to me as to why that would happen but it seems similar given the issue is related to verity.
Eugh. This is scary.
Yeah, there's a thread on reddit where I dove into the reasons why you can't install RattlesnakeOS over the top of CopperheadOS and some potential ways to workaround it.
OMG OMG. I think I know what the problem is!
Right @dan-v, but the odd thing was the factory image had the same issue, not just flashing the OTA over CopperheadOS. That's why I thought it might be relevant here.
Flashing the new image. This time the build log said this:
CERTS kernel/x509_certificate_list
- Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verifiedboot_marlin_relkeys.der.x509
- Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verifiedboot_sailfish_relkeys.der.x509
- Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verity_dev_keys.der.x509
- Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verity_rdev_keys.der.x509
- Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/keys/marlin/verity_user.der.x509
and I verified the certificate was there.
Let's hope that does the trick.
IT WORKED! Thank you so much guys!
Prerequisites
Description
Image flashed from marlin-factory-latest.tar.xz/flash-all.sh. Flashed using flash-all.sh. Flash succeeds. It bootloops and eventually gets to recovery.
The exact same flash command in the Google full factory image directory yields a working phone.
Steps to Reproduce
Expected behavior: [What you expected to happen]
Working phone booting to welcome screen.
Actual behavior: [What actually happened]
Phone boots to recovery eventually. adb shell works after mounting /system. logcat gives an error when tried via adb shell.
Platform
Linux.
Full Command Used for Setup
Script below. Derived from
build_template.go
. No changes to build process.Carrier
Unlocked Pixel XL phone, no carrier (no difference with or without SIM).
Email Notification output
Full log
How do I upload the full build log? It's like 20 MB.