Bootloop with image built from latest code

Rudd-O commented 5 years ago

Prerequisites

[x] I am running the latest version of rattlesnakeos-stack
[x] I am able to reproduce this issue without any advanced customization options like --repo-patches or --repo-apks flags

Description

Image flashed from marlin-factory-latest.tar.xz/flash-all.sh. Flashed using flash-all.sh. Flash succeeds. It bootloops and eventually gets to recovery.

The exact same flash command in the Google full factory image directory yields a working phone.

Steps to Reproduce

Clean build.
Flash.
Wait until last reboot.

Expected behavior: [What you expected to happen]

Working phone booting to welcome screen.

Actual behavior: [What actually happened]

Phone boots to recovery eventually. adb shell works after mounting /system. logcat gives an error when tried via adb shell.

Platform

Linux.

Full Command Used for Setup

Script below. Derived from build_template.go. No changes to build process.

Carrier

Unlocked Pixel XL phone, no carrier (no difference with or without SIM).

Email Notification output

aws_notify: New build is required
aws_notify:   Device: marlin
aws_notify:   Release Channel: marlin-stable
aws_notify:   Build Date: 2018.10.08.22
aws_notify:   Elapsed Time: 0hrs 0min 4sec
aws_notify:   AOSP Build: PPR2.181005.003
aws_notify:   AOSP Branch: android-9.0.0_r10
aws_notify:   Chromium Version: 69.0.3497.100
aws_notify:   F-Droid Version: 1.4
aws_notify:   F-Droid Priv Extension Version: 0.2.8

Full log

How do I upload the full build log? It's like 20 MB.

dan-v commented 5 years ago

@Rudd-O - you didn't include the majority of the details requested (version, command run, email output, etc), which are included in the template to help debug issues. I also have to ask if this is your modified version or standard latest version v9.0.6+.

Rudd-O commented 5 years ago

This is modified, the build process is identical. I can show you my script, give you a diff with the build_template.go file, everything. Will disclose everything now, save for the massive 80 MB log.

Rudd-O commented 5 years ago

My script:


#!/bin/bash

if [ $# -ne 1 ]; then
  echo "Need to specify device name as argument"
  exit 1
fi

# check if supported device
DEVICE=$1
if [ "$DEVICE" == 'sailfish' ] || [ "$DEVICE" == 'marlin' ] || [ "$DEVICE" == 'walleye' ] || [ "$DEVICE" == 'taimen' ]; then
  echo "Supported device $DEVICE - continuing build"
else 
  echo "Unsupported device $DEVICE"
  exit 1
fi

# set region
REGION=none
export AWS_DEFAULT_REGION=${REGION}

# stack name
STACK_NAME=rattlesnakeos

# version of stack running
STACK_VERSION=

# prevent default action of shutting down on exit
PREVENT_SHUTDOWN=

# force build even if no new versions exist of components
FORCE_BUILD=false

# skip chromium build if there is an existing build
SKIP_CHROMIUM_BUILD=false

# pin to specific version of android
ANDROID_VERSION="9.0"

# build type (user or userdebug)
BUILD_TYPE="user" # replaced

# build channel (stable or beta)
BUILD_CHANNEL="stable"

# user customizable things
REPO_PATCHES=
REPO_PREBUILTS=
HOSTS_FILE=

# get current instance details for reference
INSTANCE_TYPE=none
INSTANCE_REGION=none
INSTANCE_IP=none

# aws settings
AWS_KEYS_BUCKET="${STACK_NAME}-keys"
AWS_RELEASE_BUCKET="${STACK_NAME}-release"
AWS_LOGS_BUCKET="${STACK_NAME}-logs"
AWS_SNS_ARN=none

# build settings
BUILD_TARGET="release aosp_${DEVICE} ${BUILD_TYPE}"
RELEASE_URL="https://${AWS_RELEASE_BUCKET}.s3.amazonaws.com"
RELEASE_CHANNEL="${DEVICE}-${BUILD_CHANNEL}"
CHROME_CHANNEL="stable"
BUILD_DATE=$(date +%Y.%m.%d.%H)
BUILD_TIMESTAMP=$(date +%s)
BUILD_DIR="$HOME/rattlesnake-os"
CERTIFICATE_SUBJECT='/CN=RattlesnakeOS'
OFFICIAL_FDROID_KEY="43238d512c1e5eb2d6569f4a3afbf5523418b82e0a3ed1552770abb9a9c9ccab"
MARLIN_KERNEL_SOURCE_DIR="${BUILD_DIR}/kernel/google/marlin"
SECONDS=0

# urls
ANDROID_SDK_URL="https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip"
MANIFEST_URL="https://android.googlesource.com/platform/manifest"
CHROME_URL_LATEST="https://omahaproxy.appspot.com/all.json"
STACK_URL_LATEST="https://api.github.com/repos/dan-v/rattlesnakeos-stack/releases/latest"
FDROID_CLIENT_URL_LATEST="https://gitlab.com/api/v4/projects/36189/repository/tags"
FDROID_PRIV_EXT_URL_LATEST="https://gitlab.com/api/v4/projects/1481578/repository/tags"
KERNEL_SOURCE_URL="https://android.googlesource.com/kernel/msm"

STACK_UPDATE_MESSAGE=
LATEST_STACK_VERSION=
LATEST_CHROMIUM=
FDROID_CLIENT_VERSION=
FDROID_PRIV_EXT_VERSION=
AOSP_BUILD=
AOSP_BRANCH=
get_latest_versions() {
  sudo apt-get -y install jq

  # check if running latest stack
  LATEST_STACK_VERSION=$(curl -s "$STACK_URL_LATEST" | jq -r '.name' || true)
  if [ "$LATEST_STACK_VERSION" == "$STACK_VERSION" ]; then
    echo "Running the latest rattlesnakeos-stack version $LATEST_STACK_VERSION"
  else
    STACK_UPDATE_MESSAGE="WARNING: you should upgrade to the latest version: ${LATEST_STACK_VERSION}"
  fi

  # check for latest stable chromium version
  LATEST_CHROMIUM=$(curl -s "$CHROME_URL_LATEST" | jq -r '.[] | select(.os == "android") | .versions[] | select(.channel == "'$CHROME_CHANNEL'") | .current_version' || true)
  if [ -z "$LATEST_CHROMIUM" ]; then
    aws_notify_simple "ERROR: Unable to get latest Chromium version details. Stopping build."
    exit 1
  fi

  # fdroid - get latest non alpha tags from gitlab
  FDROID_CLIENT_VERSION=$(curl -s "$FDROID_CLIENT_URL_LATEST" | jq -r '[.[] | select(.name | test("^[0-9]+\\.[0-9]+")) | select(.name | contains("alpha") | not) | select(.name | contains("ota") | not)][0] | .name')
  if [ -z "$FDROID_CLIENT_VERSION" ]; then
    aws_notify_simple "ERROR: Unable to get latest F-Droid version details. Stopping build."
    exit 1
  fi
  FDROID_PRIV_EXT_VERSION=$(curl -s "$FDROID_PRIV_EXT_URL_LATEST" | jq -r '[.[] | select(.name | test("^[0-9]+\\.[0-9]+")) | select(.name | contains("alpha") | not) | select(.name | contains("ota") | not)][0] | .name')
  if [ -z "$FDROID_PRIV_EXT_VERSION" ]; then
    aws_notify_simple "ERROR: Unable to get latest F-Droid privilege extension version details. Stopping build."
    exit 1
  fi

  # attempt to automatically pick latest build version and branch. note this is likely to break with any page redesign. should also add some validation here.
  AOSP_BUILD=$(curl -s https://developers.google.com/android/images | grep -A1 "${DEVICE}" | egrep '[a-zA-Z]+ [0-9]{4}\)' | grep "${ANDROID_VERSION}" | tail -1 | cut -d"(" -f2 | cut -d"," -f1)
  if [ -z "$AOSP_BUILD" ]; then
    aws_notify_simple "ERROR: Unable to get latest AOSP build information. Stopping build. This lookup is pretty fragile and can break on any page redesign of https://developers.google.com/android/images"
    exit 1
  fi
  AOSP_BRANCH=$(curl -s https://source.android.com/setup/start/build-numbers | grep -A1 "${AOSP_BUILD}" | tail -1 | cut -f2 -d">"|cut -f1 -d"<")
  if [ -z "$AOSP_BRANCH" ]; then
    # TODO: temporary workaround until build-numbers are updated on website
    if [ "$AOSP_BUILD" == "PPR2.181005.003" ]; then
      AOSP_BRANCH="android-9.0.0_r10"
    fi
    if [ -z "$AOSP_BRANCH" ]; then
      aws_notify_simple "ERROR: Unable to get latest AOSP branch information. Stopping build. This can happen if https://source.android.com/setup/start/build-numbers hasn't been updated yet with newly released factory images."
      exit 1
    fi
  fi
}

check_for_new_versions() {
  echo "Checking if any new versions of software exist"
  needs_update=false

  # check stack version
  existing_stack_version=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/rattlesnakeos-stack/revision" - || true)
  if [ "$existing_stack_version" == "$STACK_VERSION" ]; then
    echo "Stack version ($existing_stack_version) is up to date"
  else
    echo "Last successful build (if there was one) is not with latest stack version ${STACK_VERSION}"
    needs_update=true
  fi

  # check aosp
  existing_aosp_build=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-vendor" - || true)
  if [ "$existing_aosp_build" == "$AOSP_BUILD" ]; then
    echo "AOSP build ($existing_aosp_build) is up to date"
  else
    echo "AOSP needs to be updated to ${AOSP_BUILD}"
    needs_update=true
  fi

  # check chromium
  if [ "$SKIP_CHROMIUM_BUILD" = false ]; then
    existing_chromium=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/revision" - || true)
    if [ "$existing_chromium" == "$LATEST_CHROMIUM" ]; then
      echo "Chromium build ($existing_chromium) is up to date"
    else
      echo "Chromium needs to be updated to ${LATEST_CHROMIUM}"
      needs_update=true
    fi
  else 
    echo "Skipping Chromium version check as SKIP_CHROMIUM_BUILD=true"
  fi

  # check fdroid
  existing_fdroid_client=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/fdroid/revision" - || true)
  if [ "$existing_fdroid_client" == "$FDROID_CLIENT_VERSION" ]; then
    echo "F-Droid build ($existing_fdroid_client) is up to date"
  else
    echo "F-Droid needs to be updated to ${FDROID_CLIENT_VERSION}"
    needs_update=true
  fi

  # check fdroid priv extension
  existing_fdroid_priv_version=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/fdroid-priv/revision" - || true)
  if [ "$existing_fdroid_priv_version" == "$FDROID_PRIV_EXT_VERSION" ]; then
    echo "F-Droid privilege extension build ($existing_fdroid_priv_version) is up to date"
  else
    echo "F-Droid privilege extensions needs to be updated to ${FDROID_PRIV_EXT_VERSION}"
    needs_update=true
  fi

  if [ "$needs_update" = true ]; then
    aws_notify "New build is required"
  else 
    if [ "$FORCE_BUILD" = true ]; then
      aws_notify "No build is required, but FORCE_BUILD=true"
    else
      aws_notify "RattlesnakeOS build not required as all components are already up to date."
      exit 0
    fi
  fi
}

full_run() {
  get_latest_versions
  check_for_new_versions
  aws_notify "RattlesnakeOS Build STARTED"
  setup_env
  check_chromium
  fetch_aosp_source
  setup_vendor
  aws_import_keys
  apply_patches
  # only marlin and sailfish need kernel rebuilt so that verity_key is included
  if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
    rebuild_marlin_kernel
  fi
  build_aosp
  aws_release
  checkpoint_versions
  aws_notify "RattlesnakeOS Build SUCCESS"
}

setup_env() {
  echo "=================================="
  echo "Running setup_env"
  echo "=================================="

  # setup build dir
  mkdir -p "$BUILD_DIR"

  # install packages
  sudo apt-get update
  sudo apt-get --assume-yes install openjdk-8-jdk git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip python-networkx liblz4-tool
  sudo apt-get --assume-yes build-dep "$(apt-cache search linux-image-* | awk ' { print $1 } ' | sort | egrep -v -- '(-dbg|-rt|-pae)' | grep ^linux-image-[0-9][.] | tail -1)"
  sudo apt-get --assume-yes install repo gperf jq fuseext2

  # setup android sdk (required for fdroid build)
  mkdir -p ${HOME}/sdk
  pushd ${HOME}/sdk
  if [ ! -f sdk-tools.zip ] ; then
    wget ${ANDROID_SDK_URL} -O sdk-tools.zip
fi
  unzip -o sdk-tools.zip  || {
    echo unzip failed, retrying download
    rm -f sdk-tools.zip
    wget ${ANDROID_SDK_URL} -O sdk-tools.zip
    unzip -o sdk-tools.zip
  }
  yes | ./tools/bin/sdkmanager --licenses
  ./tools/android update sdk -u --use-sdk-wrapper

  # setup git
  git config --get --global user.name || git config --global user.name 'unknown'
  git config --get --global user.email || git config --global user.email 'unknown@localhost'
  git config --global color.ui true
}

check_chromium() {
  echo "=================================="
  echo "Running check_chromium"
  echo "=================================="
  current=$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/revision" - || true)
  echo "Chromium current: $current"

  if [ "$SKIP_CHROMIUM_BUILD" = true ]; then
    if [ -z "$current" ]; then
      echo "Can't skip Chromium build as requested as Chromium hasn't been built yet previously"
    else
      echo "Skipping Chromium build as requested"
      aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
      return
    fi
  fi 

  mkdir -p $HOME/chromium
  cd $HOME/chromium
  echo "Chromium latest: $LATEST_CHROMIUM"

  if [ "$LATEST_CHROMIUM" == "$current" ]; then
    echo "Chromium latest ($LATEST_CHROMIUM) matches current ($current) - just copying s3 chromium artifact"
    aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
  else
    echo "Building chromium $LATEST_CHROMIUM"
    build_chromium $LATEST_CHROMIUM
  fi

}

build_chromium() {
  echo "=================================="
  echo "Running build_chromium"
  echo "=================================="
  CHROMIUM_REVISION=$1
  DEFAULT_VERSION=$(echo $CHROMIUM_REVISION | awk -F"." '{ printf "%s%03d52\n",$3,$4}')

  # depot tools setup
  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git $HOME/depot_tools || true
  export PATH="$PATH:$HOME/depot_tools"

  # fetch chromium 
  mkdir -p $HOME/chromium
  cd $HOME/chromium
  test -f .gclient || fetch --nohooks android
  cd src

  # checkout specific revision
  git checkout "$CHROMIUM_REVISION" -f

  # install dependencies
  echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | sudo debconf-set-selections
  sudo ./build/install-build-deps-android.sh

  # run gclient sync (runhooks will run as part of this)
  for gitdir in $( find -name .git ) ; do
    pushd $gitdir/.. || continue
    git clean -dff || { popd ; return $? ; }
    git reset --hard || { popd ; return $? ; }
    popd || return $?
  done
  yes | gclient sync --with_branch_heads --jobs 32 -RDf

  # cleanup any files in tree not part of this revision
  git clean -dff

  # reset any modifications
  git checkout -- .

  mkdir -p "$HOME"/chromium-out
  cat <<EOF > "$HOME"/chromium-out/args.gn
target_os = "android"
target_cpu = "arm64"
is_debug = false

is_official_build = true
is_component_build = false
symbol_level = 0

ffmpeg_branding = "Chrome"
proprietary_codecs = true

android_channel = "stable"
android_default_version_name = "$CHROMIUM_REVISION"
android_default_version_code = "$DEFAULT_VERSION"
EOF

  gn gen "$HOME"/chromium-out
  autoninja -C "$HOME"/chromium-out/ monochrome_public_apk

  aws s3 cp "$HOME"/chromium-out/apks/MonochromePublic.apk "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk"

  echo "${CHROMIUM_REVISION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/chromium/revision"
}

fetch_aosp_source() {
  echo "=================================="
  echo "Running fetch_aosp_source"
  echo "=================================="
  pushd "${BUILD_DIR}"
  repo init --manifest-url "$MANIFEST_URL" --manifest-branch "$AOSP_BRANCH" --depth 1 || true
  for gitdir in $(find -name .git) ; do
    pushd "$gitdir/.." || continue
    git status || { popd ; return $? ; }
    git clean -dff || { popd ; return $? ; }
    git reset --hard || { popd ; return $? ; }
    popd || return $?
  done

  # make modifications to default AOSP
  # Since we just git cleaned everything, we will have to re-copy
  # the MonochromePublic.apk file once again.
  mkdir -p ${BUILD_DIR}/external/chromium/prebuilt/arm64
  aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
  if ! grep -q "RattlesnakeOS" .repo/manifest.xml; then
    awk -i inplace \
      -v ANDROID_VERSION="$ANDROID_VERSION" \
      -v FDROID_CLIENT_VERSION="$FDROID_CLIENT_VERSION" \
      -v FDROID_PRIV_EXT_VERSION="$FDROID_PRIV_EXT_VERSION" \
      '1;/<repo-hooks in-project=/{
      print "  ";
      print "  <remote name=\"github\" fetch=\"https://github.com/RattlesnakeOS/\" revision=\"" ANDROID_VERSION "\" />";
      print "  <remote name=\"fdroid\" fetch=\"https://gitlab.com/fdroid/\" />";
      print "  <remote name=\"prepare-vendor\" fetch=\"https://github.com/anestisb/\" revision=\"master\" />";
      print "  ";
      print "  <project path=\"script\" name=\"script\" remote=\"github\" />";
      print "  <project path=\"external/chromium\" name=\"platform_external_chromium\" remote=\"github\" />";
      print "  <project path=\"packages/apps/Updater\" name=\"platform_packages_apps_Updater\" remote=\"github\" />";
      print "  <project path=\"packages/apps/F-Droid\" name=\"fdroidclient\" remote=\"fdroid\" revision=\"refs/tags/" FDROID_CLIENT_VERSION "\" />";
      print "  <project path=\"packages/apps/F-DroidPrivilegedExtension\" name=\"privileged-extension\" remote=\"fdroid\" revision=\"refs/tags/" FDROID_PRIV_EXT_VERSION "\" />";
      print "  <project path=\"vendor/android-prepare-vendor\" name=\"android-prepare-vendor\" remote=\"prepare-vendor\" />"}' .repo/manifest.xml
  else
    echo "Skipping modification of .repo/manifest.xml as they have already been made"
  fi

  # remove things from manifest
  sed -i '/chromium-webview/d' .repo/manifest.xml
  sed -i '/packages\/apps\/Browser2/d' .repo/manifest.xml
  sed -i '/packages\/apps\/Calendar/d' .repo/manifest.xml
  sed -i '/packages\/apps\/QuickSearchBox/d' .repo/manifest.xml

  # sync with retries
  for i in {1..10}; do
    repo sync -c --no-tags --no-clone-bundle --jobs 32 && break
  done

  # remove webview
  rm -rf platform/external/chromium-webview
  sed -i '/webview \\/d' build/make/target/product/core_minimal.mk

  # remove Browser2
  sed -i '/Browser2/d' build/make/target/product/core.mk

  # remove Calendar
  sed -i '/Calendar \\/d' build/make/target/product/core.mk

  # remove QuickSearchBox
  sed -i '/QuickSearchBox/d' build/make/target/product/core.mk
}

setup_vendor() {
  echo "=================================="
  echo "Running setup_vendor"
  echo "=================================="
  pushd "${BUILD_DIR}/vendor/android-prepare-vendor"
  sed -i.bkp 's/  USE_DEBUGFS=true/  USE_DEBUGFS=false/; s/  # SYS_TOOLS/  SYS_TOOLS/; s/  # _UMOUNT=/  _UMOUNT=/' execute-all.sh

  # get vendor files
  yes | "${BUILD_DIR}/vendor/android-prepare-vendor/execute-all.sh" --fuse-ext2 --device "${DEVICE}" --buildID "${AOSP_BUILD}" --output "${BUILD_DIR}/vendor/android-prepare-vendor"
  aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-vendor" --acl public-read <<< "${AOSP_BUILD}" || true

  # copy vendor files to build tree
  mkdir --parents "${BUILD_DIR}/vendor/google_devices" || true
  rm --recursive --force "${BUILD_DIR}/vendor/google_devices/$DEVICE" || true
  mv "${BUILD_DIR}/vendor/android-prepare-vendor/${DEVICE}/$(tr '[:upper:]' '[:lower:]' <<< "${AOSP_BUILD}")/vendor/google_devices/${DEVICE}" "${BUILD_DIR}/vendor/google_devices"

  # smaller devices need big brother vendor files
  if [ "$DEVICE" == 'sailfish' ]; then
    rm --recursive --force "${BUILD_DIR}/vendor/google_devices/marlin" || true
    mv "${BUILD_DIR}/vendor/android-prepare-vendor/sailfish/$(tr '[:upper:]' '[:lower:]' <<< "${AOSP_BUILD}")/vendor/google_devices/marlin" "${BUILD_DIR}/vendor/google_devices"
  fi
  if [ "$DEVICE" == 'walleye' ]; then
    rm --recursive --force "${BUILD_DIR}/vendor/google_devices/muskie" || true
    mv "${BUILD_DIR}/vendor/android-prepare-vendor/walleye/$(tr '[:upper:]' '[:lower:]' <<< "${AOSP_BUILD}")/vendor/google_devices/muskie" "${BUILD_DIR}/vendor/google_devices"
  fi

  popd
}

aws_import_keys() {
  echo "=================================="
  echo "Running aws_import_keys"
  echo "=================================="
  if [ "$(aws s3 ls "s3://${AWS_KEYS_BUCKET}/${DEVICE}" | wc -l)" == '0' ]; then
    aws_gen_keys
  else
    echo "Keys already exist for ${DEVICE} - grabbing them from S3"
    mkdir -p "${BUILD_DIR}/keys"
    aws s3 sync "s3://${AWS_KEYS_BUCKET}" "${BUILD_DIR}/keys"
  fi
}

apply_patches() {
  echo "=================================="
  echo "Running apply_patches"
  echo "=================================="
  patch_custom
  patch_apps
  patch_base_config
  patch_device_config
  patch_chromium_webview
  patch_updater
  patch_fdroid
  patch_priv_ext
  patch_launcher
}

patch_custom() {
  # allow custom patches to be applied to AOSP build tree
  patches_dir="$HOME/patches"
  if [ -z "$REPO_PATCHES" ]; then
    echo "No custom patches requested"
  else
    echo "Cloning custom patches $REPO_PATCHES to ${patches_dir}"
    git clone $REPO_PATCHES ${patches_dir}
    pushd $BUILD_DIR
    while read patch; do
      echo "Applying patch $patch"
      patch -p1 < ${patches_dir}/$patch
    done < ${patches_dir}/manifest
  fi

  # allow prebuilt applications to be added to build tree
  prebuilt_dir="$BUILD_DIR/packages/apps/Custom"
  if [ -z "$REPO_PREBUILTS" ]; then
    echo "No custom apks requested"
  else
    echo "Putting custom prebuilts from $REPO_PREBUILTS in build tree location ${prebuilt_dir}"
    git clone $REPO_PREBUILTS ${prebuilt_dir}
    for file in ${prebuilt_dir}/*/ ; do 
      package_name=$(awk -F"=" '/LOCAL_MODULE /{print $2}' $file/Android.mk)
      sed -i "\$aPRODUCT_PACKAGES += ${package_name}" ${BUILD_DIR}/build/make/target/product/core.mk
    done
  fi

  # allow custom hosts file
  hosts_file_location="$BUILD_DIR/system/core/rootdir/etc/hosts"
  if [ -z "$HOSTS_FILE" ]; then
    echo "No custom hosts file requested"
  else
    echo "Replacing hosts file with $HOSTS_FILE"
    wget -O $hosts_file_location "$HOSTS_FILE"
  fi
}

patch_base_config() {
  # enable swipe up gesture functionality as option
  sed -i 's@<bool name="config_swipe_up_gesture_setting_available">false</bool>@<bool name="config_swipe_up_gesture_setting_available">true</bool>@' ${BUILD_DIR}/frameworks/base/core/res/res/values/config.xml
}

patch_device_config() {
  # set proper model names
  sed -i 's@PRODUCT_MODEL := AOSP on msm8996@PRODUCT_MODEL := Pixel XL@' ${BUILD_DIR}/device/google/marlin/aosp_marlin.mk
  sed -i 's@PRODUCT_MANUFACTURER := google@PRODUCT_MANUFACTURER := Google@' ${BUILD_DIR}/device/google/marlin/aosp_marlin.mk
  sed -i 's@PRODUCT_MODEL := AOSP on msm8996@PRODUCT_MODEL := Pixel@' ${BUILD_DIR}/device/google/marlin/aosp_sailfish.mk
  sed -i 's@PRODUCT_MANUFACTURER := google@PRODUCT_MANUFACTURER := Google@' ${BUILD_DIR}/device/google/marlin/aosp_sailfish.mk
  sed -i 's@PRODUCT_MODEL := AOSP on taimen@PRODUCT_MODEL := Pixel 2 XL@' ${BUILD_DIR}/device/google/taimen/aosp_taimen.mk
  sed -i 's@PRODUCT_MODEL := AOSP on walleye@PRODUCT_MODEL := Pixel 2@' ${BUILD_DIR}/device/google/muskie/aosp_walleye.mk
}

patch_chromium_webview() {
  cat <<EOF > ${BUILD_DIR}/frameworks/base/core/res/res/xml/config_webview_packages.xml
<?xml version="1.0" encoding="utf-8"?>
<webviewproviders>
    <webviewprovider description="Chromium" packageName="org.chromium.chrome" availableByDefault="true">
    </webviewprovider>
</webviewproviders>
EOF
}

patch_fdroid() {
  echo "sdk.dir=${HOME}/sdk" > ${BUILD_DIR}/packages/apps/F-Droid/local.properties
  echo "sdk.dir=${HOME}/sdk" > ${BUILD_DIR}/packages/apps/F-Droid/app/local.properties
  sed -i 's/gradle assembleRelease/..\/gradlew assembleRelease/' ${BUILD_DIR}/packages/apps/F-Droid/Android.mk
  sed -i 's@fdroid_apk   := build/outputs/apk/$(fdroid_dir)-release-unsigned.apk@fdroid_apk   := build/outputs/apk/full/release/app-full-release-unsigned.apk@'  ${BUILD_DIR}/packages/apps/F-Droid/Android.mk
}

patch_apps() {
  sed -i.original "\$aPRODUCT_PACKAGES += Updater" ${BUILD_DIR}/build/make/target/product/core.mk
  sed -i.original "\$aPRODUCT_PACKAGES += F-DroidPrivilegedExtension" ${BUILD_DIR}/build/make/target/product/core.mk
  sed -i.original "\$aPRODUCT_PACKAGES += F-Droid" ${BUILD_DIR}/build/make/target/product/core.mk
  sed -i.original "\$aPRODUCT_PACKAGES += chromium" ${BUILD_DIR}/build/make/target/product/core.mk
}

patch_updater() {
  pushd "$BUILD_DIR"/packages/apps/Updater/res/values
  sed --in-place --expression "s@s3bucket@${RELEASE_URL}/@g" config.xml
}

fdpe_hash() {
  keytool -list -printcert -file "$1" | grep 'SHA256:' | tr --delete ':' | cut --delimiter ' ' --fields 3
}

patch_priv_ext() {
  unofficial_sailfish_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/sailfish/releasekey.x509.pem")
  unofficial_sailfish_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/sailfish/platform.x509.pem")
  unofficial_marlin_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/marlin/releasekey.x509.pem")
  unofficial_marlin_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/marlin/platform.x509.pem")
  unofficial_taimen_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/taimen/releasekey.x509.pem")
  unofficial_taimen_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/taimen/platform.x509.pem")
  unofficial_walleye_releasekey_hash=$(fdpe_hash "${BUILD_DIR}/keys/walleye/releasekey.x509.pem")
  unofficial_walleye_platform_hash=$(fdpe_hash "${BUILD_DIR}/keys/walleye/platform.x509.pem")

  if [ "$DEVICE" == 'marlin' ]; then
    sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_marlin_releasekey_hash}'"),\n            new Pair<>("org.fdroid.fdroid", "'${unofficial_marlin_platform_hash}'")/' \
      "${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
  fi
  if [ "$DEVICE" == 'sailfish' ]; then
    sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_sailfish_releasekey_hash}'"),\n            new Pair<>("org.fdroid.fdroid", "'${unofficial_sailfish_platform_hash}'")/' \
      "${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
  fi
  if [ "$DEVICE" == 'taimen' ]; then
    sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_taimen_releasekey_hash}'"),\n            new Pair<>("org.fdroid.fdroid", "'${unofficial_taimen_platform_hash}'")/' \
      "${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
  fi
  if [ "$DEVICE" == 'walleye' ]; then
    sed -i 's/'${OFFICIAL_FDROID_KEY}'")/'${unofficial_walleye_releasekey_hash}'"),\n            new Pair<>("org.fdroid.fdroid", "'${unofficial_walleye_platform_hash}'")/' \
      "${BUILD_DIR}/packages/apps/F-DroidPrivilegedExtension/app/src/main/java/org/fdroid/fdroid/privileged/ClientWhitelist.java"
  fi
}

patch_launcher() {
  # disable QuickSearchBox widget on home screen
  sed -i.original "s/QSB_ON_FIRST_SCREEN = true;/QSB_ON_FIRST_SCREEN = false;/" "${BUILD_DIR}/packages/apps/Launcher3/src/com/android/launcher3/config/BaseFlags.java"
  # fix compile error with uninitialized variable
  sed -i.original "s/boolean createEmptyRowOnFirstScreen;/boolean createEmptyRowOnFirstScreen = false;/" "${BUILD_DIR}/packages/apps/Launcher3/src/com/android/launcher3/provider/ImportDataTask.java"
}

rebuild_marlin_kernel() {
  echo "=================================="
  echo "Running rebuild_marlin_kernel"
  echo "=================================="
  # checkout kernel source on proper commit
  mkdir -p "${MARLIN_KERNEL_SOURCE_DIR}"
  if test -d "${MARLIN_KERNEL_SOURCE_DIR}"/.git ; then
    pushd "${MARLIN_KERNEL_SOURCE_DIR}"
    sed -i 's|url = .*|url = '"${KERNEL_SOURCE_URL}"'|' .git/config
    git fetch
    popd
  else
    git clone "${KERNEL_SOURCE_URL}" "${MARLIN_KERNEL_SOURCE_DIR}"
  fi
  # TODO: make this a bit more robust
  kernel_commit_id=$(lz4cat "${BUILD_DIR}/device/google/marlin-kernel/Image.lz4-dtb" | grep -a 'Linux version' | cut -d ' ' -f3 | cut -d'-' -f2 | sed 's/^g//g')
  cd "${MARLIN_KERNEL_SOURCE_DIR}"
  echo "Checking out kernel commit ${kernel_commit_id}"
  git checkout ${kernel_commit_id}

  # run in another shell to avoid it mucking with environment variables for normal AOSP build
  bash -c "\
    cd ${BUILD_DIR};
    . build/envsetup.sh;
    make -j$(nproc --all) dtc mkdtimg;
    export PATH=${BUILD_DIR}/out/host/linux-x86/bin:${PATH};
    ln --verbose --symbolic ${BUILD_DIR}/keys/${DEVICE}/verity_user.der.x509 ${MARLIN_KERNEL_SOURCE_DIR}/verity_user.der.x509;
    cd ${MARLIN_KERNEL_SOURCE_DIR};
    make -j$(nproc --all) ARCH=arm64 marlin_defconfig;
    make -j$(nproc --all) ARCH=arm64 CONFIG_COMPAT_VDSO=n CROSS_COMPILE=${BUILD_DIR}/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-;
    cp -f arch/arm64/boot/Image.lz4-dtb ${BUILD_DIR}/device/google/marlin-kernel/;
    rm -rf ${BUILD_DIR}/out/build_*;
  "
}

build_aosp() {
  echo "=================================="
  echo "Running build_aosp"
  echo "=================================="
  pushd "$BUILD_DIR"
  source "${BUILD_DIR}/script/setup.sh"

  choosecombo $BUILD_TARGET
  make -j $(nproc) target-files-package
  make -j $(nproc) brillo_update_payload

  bash -x "${BUILD_DIR}/script/release.sh" "$DEVICE"
}

aws_release() {
  echo "=================================="
  echo "Running aws_release"
  echo "=================================="
  pushd "${BUILD_DIR}/out"
  build_date="$(< build_number.txt)"
  build_timestamp="$(unzip -p "release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" META-INF/com/android/metadata | grep 'post-timestamp' | cut --delimiter "=" --fields 2)"

  # copy ota file to s3, update file metadata used by updater app, and remove old ota files
  read -r old_metadata <<< "$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}" -)"
  old_date="$(cut -d ' ' -f 1 <<< "${old_metadata}")"
  (
  aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" "s3://${AWS_RELEASE_BUCKET}" --acl public-read &&
  echo "${build_date} ${build_timestamp} ${AOSP_BUILD}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}" --acl public-read &&
  echo "${BUILD_TIMESTAMP}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}-true-timestamp" --acl public-read
  ) && ( aws s3 rm "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-ota_update-${old_date}.zip" || true )

  if [ "$(aws s3 ls "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-factory-latest.tar.xz" | wc -l)" == '0' ]; then
    aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-factory-${build_date}.tar.xz" "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-factory-latest.tar.xz"
  fi

  # cleanup old target files if some exist
  if [ "$(aws s3 ls "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target" | wc -l)" != '0' ]; then
    cleanup_target_files
  fi

  # copy new target file to s3
  aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-target_files-${build_date}.zip" "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target/${DEVICE}-target-files-${build_date}.zip"
}

checkpoint_versions() {
  # checkpoint stack version
  echo "${STACK_VERSION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/rattlesnakeos-stack/revision"

  # checkpoint f-droid
  echo "${FDROID_PRIV_EXT_VERSION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/fdroid-priv/revision"
  echo "${FDROID_CLIENT_VERSION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/fdroid/revision"
}

cleanup_target_files() {
  echo "=================================="
  echo "Running cleanup_target_files"
  echo "=================================="
  aws s3 sync "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target" "${BUILD_DIR}/${DEVICE}-target"
  pushd "${BUILD_DIR}/out"
  current_date="$(< build_number.txt)"
  pushd "${BUILD_DIR}/${DEVICE}-target"
  for target_file in ${DEVICE}-target-files-*.zip ; do
    old_date=$(echo "$target_file" | cut --delimiter "-" --fields 4 | cut --delimiter "." --fields 5 --complement)
    aws s3 rm "s3://${AWS_RELEASE_BUCKET}/${DEVICE}-target/${DEVICE}-target-files-${old_date}.zip" || true
  done
}

aws_notify_simple() {
  aws sns publish --region ${REGION} --topic-arn "$AWS_SNS_ARN" --message "$1"
}

aws_notify() {
  LOGOUTPUT=
  if [ ! -z "$2" ]; then
    LOGOUTPUT=$(tail -c 20000 /var/log/cloud-init-output.log)
  fi
  ELAPSED="$(($SECONDS / 3600))hrs $((($SECONDS / 60) % 60))min $(($SECONDS % 60))sec"
  aws sns publish --region ${REGION} --topic-arn "$AWS_SNS_ARN" \
    --message="$(printf "$1\n  Device: %s\n  Release Channel: %s\n  Build Date: %s\n  Elapsed Time: %s\n  AOSP Build: %s\n  AOSP Branch: %s\n  Chromium Version: %s\n  F-Droid Version: %s\n  F-Droid Priv Extension Version: %s\n%s" \
      "${DEVICE}" "${RELEASE_CHANNEL}" "${BUILD_DATE}" "${ELAPSED}" "${AOSP_BUILD}" "${AOSP_BRANCH}" "${LATEST_CHROMIUM}" "${FDROID_CLIENT_VERSION}" "${FDROID_PRIV_EXT_VERSION}" "${LOGOUTPUT}")" || true
}

aws_logging() {
  df -h
  du -chs "${BUILD_DIR}" || true
  uptime
  aws s3 cp /var/log/cloud-init-output.log "s3://${AWS_LOGS_BUCKET}/${DEVICE}/$(date +%s)"
}

aws_gen_keys() {
  gen_keys
  aws s3 sync "${BUILD_DIR}/keys" "s3://${AWS_KEYS_BUCKET}"
}

gen_keys() {
  echo "=================================="
  echo "Running gen_keys"
  echo "=================================="
  mkdir --parents "${BUILD_DIR}/keys/${DEVICE}"
  pushd "${BUILD_DIR}/keys/${DEVICE}"
  for key in {releasekey,platform,shared,media,verity} ; do
    # make_key exits with unsuccessful code 1 instead of 0, need ! to negate
    ! "${BUILD_DIR}/development/tools/make_key" "$key" "$CERTIFICATE_SUBJECT"
  done

  if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
    gen_verity_key "${DEVICE}"
  fi

  if [ "${DEVICE}" == "walleye" ] || [ "${DEVICE}" == "taimen" ]; then
    gen_avb_key "${DEVICE}"
  fi
}

gen_avb_key() {
  echo "=================================="
  echo "Running gen_avb_key"
  echo "=================================="
  pushd "$BUILD_DIR"
  openssl genrsa -out "${BUILD_DIR}/keys/$1/avb.pem" 2048
  ${BUILD_DIR}/external/avb/avbtool extract_public_key --key "${BUILD_DIR}/keys/$1/avb.pem" --output "${BUILD_DIR}/keys/$1/avb_pkmd.bin"
}

gen_verity_key() {
  echo "=================================="
  echo "Running gen_verity_key"
  echo "=================================="
  pushd "$BUILD_DIR"

  make -j 20 generate_verity_key
  "${BUILD_DIR}/out/host/linux-x86/bin/generate_verity_key" -convert "${BUILD_DIR}/keys/$1/verity.x509.pem" "${BUILD_DIR}/keys/$1/verity_key"
  make clobber
  openssl x509 -outform der -in "${BUILD_DIR}/keys/$1/verity.x509.pem" -out "${BUILD_DIR}/keys/$1/verity_user.der.x509"
}

cleanup() {
  rv=$?
  aws_logging
  if [ $rv -ne 0 ]; then
    aws_notify "RattlesnakeOS Build FAILED" 1
  fi
  if [ "${PREVENT_SHUTDOWN}" = true ]; then
    echo "Skipping shutdown"
  else
    sudo shutdown -h now
  fi
}

trap cleanup 0

set -e

aws() {
  func="$1"
  cmd="$2"
  in="$3"
  out="$4"
  if [ "$func" == "sns" ]
  then
    if [[ $7 == --message=* ]]
    then
        echo "${7#--message=}" | sed 's/^/aws_notify: /' >&2
    else
        echo "$8" | sed 's/^/aws_notify: /' >&2
    fi
  elif [ "$func" == "s3" ]
  then
    if [ "$cmd" == "cp" ]
    then
        if [[ $in == s3://* ]]
        then
            in="${in#s3://}"
            in="$HOME/s3/$in"
        fi
        if [[ $out == s3://* ]]
        then
            out="${out#s3://}"
            out="$HOME/s3/$out"
        fi
        if [ "$in" == "-" ]
        then
            mkdir -p $( dirname "$out" )
            cat - > "$out"
        elif [ "$out" == "-" ]
        then
            cat "$in"
        else
            mkdir -p $( dirname "$out" )
            cp -f "$in" "$out"
        fi
    elif [ "$cmd" == "ls" ]
    then
        if [[ $in == s3://* ]]
        then
            in="${in#s3://}"
            in="$HOME/s3/$in"
        fi
        ls -1 "$in"
    elif [ "$cmd" == "rm" ]
    then
        rm -f -- "$in"
    elif [ "$cmd" == "sync" ]
    then
        if [[ $in == s3://* ]]
        then
            in="${in#s3://}"
            in="$HOME/s3/$in"
        fi
        if [[ $out == s3://* ]]
        then
            out="${out#s3://}"
            out="$HOME/s3/$out"
        fi
        rsync -a --delete -- "$in/" "$out/"
    fi
  fi
}
gen_keys() {
    echo "This program needs the keys already present in s3://${AWS_KEYS_BUCKET}/${DEVICE}" >&2
    false
}
aws_logging()
{
    return
}
cleanup() {
  rv=$?
  if [ $rv -ne 0 ]
  then
    aws_notify "RattlesnakeOS Build FAILED"
  fi
  exit $rv
}
persist_latest_versions() {
  rm -rf env*.save
  mkdir -p s3/interstage
  cat > s3/interstage/env.$BUILD_NUMBER.save <<EOF
STACK_UPDATE_MESSAGE="$STACK_UPDATE_MESSAGE"
LATEST_STACK_VERSION="$LATEST_STACK_VERSION"
LATEST_CHROMIUM="$LATEST_CHROMIUM"
FDROID_CLIENT_VERSION="$FDROID_CLIENT_VERSION"
FDROID_PRIV_EXT_VERSION="$FDROID_PRIV_EXT_VERSION"
AOSP_BUILD="$AOSP_BUILD"
AOSP_BRANCH="$AOSP_BRANCH"
EOF
}
reload_latest_versions() {
  source s3/interstage/env.$BUILD_NUMBER.save
}
if [ "$ONLY_REPORT" == "true" ]
then
full_run() {
  get_latest_versions
  persist_latest_versions
  check_for_new_versions
}
else
full_run() {
  if [ "$STAGE" != "" ] ; then
    reload_latest_versions
    if [ "$STAGE" == "rebuild_marlin_kernel" ] ; then
      if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
        "$STAGE"
      fi
    else
      "$STAGE"
    fi
  else
    get_latest_versions
    check_for_new_versions
    aws_notify "RattlesnakeOS Build STARTED"
    setup_env
    check_chromium
    fetch_aosp_source
    setup_vendor
    aws_import_keys
    apply_patches
    # only marlin and sailfish need kernel rebuilt so that verity_key is included
    if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
      rebuild_marlin_kernel
    fi
    build_aosp
    aws_release
    checkpoint_versions
    aws_notify "RattlesnakeOS Build SUCCESS"
  fi
}
fi

RELEASE_URL=http://example.com/
full_run

Diff from your script to mine:

--- templates/build_template.go 2018-10-08 21:43:08.250800408 +0000
+++ stack-builder   2018-10-09 19:42:24.964696477 +0000
@@ -1,6 +1,4 @@
-package templates

-const BuildTemplate = `
 #!/bin/bash

 if [ $# -ne 1 ]; then
@@ -18,48 +16,48 @@
 fi

 # set region
-REGION=<% .Region %>
+REGION=none
 export AWS_DEFAULT_REGION=${REGION}

 # stack name
-STACK_NAME=<% .Name %>
+STACK_NAME=rattlesnakeos

 # version of stack running
-STACK_VERSION=<% .Version %>
+STACK_VERSION=

 # prevent default action of shutting down on exit
-PREVENT_SHUTDOWN=<% .PreventShutdown %>
+PREVENT_SHUTDOWN=

 # force build even if no new versions exist of components
-FORCE_BUILD=<% .Force %>
+FORCE_BUILD=false

 # skip chromium build if there is an existing build
-SKIP_CHROMIUM_BUILD=<% .SkipChromiumBuild %>
+SKIP_CHROMIUM_BUILD=false

 # pin to specific version of android
 ANDROID_VERSION="9.0"

 # build type (user or userdebug)
-BUILD_TYPE="user"
+BUILD_TYPE="user" # replaced

 # build channel (stable or beta)
 BUILD_CHANNEL="stable"

 # user customizable things
-REPO_PATCHES=<% .RepoPatches %>
-REPO_PREBUILTS=<% .RepoPrebuilts %>
-HOSTS_FILE=<% .HostsFile %>
+REPO_PATCHES=
+REPO_PREBUILTS=
+HOSTS_FILE=

 # get current instance details for reference
-INSTANCE_TYPE=$(curl -s http://169.254.169.254/latest/meta-data/instance-type)
-INSTANCE_REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | awk -F\" '/region/ {print $4}')
-INSTANCE_IP=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4)
+INSTANCE_TYPE=none
+INSTANCE_REGION=none
+INSTANCE_IP=none

 # aws settings
 AWS_KEYS_BUCKET="${STACK_NAME}-keys"
 AWS_RELEASE_BUCKET="${STACK_NAME}-release"
 AWS_LOGS_BUCKET="${STACK_NAME}-logs"
-AWS_SNS_ARN=$(aws --region ${REGION} sns list-topics --query 'Topics[0].TopicArn' --output text | cut -d":" -f1,2,3,4,5)":${STACK_NAME}"
+AWS_SNS_ARN=none

 # build settings
 BUILD_TARGET="release aosp_${DEVICE} ${BUILD_TYPE}"
@@ -193,10 +191,10 @@
   fi

   if [ "$needs_update" = true ]; then
-    echo "New build is required"
+    aws_notify "New build is required"
   else 
     if [ "$FORCE_BUILD" = true ]; then
-      echo "No build is required, but FORCE_BUILD=true"
+      aws_notify "No build is required, but FORCE_BUILD=true"
     else
       aws_notify "RattlesnakeOS build not required as all components are already up to date."
       exit 0
@@ -235,14 +233,21 @@
   # install packages
   sudo apt-get update
   sudo apt-get --assume-yes install openjdk-8-jdk git-core gnupg flex bison build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip python-networkx liblz4-tool
-  sudo apt-get --assume-yes build-dep "linux-image-$(uname --kernel-release)"
+  sudo apt-get --assume-yes build-dep "$(apt-cache search linux-image-* | awk ' { print $1 } ' | sort | egrep -v -- '(-dbg|-rt|-pae)' | grep ^linux-image-[0-9][.] | tail -1)"
   sudo apt-get --assume-yes install repo gperf jq fuseext2

   # setup android sdk (required for fdroid build)
   mkdir -p ${HOME}/sdk
   pushd ${HOME}/sdk
-  wget ${ANDROID_SDK_URL} -O sdk-tools.zip
-  unzip sdk-tools.zip
+  if [ ! -f sdk-tools.zip ] ; then
+   wget ${ANDROID_SDK_URL} -O sdk-tools.zip
+fi
+  unzip -o sdk-tools.zip  || {
+   echo unzip failed, retrying download
+   rm -f sdk-tools.zip
+   wget ${ANDROID_SDK_URL} -O sdk-tools.zip
+   unzip -o sdk-tools.zip
+  }
   yes | ./tools/bin/sdkmanager --licenses
   ./tools/android update sdk -u --use-sdk-wrapper

@@ -280,7 +285,7 @@
     echo "Building chromium $LATEST_CHROMIUM"
     build_chromium $LATEST_CHROMIUM
   fi
-  rm -rf $HOME/chromium
+  
 }

 build_chromium() {
@@ -297,7 +302,7 @@
   # fetch chromium 
   mkdir -p $HOME/chromium
   cd $HOME/chromium
-  fetch --nohooks android
+  test -f .gclient || fetch --nohooks android
   cd src

   # checkout specific revision
@@ -308,6 +313,12 @@
   sudo ./build/install-build-deps-android.sh

   # run gclient sync (runhooks will run as part of this)
+  for gitdir in $( find -name .git ) ; do
+   pushd $gitdir/.. || continue
+   git clean -dff || { popd ; return $? ; }
+   git reset --hard || { popd ; return $? ; }
+   popd || return $?
+  done
   yes | gclient sync --with_branch_heads --jobs 32 -RDf

   # cleanup any files in tree not part of this revision
@@ -316,8 +327,8 @@
   # reset any modifications
   git checkout -- .

-  mkdir -p out/Default
-  cat <<EOF > out/Default/args.gn
+  mkdir -p "$HOME"/chromium-out
+  cat <<EOF > "$HOME"/chromium-out/args.gn
 target_os = "android"
 target_cpu = "arm64"
 is_debug = false
@@ -334,12 +345,12 @@
 android_default_version_code = "$DEFAULT_VERSION"
 EOF

-  gn gen out/Default
-  autoninja -C out/Default/ monochrome_public_apk
+  gn gen "$HOME"/chromium-out
+  autoninja -C "$HOME"/chromium-out/ monochrome_public_apk

-  mkdir -p ${BUILD_DIR}/external/chromium/prebuilt/arm64
-  cp out/Default/apks/MonochromePublic.apk ${BUILD_DIR}/external/chromium/prebuilt/arm64/
-  aws s3 cp "${BUILD_DIR}/external/chromium/prebuilt/arm64/MonochromePublic.apk" "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk"
+  
+  aws s3 cp "$HOME"/chromium-out/apks/MonochromePublic.apk "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk"
+ 
   echo "${CHROMIUM_REVISION}" | aws s3 cp - "s3://${AWS_RELEASE_BUCKET}/chromium/revision"
 }

@@ -349,8 +360,20 @@
   echo "=================================="
   pushd "${BUILD_DIR}"
   repo init --manifest-url "$MANIFEST_URL" --manifest-branch "$AOSP_BRANCH" --depth 1 || true
+  for gitdir in $(find -name .git) ; do
+    pushd "$gitdir/.." || continue
+    git status || { popd ; return $? ; }
+    git clean -dff || { popd ; return $? ; }
+    git reset --hard || { popd ; return $? ; }
+    popd || return $?
+  done
+

   # make modifications to default AOSP
+  # Since we just git cleaned everything, we will have to re-copy
+  # the MonochromePublic.apk file once again.
+  mkdir -p ${BUILD_DIR}/external/chromium/prebuilt/arm64
+  aws s3 cp "s3://${AWS_RELEASE_BUCKET}/chromium/MonochromePublic.apk" ${BUILD_DIR}/external/chromium/prebuilt/arm64/
   if ! grep -q "RattlesnakeOS" .repo/manifest.xml; then
     awk -i inplace \
       -v ANDROID_VERSION="$ANDROID_VERSION" \
@@ -581,7 +604,14 @@
   echo "=================================="
   # checkout kernel source on proper commit
   mkdir -p "${MARLIN_KERNEL_SOURCE_DIR}"
-  git clone "${KERNEL_SOURCE_URL}" "${MARLIN_KERNEL_SOURCE_DIR}"
+  if test -d "${MARLIN_KERNEL_SOURCE_DIR}"/.git ; then
+   pushd "${MARLIN_KERNEL_SOURCE_DIR}"
+   sed -i 's|url = .*|url = '"${KERNEL_SOURCE_URL}"'|' .git/config
+   git fetch
+   popd
+  else
+   git clone "${KERNEL_SOURCE_URL}" "${MARLIN_KERNEL_SOURCE_DIR}"
+  fi
   # TODO: make this a bit more robust
   kernel_commit_id=$(lz4cat "${BUILD_DIR}/device/google/marlin-kernel/Image.lz4-dtb" | grep -a 'Linux version' | cut -d ' ' -f3 | cut -d'-' -f2 | sed 's/^g//g')
   cd "${MARLIN_KERNEL_SOURCE_DIR}"
@@ -614,7 +644,7 @@
   make -j $(nproc) target-files-package
   make -j $(nproc) brillo_update_payload

-  "${BUILD_DIR}/script/release.sh" "$DEVICE"
+  bash -x "${BUILD_DIR}/script/release.sh" "$DEVICE"
 }

 aws_release() {
@@ -626,7 +656,7 @@
   build_timestamp="$(unzip -p "release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" META-INF/com/android/metadata | grep 'post-timestamp' | cut --delimiter "=" --fields 2)"

   # copy ota file to s3, update file metadata used by updater app, and remove old ota files
-  read -r old_metadata <<< "$(wget -O - "${RELEASE_URL}/${RELEASE_CHANNEL}")"
+  read -r old_metadata <<< "$(aws s3 cp "s3://${AWS_RELEASE_BUCKET}/${RELEASE_CHANNEL}" -)"
   old_date="$(cut -d ' ' -f 1 <<< "${old_metadata}")"
   (
   aws s3 cp "${BUILD_DIR}/out/release-${DEVICE}-${build_date}/${DEVICE}-ota_update-${build_date}.zip" "s3://${AWS_RELEASE_BUCKET}" --acl public-read &&
@@ -681,8 +711,8 @@
   fi
   ELAPSED="$(($SECONDS / 3600))hrs $((($SECONDS / 60) % 60))min $(($SECONDS % 60))sec"
   aws sns publish --region ${REGION} --topic-arn "$AWS_SNS_ARN" \
-    --message="$(printf "$1\n  Device: %s\n  Stack Name: %s\n  Stack Version: %s %s\n  Stack Region: %s\n  Release Channel: %s\n  Instance Type: %s\n  Instance Region: %s\n  Instance IP: %s\n  Build Date: %s\n  Elapsed Time: %s\n  AOSP Build: %s\n  AOSP Branch: %s\n  Chromium Version: %s\n  F-Droid Version: %s\n  F-Droid Priv Extension Version: %s\n%s" \
-      "${DEVICE}" "${STACK_NAME}" "${STACK_VERSION}" "${STACK_UPDATE_MESSAGE}" "${REGION}" "${RELEASE_CHANNEL}" "${INSTANCE_TYPE}" "${INSTANCE_REGION}" "${INSTANCE_IP}" "${BUILD_DATE}" "${ELAPSED}" "${AOSP_BUILD}" "${AOSP_BRANCH}" "${LATEST_CHROMIUM}" "${FDROID_CLIENT_VERSION}" "${FDROID_PRIV_EXT_VERSION}" "${LOGOUTPUT}")" || true
+    --message="$(printf "$1\n  Device: %s\n  Release Channel: %s\n  Build Date: %s\n  Elapsed Time: %s\n  AOSP Build: %s\n  AOSP Branch: %s\n  Chromium Version: %s\n  F-Droid Version: %s\n  F-Droid Priv Extension Version: %s\n%s" \
+      "${DEVICE}" "${RELEASE_CHANNEL}" "${BUILD_DATE}" "${ELAPSED}" "${AOSP_BUILD}" "${AOSP_BRANCH}" "${LATEST_CHROMIUM}" "${FDROID_CLIENT_VERSION}" "${FDROID_PRIV_EXT_VERSION}" "${LOGOUTPUT}")" || true
 }

 aws_logging() {
@@ -755,5 +785,142 @@

 set -e

+aws() {
+  func="$1"
+  cmd="$2"
+  in="$3"
+  out="$4"
+  if [ "$func" == "sns" ]
+  then
+   if [[ $7 == --message=* ]]
+   then
+       echo "${7#--message=}" | sed 's/^/aws_notify: /' >&2
+   else
+       echo "$8" | sed 's/^/aws_notify: /' >&2
+   fi
+  elif [ "$func" == "s3" ]
+  then
+   if [ "$cmd" == "cp" ]
+   then
+       if [[ $in == s3://* ]]
+       then
+           in="${in#s3://}"
+           in="$HOME/s3/$in"
+       fi
+       if [[ $out == s3://* ]]
+       then
+           out="${out#s3://}"
+           out="$HOME/s3/$out"
+       fi
+       if [ "$in" == "-" ]
+       then
+           mkdir -p $( dirname "$out" )
+           cat - > "$out"
+       elif [ "$out" == "-" ]
+       then
+           cat "$in"
+       else
+           mkdir -p $( dirname "$out" )
+           cp -f "$in" "$out"
+       fi
+   elif [ "$cmd" == "ls" ]
+   then
+       if [[ $in == s3://* ]]
+       then
+           in="${in#s3://}"
+           in="$HOME/s3/$in"
+       fi
+       ls -1 "$in"
+   elif [ "$cmd" == "rm" ]
+   then
+       rm -f -- "$in"
+   elif [ "$cmd" == "sync" ]
+   then
+       if [[ $in == s3://* ]]
+       then
+           in="${in#s3://}"
+           in="$HOME/s3/$in"
+       fi
+       if [[ $out == s3://* ]]
+       then
+           out="${out#s3://}"
+           out="$HOME/s3/$out"
+       fi
+       rsync -a --delete -- "$in/" "$out/"
+   fi
+  fi
+}
+gen_keys() {
+   echo "This program needs the keys already present in s3://${AWS_KEYS_BUCKET}/${DEVICE}" >&2
+   false
+}
+aws_logging()
+{
+   return
+}
+cleanup() {
+  rv=$?
+  if [ $rv -ne 0 ]
+  then
+    aws_notify "RattlesnakeOS Build FAILED"
+  fi
+  exit $rv
+}
+persist_latest_versions() {
+  rm -rf env*.save
+  mkdir -p s3/interstage
+  cat > s3/interstage/env.$BUILD_NUMBER.save <<EOF
+STACK_UPDATE_MESSAGE="$STACK_UPDATE_MESSAGE"
+LATEST_STACK_VERSION="$LATEST_STACK_VERSION"
+LATEST_CHROMIUM="$LATEST_CHROMIUM"
+FDROID_CLIENT_VERSION="$FDROID_CLIENT_VERSION"
+FDROID_PRIV_EXT_VERSION="$FDROID_PRIV_EXT_VERSION"
+AOSP_BUILD="$AOSP_BUILD"
+AOSP_BRANCH="$AOSP_BRANCH"
+EOF
+}
+reload_latest_versions() {
+  source s3/interstage/env.$BUILD_NUMBER.save
+}
+if [ "$ONLY_REPORT" == "true" ]
+then
+full_run() {
+  get_latest_versions
+  persist_latest_versions
+  check_for_new_versions
+}
+else
+full_run() {
+  if [ "$STAGE" != "" ] ; then
+    reload_latest_versions
+    if [ "$STAGE" == "rebuild_marlin_kernel" ] ; then
+      if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
+        "$STAGE"
+      fi
+    else
+      "$STAGE"
+    fi
+  else
+    get_latest_versions
+    check_for_new_versions
+    aws_notify "RattlesnakeOS Build STARTED"
+    setup_env
+    check_chromium
+    fetch_aosp_source
+    setup_vendor
+    aws_import_keys
+    apply_patches
+    # only marlin and sailfish need kernel rebuilt so that verity_key is included
+    if [ "${DEVICE}" == "marlin" ] || [ "${DEVICE}" == "sailfish" ]; then
+      rebuild_marlin_kernel
+    fi
+    build_aosp
+    aws_release
+    checkpoint_versions
+    aws_notify "RattlesnakeOS Build SUCCESS"
+  fi
+}
+fi
+
+RELEASE_URL=http://example.com/
 full_run
-`

Rudd-O commented 5 years ago

I stress this was built with a 100% clean workspace directory:

sudo rm -rf rattlesnake-os rattlesnakeos-stack rattlesnakeos-stack@tmp s3 sdk tmp .android .cache .gitconfig .gsutil .repo_.gitconfig.json .repoconfig .vpython-root .vpython_cipd_cache .wget-hsts

Rudd-O commented 5 years ago

Version: > git checkout -f d45d3a49b0989497841390d593518b3611e6710e

dan-v commented 5 years ago

I'm sure it's very similar to the original, but there is likely some difference causing issues here as I've done a successful flash as of yesterday for marlin. I don't at the moment have time to dig in the differences between the two build scripts, but if I have some free cycles I'll take a peak.

Usually you get into a boot loop state without loading the boot animation screen when there is an issue with the kernel. Unfortunately that also means you can't get really any output through ADB to troubleshoot unless you have a debug cable. I'd double check you are properly building the kernel and also including the appropriate verity key with the kernel.

Hopefully you didn't lock the bootloader before doing any of this experimentation as this means you can easily get out of the situation. If you have the bootloader locked, you are basically in a situation where you now need to produce a properly booting image signed with the same key to flash on your device.

Rudd-O commented 5 years ago

Yes, absolutely no boot animation screen.

Fortunately for me, my phone is both unlocked and OEM unlocked right now -- it's sitting next to my keyboard, idling in the recovery screen.

Perhaps you can furnish me with a build logfile of one of your builds, so that I can diff it with mine, and see if there's anything different?

Rudd-O commented 5 years ago

In the meantime, here's the kernel stage build log. consoleText kernel build.log I see nothing that stands out in it.

dan-v commented 5 years ago

1539069428.txt

Rudd-O commented 5 years ago

Man, I see no meaningful differences in our kernel build. About the only things different are the order of compilation of the .o files. Will continu to investigate.

Rudd-O commented 5 years ago

I need to get a console-ramoops dump of the machine while it is running. How do I build a recovery image that will not actually bootloop back to recovery when something goes wrong?

Rudd-O commented 5 years ago

I believe I have found the bug. My script is more aggressive than yours in cleaning junk from source directories. Since, in my edition, the step fetch_aosp_source resets the device/google/marlin-kernel repository to its original state, I have to suppress the copy of the kernel in rebuild_marlin_kernel and move it to fetch_aosp_source (where I also integrate the Chromium APK).

In pseudocode:

rebuild_marlin_kernel builds kernel
rebuild_marlin_kernel hands over the kernel to build_aosp phase by copying it to $BUILD_DIR/device/google/marlin-kernel
fetch_aosp does a git clean -fxd of everything that is a git repo -- this resets the Image.lz4-dtb kernel image to the stock one shipping on the repo
fetch_aosp patches stuff after cleaning everything
build_aosp builds stuff -- this builds the incorrect kernel image into the final build products
release packages everything up neatly
Oops, my phone won't boot.

The reason we did not catch this, is that (unlike the Chromium prebuilt APKs git repo) the kernel directory actually has a file Image.lz4-dtb, so there is never a lack of kernel that would trigger a build failure.

There are too many side effects not tracked by version control in this pipeline. We should collaborate to reduce them. In all frankness, something like a Makefile to tie up the three main phases (as well as the patch phase) together sounds precisely like what we need here. I can understand patching the source in a shell script because there's no alternative, but putting build products in the source tree is usually a recipe for defective builds.

My edition of the program also does out-of-tree kernel builds now. This seriously improves the certainty that the kernel is being built pristinely and only with whatever patches your program may contain, never confused by .gitignore.

Rudd-O commented 5 years ago

(Will report soon.)

Rudd-O commented 5 years ago

Yeah, I've verified my code actually builds in the new kernel, but to no avail. As a last ditch attempt, I will attempt another full clean build with the code updates.

Rudd-O commented 5 years ago

Wait. After my latest rebuild, I found this in the ramoops log:

[    1.878308] c2      1 apcc_l3_mem_acc_corner: disabling
[    1.878312] c2      1 gfx_mem_acc_corner: disabling
[    1.878317] c2      1 kryo0-retention: disabling
[    1.878322] c2      1 kryo1-retention: disabling
[    1.878376] c2      1 gdsc_mmagic_bimc: disabling
[    1.878381] c2      1 rome_vreg: disabling
[    1.878437] c2      1 usb_otg_vreg: disabling
[    1.878445] c2      1 regulator_proxy_consumer_remove_all: removing regulator proxy consumer requests
[    1.878772] c2      1 clock_late_init: Removing enables held for handed-off clocks
[    1.879367] c1    300 [AUD] snd_soc_msm8996: uart_gpios_init done
[    1.879379] c1    300 [AUD] snd_soc_msm8996: aud,uart-disable-mbhc is not set
[    1.879390] c1    300 msm8996-asoc-snd soc:sound-9335: property qcom,tfa9888-reset-gpio not detected in node /soc/sound-9335
[    1.879509] c1    300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-dout no 67
[    1.879545] c1    300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-din no 68
[    1.879575] c1    300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-clock no 65
[    1.879601] c1    300 [AUD] snd_soc_msm8996: msm8994_init_ftm_btpcm: btpcm gpio ftm-btpcm-sync no 66
[    1.879629] c1    300 [AUD] snd_soc_msm8996: htc_msm8996_parse_gpio: gpio rcv-amp-gpio-en no 31
[    1.881506] c1    300 msm8996-asoc-snd soc:sound-9335: ASoC: platform (null) not registered
[    1.882916] c3      1 ALSA device list:
[    1.882922] c3      1   No soundcards found.
[    1.882974] c3      1 Warning: unable to open an initial console.
[    1.882995] c3      1 dm: attempting early device configuration.
[    1.883485] c3      1 device-mapper: init: adding target '0 1 android-verity 259:17    '
[    1.883498] c3      1 device-mapper: android-verity: key:id:5d0936bec3848ab90c5dfaa8c80ab1f4f9dd5068 dev:259:17
[    1.887540] c3      1 device-mapper: android-verity: bio magic_number:2952900609 protocol_version:0 table_length:266
[    1.887551] c3      1 device-mapper: android-verity: verity_table: 1 /dev/block/platform/soc/624000.ufshc/by-name/system /dev/block/platform/soc/624000.ufshc/by-name/system 4096 4096 516099 516099 sha256 a1beaf398c2d163e3ce048fa3a69dc25948d9227371ba5b44df712611ac061ad aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7
[    1.887570] c3      1 device-mapper: android-verity: keyring: key not found
[    1.887575] c3      1 device-mapper: android-verity: Signature verification failed
[    1.887585] c3      1 device-mapper: android-verity: triggering restart
[    1.887603] c3      1 htc_reboot_block reboot_block: reason=77665508 msg=dm-verity device corrupted save to /misc+00003800
[    1.887624] c3      1 htc_reboot_block reboot_block: unable to open file: /dev/block/platform/soc/624000.ufshc/by-name/misc
[    1.887629] c3      1 [BATT] reboot_consistent_command_call: save batt cycle data
[    1.887633] c3      1 [BATT] reboot_consistent_command_call: save consistent data
[    1.887638] c3      1 [BATT] change_level_by_consistent_and_store_into_emmc: Battery data not ready, don't save consistent data.
[    1.887643] c3      1  at 1887627895 (1970-02-01 07:58:06.530014065 UTC)
[    1.887653] c2     58 [TP]: [ERR]:fwu_start_reflash: Firmware image synaptics_bl71.img not available
[    1.887659] c2     58 [TP]:fwu_start_reflash: End of reflash process
[    1.887664] c2     58 [TP]:synaptics_config_updater
[    1.888246] c3      1 sd 0:0:0:4: [sde] Synchronizing SCSI cache
[    1.888621] c2     58 [TP]:synaptics_config_updater: Device config ID = 4d31763200180000000000000000000000000000000000000000000000000000
[    1.888626] c2     58 [TP]:synaptics_config_updater: Start of write config process
[    1.888971] c3      1 sd 0:0:0:3: [sdd] Synchronizing SCSI cache
[    1.889663] c3      1 sd 0:0:0:2: [sdc] Synchronizing SCSI cache
[    1.890438] c3      1 sd 0:0:0:1: [sdb] Synchronizing SCSI cache
[    1.891132] c3      1 sd 0:0:0:0: [sda] Synchronizing SCSI cache
[    1.892031] c3      1 mdss_fb_release_all: try to close unopened fb 2! from pid:1 name:swapper/0
[    1.892055] c3      1 mdss_fb_release_all: try to close unopened fb 1! from pid:1 name:swapper/0
[    1.892073] c3      1 mdss_fb_release_all: try to close unopened fb 0! from pid:1 name:swapper/0
[    1.892380] c3      1 [NFC] pn551_power_off_sequence ++
[    1.892403] c3      1 [NFC] pn551_power_off_sequence : NFC_I2C_SCL set 0 0 
[    1.893417] c3      1 [NFC] pn551_power_off_sequence : NFC_I2C_SDA set 0 0 
[    1.941394] c2     58 [TP]: fwu_get_tw_vendor_v7: tw_vendor = 0x2
[    1.941406] c2     58 [TP]:synaptics_config_updater: Image config ID = 4d31763200180000000000000000000000000000000000000000000000000000
[    1.941411] c2     58 [TP]:synaptics_config_updater: config_size = 2048
[    1.941417] c2     58 [TP]: synaptics_config_updater: crc_cksum = 66A6880E
[    1.943071] c2     58 [TP]: crc_comparison_v7: config_crc = 66A6880E, flash_crc = 66A6880E
[    1.943076] c2     58 [TP]: synaptics_config_updater: No need to update
[    1.943088] c2     58 [TP]:synaptics_rmi4_reset_device from synaptics_config_updater+0x128/0xd38, rebuild = 0
[    1.943424] c3      1 [NFC] pn551_power_off_sequence --
[    1.972125] c2     58 [TP]:synaptics_rmi4_f12_init: Function 12 max x = 1439 max y = 2559 Rx: 17 Tx: 30
[    1.972788] c2     58 [TP]synaptics_rmi4_f12_init:Wakeup Gesture range (0,0) -> (1439,2559)
[    1.973050] c2     58 [TP]:synaptics_rmi4_f12_init report data init done
[    1.981176] c2     58 [TP]:synaptics_rmi4_query_device: chip_id:3708, firmware_id:2433782
[    1.982108] c2     58 [TP]:synaptics_rmi4_query_device: config_version: 4d31763200180000000000000000000000000000000000000000000000000000
[    1.988152] c2     58 [TP]:synaptics_config_updater: End of write config process
[    1.988157] c2     58 [TP]: synaptics_config_updater end
[    2.236747] c1    376 SMBCHG: smbchg_parallel_usb_en_work: AICL at 1700
[    2.236783] c1    376 SMBCHG: smbchg_is_parallel_usb_ok: Not in fast charge, skipping
[    2.236790] c1    376 SMBCHG: smbchg_relax: relaxing: 0x00 (bit 1)
[    2.740081] c1    376 SMBCHG: smbchg_parallel_usb_en_work: AICL at 1700
[    2.740117] c1    376 SMBCHG: smbchg_is_parallel_usb_ok: Not in fast charge, skipping
[    3.086905] c3      1 SMBCHG: usb_suspend: 5 voting for 0 - off
[    3.087101] c0      4 of_batterydata_get_best_profile: Find batterydata path: qcom,m1-id1-batterydata, id_ohm=33320, raw_min=0, raw_max=16000.
[    3.087144] c0      4 of_batterydata_get_best_profile: Find batterydata path: qcom,m1-id2-batterydata, id_ohm=33320, raw_min=27501, raw_max=40000.
[    3.087158] c0      4 FG: fg_batt_profile_init: batt_full_charge_criteria_ma = 345
[    3.087164] c0      4 FG: fg_batt_profile_init: batt_fcc_ma = 3400
[    3.087170] c0      4 FG: fg_batt_profile_init: batt_capacity_mah = 3450
[    3.087175] c0      4 FG: fg_batt_profile_init: fcc_half_capacity_ma = 1600
[    3.087183] c0      4 FG: fg_batt_profile_init: failed to read profile integrity rc=-22
[    3.087192] c0      4 [BATT] Probe process: (1, 3)
[    3.087207] c0      4 [BATT] htc_battery_probe_process: catch name Unknown Battery, set batt id=255, full_ma=345, fcc_ma=3400, capacity=3450, half_capacity_ma:1600, pd_is_limited:0, batt_eoc_current_ma:100
[    3.087212] c0      4 [BATT] Probe process done.
[    3.087217] c0      4 FG: batt_profile_init: failed to initialize profile
[    3.087230] c0      4 SMBCHG: battchg_suspend: 1 voting for 0 - on
[    3.087267] c0      4 SMBCHG: smbchg_external_power_changed: usb type = 4, current_limit = 0, rc = 0
[    3.087271] c0      4 SMBCHG: smbchg_stay_awake: staying awake: 0x02 (bit 2)
[    3.087274] c0      4 SMBCHG: smbchg_vfloat_adjust_check: Starting vfloat adjustments
[    3.087370] c0      4 SMBCHG: smbchg_relax: relaxing: 0x00 (bit 2)
[    3.087598] c3      1 SMB1351 smb1351_parallel_set_property: SMB1351 OTG Boost is disabled
[    3.087604] c3      1 SMBCHG: smbchg_external_otg_regulator_disable: Disabling external OTG Boost
[    3.087608] c3      1 SMB1351 smb1351_parallel_set_chg_present: set slave present = 1 -> 0
[    3.087613] c3      1 SMB1351 smb1351_usb_suspend: reason = 16 requested_suspend = 1 suspended_status = 4 -> 20
[    3.087908] c3      1 SMBCHG: is_hvdcp_present: HVDCP not support, return false
[    3.088339] c0      1 reboot: Restarting system with command 'dm-verity device corrupted'
[    3.088368] c0      1 Kernel command line: console=ttyHSL0,115200,n8 androidboot.console=ttyHSL0 androidboot.hardware=marlin user_debug=31 ehci-hcd.park=3 lpm_levels.sleep_disabled=1 cma=32M@0-0xffffffff loop.max_part=7 buildvariant=userdebug veritykeyid=id:5d0936bec3848ab90c5dfaa8c80ab1f4f9dd5068 androidboot.bootdevice=624000.ufshc androidboot.verifiedbootstate=orange androidboot.veritymode=enforcing androidboot.keymaster=1 androidboot.serialno=HT74C0201441 androidboot.baseband=msm mdss_mdp.panel=1:dsi:0:qcom,mdss_dsi_samsung_s6e3ha3_wqhd_cmd:1:qcom,mdss_dsi_samsung_s6e3ha3_wqhd_cmd:cfg:split_dsi androidboot.slot_suffix=_a fpsimd.fpsimd_settings=0 app_setting.use_app_setting=0 kernelflag=0x00000000 debugflag=0x00000000 androidboot.hardware.revision=PVT radioflag=0x00000008 radioflagex1=0x00000000 radioflagex2=0x00000000 cpumask=0x00000000 androidboot.hardware.ddr=4096MB,Hynix,LPDDR4 androidboot.ddrinfo=00000006 androidboot.ddrsize=4GB androidboot.hardware.color=GRA00 androidboot.hardware.ufs=128[    3.088399] c0      1 Going down for restart now
[    3.088441] c0      1 msm_restart_prepare: restart by command: [dm-verity device corrupted]
[    3.089028] c0      1 qcom,qpnp-power-on qpnp-power-on-11: PMIC@SID2: configuring PON for reset
[    3.089629] c0      1 set restart reason = 0x77665508
[    3.089635] c0      1 set restart msg = 'DM verity device corrupted'

Is this why the thing restarts now?

dan-v commented 5 years ago

Indeed that looks like the issue.

Rudd-O commented 5 years ago

Goddammit. I'm positive this is another problem with my code.

Please stand by 16 hours for more info...

(I seriously think I need more RAM and SSDs on that box.)

dan-v commented 5 years ago

Hopefully you get it figured out. 16 hours is a painful turn around time :( On the c5.9xlarge instances I can do a full rebuild in 1 hour now which is pretty nice.

Rudd-O commented 5 years ago

I might just dispatch test builds directly on Amazon AWS, as my build server is already fully-equipped to dispatch such builds -- just a bit of tweaking on the Jenkins pipeline and that should do it.

Rudd-O commented 5 years ago

(The problem is I'm not willing to flash them on my phone. I'm gonna have to get another one just to test on it.)

dan-v commented 5 years ago

Makes sense, probably good for testing at least for your threat model. My primary concern with cloud builds is just the private keys in S3. It would be neat to come up with a next level of security for cloud builds where: 1) maybe the keys are all stored with a passphrase, and you get an email when keys need to be decrypted during build, you login manually and decrypt and it stays decrypted just long enough to execute the required task. 2) or maybe just the build happens in the cloud and all signing could be done offline (this would work better with pixel 2 devices as the kernel doesn't have to be built with verity key)

Rudd-O commented 5 years ago

Hm, key theft is a problem, but not the only one -- machines not under your control can inject malicious code more easily than those under your control.

It's probably obvious that I'm not a big fan of "other people's computers". What I should probably clarify is that I am aware that a machine under my control is running other people's "mystery meat" in order to build this thing -- but I have already assumed and contained that specific risk while minimizing the total TCB.

dan-v commented 5 years ago

Yeah that's a bit beyond my threat model for sure, but I get it.

stevesoltys commented 5 years ago

FWIW I ran into this issue (that being a kernel boot loop) after uploading my previous CopperheadOS keys to the S3 bucket and doing a build. The solution was to have the build script generate the keys - I'm not sure if this is related but I had the same symptoms. It didn't make sense to me as to why that would happen but it seems similar given the issue is related to verity.

Rudd-O commented 5 years ago

Eugh. This is scary.

dan-v commented 5 years ago

Yeah, there's a thread on reddit where I dove into the reasons why you can't install RattlesnakeOS over the top of CopperheadOS and some potential ways to workaround it.

Rudd-O commented 5 years ago

OMG OMG. I think I know what the problem is!

stevesoltys commented 5 years ago

Right @dan-v, but the odd thing was the factory image had the same issue, not just flashing the OTA over CopperheadOS. That's why I thought it might be relevant here.

Rudd-O commented 5 years ago

Flashing the new image. This time the build log said this:

  CERTS   kernel/x509_certificate_list
  - Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verifiedboot_marlin_relkeys.der.x509
  - Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verifiedboot_sailfish_relkeys.der.x509
  - Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verity_dev_keys.der.x509
  - Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/kernel/google/marlin/verity_rdev_keys.der.x509
  - Including cert /home/user/jenkins/workspace/RattlesnakeOS_images_master-ROOFFD3KKAISMC4G76UPIWCPKBIHZ73OD3D77HHSZLNDKO3FBUSA/rattlesnake-os/keys/marlin/verity_user.der.x509

and I verified the certificate was there.

Let's hope that does the trick.

Rudd-O commented 5 years ago

IT WORKED! Thank you so much guys!

dan-v / rattlesnakeos-stack