Resolve SD Card Image Creation

[CalebProvost]

DoD: Fix SD Card Image creation (blocker to https://github.com/AESD-Course-Project/AESD-Course-Project.github.io/issues/15#issuecomment-817432364)

[CalebProvost]

Oddly enough... Fresh build results in an image being created... I think the preexisting tools might have an issue with creating an image if one already exists. Inserted a test which will remove the image first before making it.

Testing now.

[CalebProvost]

It doesn't appear to be an actual issue... It'll fail inside the docker container... sometimes...
I think we'll need to not automate this step in order to have this step work...
Or... Automate this step outside of the docker container...

I'll work on extracting this step from the docker script and instead insert it into a script which will trigger each docker execution, then the sd card creation.

[CalebProvost]

I still have yet to determine the reasoning, but the dosdcard.sh script we're using to create the tegra's sd card image doesn't want to hand off the GPT information to the tegrra210-flash-helper.sh script which creates our image. However, because it does work outside of docker, I'll write the all-integrated-run script to to do a docker container run to build the tegra image, then do the do sd card image, then back to docker to flash the arduino with it's needed firmware.

[CalebProvost]

On second thought... because the tegra build takes so long, I'll have the integrated script do the arduino work via docker 1st, then do the tegra image via docker, then (outside of docker) create the sd card image. That way a user can unplug the arduino and wait for the tegra stuff without having to wait for hours if the arduino has issues.

[CalebProvost]

Potential new build script for yocto:

#!/bin/bash

# Set Defaults if not provided
CUR_DIR=$(pwd)
# For Yocto Build
[ -z "${MACHINE}" ] && export MACHINE="jetson-nano-2gb-devkit"
[ -z "${BRANCH}" ] && export BRANCH="master"
[ -z "${DISTRO}" ] && export DISTRO="final-project"
[ -z "${BUILD_IMAGE}" ] && export BUILD_IMAGE="demo-image-full"
[ -z "${NVIDIA_DEVNET_MIRROR}" ] && export NVIDIA_DEVNET_MIRROR="file:///home/user/sdk_downloads"

# Clone L4T Yocto Base
YL4T_SUCCESS="false"
git clone https://github.com/OE4T/tegra-demo-distro.git ${CUR_DIR}/tegra-demo-distro
cd "${CUR_DIR}/tegra-demo-distro/" || echo "Could not enter L4T directory"
git checkout ${BRANCH}
git submodule update --init --recursive

# Clone AESD Final Project
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-final-project
. ./setup-env --machine ${MACHINE} --distro ${DISTRO}

# Begin OE4T Build
bitbake ${BUILD_IMAGE} && export YL4T_SUCCESS="true"

if [ "${YL4T_SUCCESS}" = "true" ]; then
    mkdir -p "${CUR_DIR}/tegraflash"
    cd "${CUR_DIR}/tegraflash" || echo "Could Not Enter SD Card Staging Directory"
    cp "${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz" "${CUR_DIR}/tegraflash/"
    tar -xf "${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz"
    [ -f "../${BUILD_IMAGE}-${MACHINE}.img" ] && rm -rf "../${BUILD_IMAGE}-${MACHINE}.img"
    ./dosdcard.sh "../${BUILD_IMAGE}-${MACHINE}.img" && SDCARD_IMAGE="true"

    echo "" && echo ""
    echo "####################################################################################################"
    echo "Yocto has finished building the OE4T image \"${BUILD_IMAGE}\" for the \"${MACHINE}\"."
    echo "Deloyment files can be found here: ${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/"
    if [ "${SDCARD_IMAGE}" = "true" ]; then
        echo ""
        echo "SD Card image for flashing can be found here: ${CUR_DIR}/${BUILD_IMAGE}-${MACHINE}.img"
    else
        echo "Yocto Build of OE4T Complete. SD Card Image creation has failed and is known to do so within Docker."
        echo "Follow the steps echo'ed below to try again (externally from docker)"
        echo 'mkdir -p "$PWD/tegraflash" && cd "$PWD/tegraflash"'
        echo 'cp "$PWD/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz" .'
        echo 'tar -xf "$PWD/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz"'
        echo './dosdcard.sh "../${BUILD_IMAGE}-${MACHINE}.img"'
    fi
    echo "####################################################################################################"
    echo ""
    exit 0

else
    echo "Could not determine the success of Yocto building L4T; exiting..."
    exit 1
fi

[CalebProvost]

Caleb Provost 1 hour ago The output of the working SD Creation: log.log

aesd@aesdvm:/home/user/build/sdcardprep$ bash -x ./dosdcard.sh ../NAME.img + MACHINE=jetson-nano-2gb-devkit

• BOARDID=3448

• FAB=400

• ./tegra210-flash-helper.sh --sdcard -B 1048576 -s 16G -b demo-image-full flash.xml.in tegra210-p3448-0003-p3542-0000.dtb jetson-nano-2gb-devkit.cfg 0xA4000 '' boot.img demo-image-full.ext4 ../NAME.img

Board ID(3448) version(400) SKU() revision() from environment

New Dan 32 minutes ago Looks like there's a tegraflash helper script it's calling that needs bash -x? You need to trace wherever it's failing and add some debug information there, either with bash -x or echo statements or whatever.

[CalebProvost]

Caleb Provost 1 hour ago The output of the working SD Creation: log.log

aesd@aesdvm:/home/user/build/sdcardprep$ bash -x ./dosdcard.sh ../NAME.img + MACHINE=jetson-nano-2gb-devkit

* BOARDID=3448

* FAB=400

* ./tegra210-flash-helper.sh --sdcard -B 1048576 -s 16G -b demo-image-full flash.xml.in tegra210-p3448-0003-p3542-0000.dtb jetson-nano-2gb-devkit.cfg 0xA4000 '' boot.img demo-image-full.ext4 ../NAME.img

Board ID(3448) version(400) SKU() revision() from environment

New Dan 32 minutes ago Looks like there's a tegraflash helper script it's calling that needs bash -x? You need to trace wherever it's failing and add some debug information there, either with bash -x or echo statements or whatever.

The results when following his instructions:

user@1fe92c6db610:~/build/sdprep$ export MACHINE=jetson-nano-2gb-devkit && export BOARDID=${BOARDID:-3448} && export FAB=${FAB:-400}
user@1fe92c6db610:~/build/sdprep$ bash -x ./tegra210-flash-helper.sh --sdcard -B 1048576 -s 16G -b final-project-image  flash.xml.in tegra210-p3448-0003-p3542-0000.dtb jetson-nano-2gb-devkit.cfg 0xA4000 "" boot.img final-project-image.ext4 "../test.img"
+ bup_blob=0
+ keyfile=
+ keyfile_args=
+ spi_only=
+ no_flash=0
+ flash_cmd=
+ sdcard=
+ make_sdcard_args=
+ imgfile=
+ dataimg=
+ inst_args=
+ blocksize=4096
+++ basename ./tegra210-flash-helper.sh
++ getopt -n tegra210-flash-helper.sh -l bup,no-flash,sdcard,spi-only,datafile:,usb-instance: -o u:s:b:B:yc: -- --sdcard -B 1048576 -s 16G -b final-project-image flash.xml.in tegra210-p3448-0003-p3542-0000.dtb jetson-nano-2gb-devkit.cfg 0xA4000 '' boot.img final-project-image.ext4 ../test.img
+ ARGS=' --sdcard -B '\''1048576'\'' -s '\''16G'\'' -b '\''final-project-image'\'' -- '\''flash.xml.in'\'' '\''tegra210-p3448-0003-p3542-0000.dtb'\'' '\''jetson-nano-2gb-devkit.cfg'\'' '\''0xA4000'\'' '\'''\'' '\''boot.img'\'' '\''final-project-image.ext4'\'' '\''../test.img'\'''
+ '[' 0 -ne 0 ']'
+ eval set -- ' --sdcard -B '\''1048576'\'' -s '\''16G'\'' -b '\''final-project-image'\'' -- '\''flash.xml.in'\'' '\''tegra210-p3448-0003-p3542-0000.dtb'\'' '\''jetson-nano-2gb-devkit.cfg'\'' '\''0xA4000'\'' '\'''\'' '\''boot.img'\'' '\''final-project-image.ext4'\'' '\''../test.img'\'''
++ set -- --sdcard -B 1048576 -s 16G -b final-project-image -- flash.xml.in tegra210-p3448-0003-p3542-0000.dtb jetson-nano-2gb-devkit.cfg 0xA4000 '' boot.img final-project-image.ext4 ../test.img
+ unset ARGS
+ true
+ case "$1" in
+ sdcard=yes
+ shift
+ true
+ case "$1" in
+ blocksize=1048576
+ shift 2
+ true
+ case "$1" in
+ make_sdcard_args=' -s 16G'
+ shift 2
+ true
+ case "$1" in
+ make_sdcard_args=' -s 16G -b final-project-image'
+ shift 2
+ true
+ case "$1" in
+ shift
+ break
+ flash_in=flash.xml.in
+ dtb_file=tegra210-p3448-0003-p3542-0000.dtb
+ sdramcfg_file=jetson-nano-2gb-devkit.cfg
+ odmdata=0xA4000
+ boardcfg_file=
+ kernfile=boot.img
+ imgfile=final-project-image.ext4
+ shift 7
+++ dirname ./tegra210-flash-helper.sh
++ readlink -f .
+ here=/home/user/build/sdprep
+ flashapp=/home/user/build/sdprep/tegraflash.py
+ '[' -e ./flashvars ']'
+ . ./flashvars
++ mktemp cvm.bin.XXXXX
+ cvm_bin=cvm.bin.a5gxF
+ '[' -z 3448 -a -z 400 ']'
+ boardid=3448
+ boardver=400
+ boardsku=
+ boardrev=
+ echo 'Board ID(3448) version(400) SKU() revision() from environment'
Board ID(3448) version(400) SKU() revision() from environment
+ '[' 3448 = 3448 ']'
+ expr 400 '<' 300
+ dtbfab=b00
+ '[' -z '' ']'
+ boardsku=0000
+ BOARDSKU=0000
+ '[' -n '' ']'
+ DTBFILE=tegra210-p3448-0003-p3542-0000.dtb
+ '[' -f cvm.bin.a5gxF ']'
+ rm -f cvm.bin.a5gxF
+ '[' -n '' ']'
+ fuselevel=fuselevel_production
+ '[' -n '' ']'
+ BOOTDEV=mmcblk0p1
+ rm -f jetson-nano-2gb-devkit_bootblob_ver.txt
+ echo NV3
+ . bsp_version
++ BSP_BRANCH=32
++ BSP_MAJOR=5
++ BSP_MINOR=1
++ BSP_VERSION=32.5.1
+ echo '# R32 , REVISION: 5.1'
+ echo 'BOARDID=3448 BOARDSKU=0000 FAB=400'
+ date +%Y%m%d%H%M%S
++ cut '-d ' -f2
++ cksum jetson-nano-2gb-devkit_bootblob_ver.txt
+ bytes=76
++ cut '-d ' -f1
++ cksum jetson-nano-2gb-devkit_bootblob_ver.txt
+ cksum=4167525033
+ echo 'BYTES:76 CRC32:4167525033'
+ '[' -z yes ']'
+ appfile=final-project-image.ext4
+ datafile=
+ appfile_sed=
+ '[' 0 -ne 0 ']'
+ '[' 0 -eq 0 -a -z yes ']'
+ pre_sdcard_sed=-es,APPFILE,final-project-image.ext4,
+ '[' -n '' ']'
+ touch APPFILE
+ '[' '' = yes ']'
+ cp flash.xml.in flash.xml.tmp
+ sed -es,VERFILE,jetson-nano-2gb-devkit_bootblob_ver.txt, -es,DTBFILE,tegra210-p3448-0003-p3542-0000.dtb, flash.xml.tmp
+ rm flash.xml.tmp
+ boardcfg=
+ '[' -z '' ']'
+ '[' 0 -ne 0 -o yes = yes ']'
+ cmd=sign
+ binargs=
+ '[' -n '' ']'
+ flashcmd='python3 /home/user/build/sdprep/tegraflash.py  --bl cboot.bin --bct "jetson-nano-2gb-devkit.cfg" --odmdata 0xA4000  --bldtb "tegra210-p3448-0003-p3542-0000.dtb" --applet nvtboot_recovery.bin   --cfg flash.xml --chip 0x21 --cmd "sign" '
+ '[' 0 -ne 0 ']'
+ '[' -z '' ']'
+ eval 'python3 /home/user/build/sdprep/tegraflash.py  --bl cboot.bin --bct "jetson-nano-2gb-devkit.cfg" --odmdata 0xA4000  --bldtb "tegra210-p3448-0003-p3542-0000.dtb" --applet nvtboot_recovery.bin   --cfg flash.xml --chip 0x21 --cmd "sign" '
++ python3 /home/user/build/sdprep/tegraflash.py --bl cboot.bin --bct jetson-nano-2gb-devkit.cfg --odmdata 0xA4000 --bldtb tegra210-p3448-0003-p3542-0000.dtb --applet nvtboot_recovery.bin --cfg flash.xml --chip 0x21 --cmd sign
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands

[   0.1645 ] tegrasign --key None --getmode mode.txt
[   0.1719 ] Assuming zero filled SBK key
[   0.4243 ] 
[   0.5412 ] Generating RCM messages
[   0.5777 ] tegrarcm --listrcm rcm_list.xml --chip 0x21 0 --download rcm nvtboot_recovery.bin 0 0
[   0.5903 ] RCM 0 is saved as rcm_0.rcm
[   0.6168 ] RCM 1 is saved as rcm_1.rcm
[   0.6168 ] List of rcm files are saved in rcm_list.xml
[   0.6168 ] 
[   0.6169 ] Signing RCM messages
[   0.6194 ] tegrasign --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.6375 ] Assuming zero filled SBK key
[   0.6391 ] 
[   0.6391 ] Copying signature to RCM mesages
[   0.6410 ] tegrarcm --chip 0x21 0 --updatesig rcm_list_signed.xml
[   0.6501 ] 
[   0.6504 ] Parsing partition layout
[   0.7122 ] tegraparser --pt flash.xml.tmp
[   0.8749 ] 
[   0.8751 ] Creating list of images to be signed
[   1.0182 ] tegrahost --chip 0x21 0 --partitionlayout flash.xml.bin --list images_list.xml
[   1.2938 ] 
[   1.2942 ] Generating signatures
[   1.2973 ] tegrasign --key None --list images_list.xml --pubkeyhash pub_key.key
[   1.3020 ] Assuming zero filled SBK key
[   1.4597 ] 
[   1.4598 ] Generating br-bct
[   1.4925 ] tegrabct --bct jetson-nano-2gb-devkit.cfg --chip 0x21 0
[   1.4982 ] Copying Sdram info from 2 to 3 set
[   1.6015 ] 
[   1.6018 ] Updating boot device parameters
[   1.6057 ] tegrabct --bct jetson-nano-2gb-devkit.bct --chip 0x21 0 --updatedevparam flash.xml.bin
[   1.6081 ] Warning: No sdram params
[   1.6089 ] 
[   1.6089 ] Updating bl info
[   1.6118 ] tegrabct --bct jetson-nano-2gb-devkit.bct --chip 0x21 0 --updateblinfo flash.xml.bin --updatesig images_list_signed.xml
[   1.6184 ] 
[   1.6185 ] Updating secondary storage information into bct
[   1.6208 ] tegraparser --pt flash.xml.bin --chip 0x21 0 --updatecustinfo jetson-nano-2gb-devkit.bct
[   1.6815 ] 
[   1.6819 ] Updating Odmdata
[   1.6845 ] tegrabct --bct jetson-nano-2gb-devkit.bct --chip 0x21 0 --updatefields Odmdata =0xA4000
[   1.6956 ] Warning: No sdram params
[   1.6957 ] 
[   1.6958 ] Get Signed section of bct
[   1.7018 ] tegrabct --bct jetson-nano-2gb-devkit.bct --chip 0x21 0 --listbct bct_list.xml
[   1.7173 ] 
[   1.7174 ] Signing BCT
[   1.7302 ] tegrasign --key None --list bct_list.xml --pubkeyhash pub_key.key
[   1.7371 ] Assuming zero filled SBK key
[   1.7372 ] 
[   1.7373 ] Updating BCT with signature
[   1.7481 ] tegrabct --bct jetson-nano-2gb-devkit.bct --chip 0x21 0 --updatesig bct_list_signed.xml
[   1.7576 ] 
[   1.7578 ] Copying signatures
[   1.7612 ] tegrahost --chip 0x21 0 --partitionlayout flash.xml.bin --updatesig images_list_signed.xml
[   1.8065 ] 
[   1.8068 ] Updating BFS information on BCT
[   1.8116 ] tegrabct --bct jetson-nano-2gb-devkit.bct --chip 0x21 0 --updatebfsinfo flash.xml.bin
[   1.8267 ] 
[   1.8268 ] Copying signed file in /home/user/build/sdprep/signed
+ '[' -n yes ']'
+ '[' -n -es,APPFILE,final-project-image.ext4, ']'
+ rm -f signed/flash.xml.in
+ mv signed/flash.xml signed/flash.xml.in
+ sed -es,APPFILE,final-project-image.ext4, signed/flash.xml.in
+ /home/user/build/sdprep/make-sdcard -s 16G -b final-project-image signed/flash.xml ../test.img
Init...ERR: could not initialize GPT on ../test.img

[CalebProvost]

It appears docker was missing a dependency to run the ./dosdcard.sh scripts and none of the tools checked before-hand to validate they were installed or not.

Updating docker container to have dependency gdisk is the fix to this bug.

[CalebProvost]

Installed the missing dependency to the docker container and pushed the changes to docker hub.
I've updated the instructions to reflect the changes ... which btw makes things easier for end-users.

Closing as blocker is resolved.