Add Yocto-Layer's Recipe to OE4T Image

CalebProvost commented 3 years ago

Clarification:

Prior issue https://github.com/AESD-Course-Project/AESD-Course-Project.github.io/issues/15 successfully added the layer to the build. However, due to such long build times, current validation demonstrates that the uartserver code was not being installed/packaged with the end image.

This issue is opened to resolve the current blocker which is impeding the installation of our the binaries into the end image.

DoD:

[x] binaries compiled via bitbake -c compile final-project
[x] binaries installed to staged rootfs via bitbake -c install final-project
[x] binaries within end system's SD Card image via bitbake demo-image-full and successive SD Creation steps

CalebProvost commented 3 years ago

Discussion documentation:

Caleb Provost Tested the image this morning and it still does not contain our uartserver... ideas? https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/yocto-layer

Dan It looks like you are missing the yocto build files? What image are you building and where are you adding final-project to it?
Caleb Provost The build script (tegra_install_n_build.sh):
# Clone L4T Yocto Base
YL4T_SUCCESS="false"
git clone https://github.com/OE4T/tegra-demo-distro.git ${CUR_DIR}/tegra-demo-distro
cd "${CUR_DIR}/tegra-demo-distro/" || echo "Could not enter L4T directory"
git checkout ${BRANCH}
git submodule update --init --recursive# Clone AESD Final Project Layer & add it to L4T layer
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-aesd-final
. ./setup-env --machine ${MACHINE} --distro ${DISTRO}
bitbake-layers add-layer "${CUR_DIR}/tegra-demo-distro/layers/meta-aesd-final"
# Begin L4T Tegra Build
bitbake ${BUILD_IMAGE} && export YL4T_SUCCESS="true"
Caleb Provost At built time, bitbake shows it as a layer which it's building, and when cancelling the build and checking the layers (bitbake-layers show-recipes) it shows that it's betting built... But whenever I flash the created SD Card image... The uartserver is never in the image...
Caleb Provost The script which builds the SD Card image for flashing:
echo "Yocto Build of L4T Complete. Preping Image to flash to SD Card."
mkdir -p "${CUR_DIR}/tegraflash"
cd "${CUR_DIR}/tegraflash" || echo "Could Not Enter SD Card Staging Directory"
cp "${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz" "${CUR_DIR}/tegraflash/"
tar -xf "${CUR_DIR}/tegra-demo-distro/build/tmp/deploy/images/${MACHINE}/${BUILD_IMAGE}-${MACHINE}.tegraflash.tar.gz"
[ -f "../${BUILD_IMAGE}-${MACHINE}.img" ] && rm -rf "../${BUILD_IMAGE}-${MACHINE}.img"
./dosdcard.sh "../${BUILD_IMAGE}-${MACHINE}.img" && SDCARD_IMAGE="true"
Dan what is BUILD_IMAGE and where does this image include the final-project recipe?
Caleb Provost bitbake-layers add-layer "${CUR_DIR}/tegra-demo-distro/layers/meta-aesd-final" And BUILD_IMAGE is a passed in variable (or environment default) which for my builds is demo-image-full The build script (tegra_install_n_build.sh):
# Clone L4T Yocto Base
YL4T_SUCCESS="false"
git clone https://github.com/OE4T/tegra-demo-distro.git ${CUR_DIR}/tegra-demo-distro
cd "${CUR_DIR}/tegra-demo-distro/" || echo "Could not enter L4T directory"
git checkout ${BRANCH}
Caleb Provost More specifically, it's included with:
# Clone AESD Final Project Layer & add it to L4T layer
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-aesd-final
. ./setup-env --machine ${MACHINE} --distro ${DISTRO}
bitbake-layers add-layer "${CUR_DIR}/tegra-demo-distro/layers/meta-aesd-final"
Dan adding a layer won't include every recipe in the layer

Caleb Provost But when I run bitbake-layers show-recipes it's in the list...

Dan yes but that doesn't mean it's included in the image

Caleb Provost Crap...

Dan Look at assignment 6 for instance, to see how we did it there with a custom image

Caleb Provost So I need to add an image append huh?

Caleb Provost So... add this to local.conf? IMAGE_INSTALL_append = " final-project"

Caleb Provost I'm adding demo-image-base.bbappend to our layer now.

Caleb Provost @Dan, I've pushed an update to https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/yocto-layer and the image append fails to add the files...
Caleb Provost
ERROR: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
 0001:
*** 0002:extend_recipe_sysroot(d)
 0003:
File: '/home/user/build/tegra-demo-distro/layers/meta/classes/staging.bbclass', lineno: 582, function: extend_recipe_sysroot
 0578:                    if "/bin/" in l or "/sbin/" in l:
 0579:                        # defer /*bin/* files until last in case they need libs
 0580:                        binfiles[l] = (targetdir, dest)
 0581:                    else:
*** 0582:                        staging_copyfile(l, targetdir, dest, postinsts, seendirs)
 0583:
 0584:    # Handle deferred binfiles
 0585:    for l in binfiles:
 0586:        (targetdir, dest) = binfiles[l]
File: '/home/user/build/tegra-demo-distro/layers/meta/classes/staging.bbclass', lineno: 157, function: staging_copyfile
 0153:        os.symlink(linkto, dest)
 0154:        #bb.warn(c)
 0155:    else:
 0156:        try:
*** 0157:            os.link(c, dest)
 0158:        except OSError as err:
 0159:            if err.errno == errno.EXDEV:
 0160:                bb.utils.copyfile(c, dest)
 0161:            else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/user/tegra-demo-distro/build/tmp/sysroots-components/armv8a/gcc-runtime/sysroot-providers/virtual_aarch64-oe4t-linux-compilerlibs' -> '/home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/recipe-sysroot/tegra-demo-distro/build/tmp/sysroots-components/armv8a/gcc-runtime/sysroot-providers/virtual_aarch64-oe4t-linux-compilerlibs'
DEBUG: Python function extend_recipe_sysroot finished
Dan what is at the path referenced there? Are the subdirectories present?

Dan Have you tried building outside your docker container?

Dan Look at permissions on the files/directories referenced there, I'm worried that docker builds might be confusing things.

Dan Also attach the log file referenced
Caleb Provost log.do_patch.314
DEBUG: Executing python function extend_recipe_sysroot
NOTE: Direct dependencies are ['/home/user/build/tegra-demo-distro/layers/meta/recipes-devtools/quilt/quilt-native_0.66.bb:do_populate_sysroot', 'virtual:native:/home/user/build/tegra-demo-distro/layers/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot']

NOTE: patch-native exists in sysroot, but is stale (patch-native.f8fc7e3bfe25fc4e17d449ee25e5f2322d82189ad4bd192c27c9b2a851cce59a vs. patch-native.f8fc7e3bfe25fc4e17d449ee25e5f2322d82189ad4bd192c27c9b2a851cce59a), removing.

DEBUG: Removing manifest: /home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/recipe-sysroot-native/sysroot-providers/patch-native

DEBUG: Removing manifest: /home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/recipe-sysroot-native/sysroot-providers/patch-replacement-native
- Dan
Looks like the error message changed, now referenced /home/user/tegra-demo-distro/build/tmp/sysroots-components/x86_64/quilt-native/sysroot-providers/quilt-native?

- Dan
Do these directories exist?

- Dan
what are the permissions?

- Caleb Provost
I'm trying to settup another build system outside of docker so that I can validate the issue exists outside of docker

- Caleb Provost
Looks like I gave the wrong log. 1 sec

- Caleb Provost
Zip 
logs.zip
39 kB Zip

- Caleb Provost
what permissions for what files are you asking about?

- Caleb Provost
Above is the log to the imaged issue.
I finally got it to run without having to rebuild the entire thing from scratch (a whole lot of sym links were used...)

- Caleb Provost
Both in docker and out of docker I get the same error. I don't think its a docker container permissions issue.:
In Docker Log is .376 and our of docker log is .7253
2 files 
log.do_prepare_recipe_sysroot.376
173 kB Plain Text
Click to view details
log.do_prepare_recipe_sysroot.7253
129 kB Plain Text
Click to view details

- Dan
What parts of this path exist? /home/user/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc

- Dan
You can make this problem disappear if you just remove your new recipe, correct?

- Dan
Then the build succeeds?

- Caleb Provost
Interestingly enough, /home/user/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc doesn't exist, but /home/user/build/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc does. How is it missing the directory?
Result:
```shell
ls -alh /home/user/build/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc
-rw-r--r-- 391 user user 17 Apr 11 22:14 /home/user/build/tegra-demo-distro/build/tmp/sysroots-components/x86_64/gcc-cross-aarch64/sysroot-providers/virtual_aarch64-oe4t-linux-gcc
Caleb Provost Removing recipe now and rerunning.
Caleb Provost Successfully builds without our layer/recipe:
aesd@aesdvm:~/yocto (dl4t %=)$ docker attach dl4t 
user@f416ed90a70d:~/build/tegra-demo-distro/build$ bitbake demo-image-full
Loading cache: 100% |##############################################| Time: 0:00:01
Loaded 3940 entries from dependency cache.
Parsing recipes: 100% |############################################| Time: 0:00:00
Parsing of 2588 .bb files complete (2587 cached, 1 parsed). 3941 targets, 124 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Multiple providers are available for runtime libxcb-glx0 (libxcb, libxcb-native)
Consider defining a PREFERRED_RPROVIDER entry to match libxcb-glx0Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-oe4t-linux"
MACHINE              = "jetson-nano-2gb-devkit"
DISTRO               = "tegrademo"
DISTRO_VERSION       = "3.2+snapshot-20210415"
TUNE_FEATURES        = "aarch64 armv8a crc"
TARGET_FPU           = ""
meta                 = "HEAD:ea455ca8671d3bc2a1097989bfaabe92f3ca37ab"
meta-tegra           
contrib              = "HEAD:40701c99c42563a871b8fca0116440f0958a1124"
meta-oe              
meta-python          
meta-networking      
meta-filesystems     = "HEAD:6e99122ed81b7dcb8bffd82f0fefaed3898de8b0"
meta-virtualization  = "HEAD:438ed8b5065aac2bfd69c57e807befb42c8cc889"
meta-tegra-support   
meta-demo-ci         
meta-tegrademo       = "HEAD:c4ef10f44d92ac9f1e4725178ab0cefd9add8126"Initialising tasks: 100% |#########################################| Time: 0:00:07
Sstate summary: Wanted 0 Local 0 Network 0 Missed 0 Current 3598 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 8738 tasks of which 8738 didn't need to be rerun and all succeeded.
user@f416ed90a70d:~/build/tegra-demo-distro/build$ 
Dan Can you try just moving your recipe into one of the existing layers in tegra-demo-distro as a test point?

Dan to remove your layer from the mix...

Caleb Provost nice idea. 1 sec

Caleb Provost Also succeeds outside of docker btw (was running after I posted the prior)

Caleb Provost recipe was added to tegra-demo-distro/layers/meta-tegrademo/recipes-demo/final-project/final-project.bb Running now

Caleb Provost success.... looking for binary uartserver now...

Dan OK so take a close look at your layer config and compare that to the layer you added it to, something must be wrong there.

Caleb Provost where would the binary be staged?

Caleb Provost I ask because it's gonna take forever to validate the binary was compiled via: find tmp/ -type f -name uartserver

Caleb Provost Should I have added the image.bbappend recipe too?

Dan you'll need to add to your image, you don't need your own custom image though, you could add to the distro packages

Caleb Provost I ask because I'm basing that append off of the layer I just used.

Caleb Provost How so?

Caleb Provost How do I add the binaries to the packaged image. Everything I've been trying fails.

Caleb Provost I've tried the Assignment 6 method and that leads to the errors that started this thread.

Caleb Provost So.. The bitbake is never actually building the recipe. None of the functions are getting triggered. When I try to manually trigger a function bitbake -c do_compile final-project it fails for the same reason that started all this.

Caleb Provost Just doing a straight bitbake demo-image-full with this recipe in the layer, it never triggers it...

Caleb Provost Why is the recipe being seen, but not being ran? It's pulling it from github... just not running oe_runmake...

CalebProvost commented 3 years ago

From what I interpreted from the discussion, it wasn't a docker user permissions issue, it wasn't the way our layer is structured (proved as it had the same error inside the demo layer), it's not getting compiled by OE/Bitbake...

CalebProvost commented 3 years ago

In lieu of waiting for the entire build executing, I deleted the directory and started the build like normal until bitbake started. I then executed bitbake -c compile final-project so I could test if it was the recipe or something else entirely. The recipe now compiles successfully, with no issues, and now the binaries are showing up, unlike before: file /home/user/build/tegra-demo-distro/build/tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/git/uartserver uartserver: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=3d13bc92198a46c68cdf06c996f3742106e703bc, for GNU/Linux 3.14.0, with debug_info, not stripped

Now running build like normal...

CalebProvost commented 3 years ago

Stopped the full build to quickly check and verify if the do_install portion of the recipe worked correctly.:

bitbake -c install final-project
Completed successfully and verified binary was "installed" to pre-image-packaging location with find temp/ -type f -name uartserver
Location confirmed to be tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/image/usr/bin/uartserver.

So far... nothing was changed besides deleting the existing build and starting from scratch.

CalebProvost commented 3 years ago

Just flashed and loaded the SD Card image... The uartserver binary is not on it...

CalebProvost commented 3 years ago

From the discussion with the professor (attached below), we'll need to use a distro append/creation method instead of the image append method that we were taught with assignment 6.

The key points of take-away (for me at least) is that we'll need to change our meta-aesd-final layer's structure to be a distro based off of the OE4T image we want. Then, add our recreated repo as a submodule under the OE4T's repos directory which is sym-linked to from within OE4T's layers directory. From there its apparently as easy as "clone, submodule update, setup-env and build."

Dan

How do I add the binaries to the packaged image. Everything I've been trying fails.

You can do an image install append in the distro file like you see at https://github.com/OE4T/tegra-demo-distro/blob/dunfell-l4t-r32.4.3/layers/meta-tegrademo/conf/distro/tegrademo-mender.conf#L38 tegrademo-mender.conf IMAGE_INSTALL_append_tegra = " tegra-eeprom-tool i2c-tools tegra-bup-payload"
Dan Look at this section of the yocto reference manual: https://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#image-generation-dev-environment

Dan A good way to start though is just to look at how/where a distribution is including packages and add yours there.

Caleb Provost The build I currently have running is using IMAGE_INSTALL_append = " final-project"... should this be IMAGE_INSTALL_append_tegra = " final-project" instead?

Dan no it shouldn't matter

Dan so are you saying you still don't have your package included in the build?

Caleb Provost It's still building... The only thing I could think of was that something got corrupt with the build packages... So I deleted it all and am starting from scratch. But I made sure to run bitbake -c compile final-project before doing so to validate that the recipe does actually build the binary.

Caleb Provost I just stopped it and am about to run bitbake -c install final-project in order to see if it'll stage it for the rootfs
Caleb Provost That ran without a problem... Checking to see with find tmp/ -type f -name uartserver results with:
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/image/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/git/uartserver
Caleb Provost @Dan, is image/usr/bin/ the rootfs staging area? If so... then is everything working so far.

Dan there's a separate area used for the image when you get to that point in the build, but this looks like the right place for the recipe output
... {Omitted progress notifications from Caleb Provost} ...

Caleb Provost Build just completed successfully. No issues encountered. Flashing SD Card now for final test...

Caleb Provost @Dan, It didn't work. The image still doesn't have our binary in it. You mentioned:

there's a separate area used for the image when you get to that point in the build, but this looks like the right place for the recipe output

What am I missing for it to go from the staging to inside the rootfs?
Caleb Provost Whelp... I changed it from IMAGE_INSTALL_append to IMAGE_INSTALL_append_tegra and placed that line inside the recipe itself... and now it's showing up in more locations. I don't know how to validate if any of these are going to go into the final image though.
user@1fe92c6db610:~/build/tegra-demo-distro/build$ find tmp/ -type f -name uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/packages-split/final-project-dbg/usr/bin/.debug/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/packages-split/final-project/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/package/usr/bin/.debug/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/package/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/image/usr/bin/uartserver
tmp/work/armv8a-oe4t-linux/final-project/1.0+gitAUTOINC+bb18ccaaa1-r0/git/uartserver
Dan it won't help to put in the recipe

Dan Can you please check in what you are building and then send me a link to the image install location as well as the commands you are using for setup_env and bitbake build?

Caleb Provost

check in what you are building
I don't understand what that means.

link to the image install location

like... the repo where the code that is doing this, or the physical location in my build tree?
the commands you are using for setup_env and bitbake build
docker pull calebprovost/dockter-l4t:sdk_installed
docker run -it -v /home/user/build:/home/user/build --name dl4t calebprovost/dockter-l4t:sdk_installed bash
then within the docker container:
git clone https://github.com/OE4T/tegra-demo-distro.git tegra-demo-distro
cd tegra-demo-distro
git checkout c4ef10f44d92ac9f1e4725178ab0cefd9add8126
git submodule update --init --recursive
git clone https://github.com/cu-ecen-5013/final-project-CalebProvost.git --branch yocto-layer layers/meta-aesd-final
. ./setup-env --machine jetson-nano-2gb-devkit --distro tegrademo
bitbake-layers add-layer ../layers/meta-aesd-final
bitbake demo-image-full
Dan Where are you adding your IMAGE_INSTALL package?

Dan

I don't understand what that means.

Typically what you'd do here is add your meta-aesd-final as a submodule under the repos directory

Dan and then symlink from layers

Dan and then make a new fork of tegra-demo-distro with your customized content

Dan so then all you need to do is clone, submodule update, setup-env and build

CalebProvost commented 3 years ago

I shared the output build tree with Dan and asked if the latest changes actually look like it'll have our code in it. He says it looks like it should be good...

The commit to the changes which should be the new structure which yields our work in the image:
https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/eb528d29b93e22669ea777678928a2d766027269

Flashing the image to SD card now, confirmation pending...

CalebProvost commented 3 years ago

Nope... Didn't work. Not only was our code not in it, it also was not the full demo image...

I'm just gonna copy and paste what already exists with OE4T's distro has and insert our stuff into it. None of the "append" stuff we've learned is working and Dan hasn't been pointing out why... Just tells me that I need to back track.

CalebProvost commented 3 years ago

This took way too long to do.
I got it working. I just had to recreate the entire image that OE4T already did as a simple image.bbappend didn't want to work...

https://github.com/cu-ecen-5013/final-project-CalebProvost/tree/c4ca47058b9b14bc6c87c27984c22828e96ce770

AESD-Course-Project / AESD-Course-Project.github.io

Add Yocto-Layer's Recipe to OE4T Image #25

Clarification:

DoD: