Fail to build image - Githubissues

galvesribeiro commented 4 years ago

Hello folks!

I'm new on both yocto and Jetson Nano space so even thought this question may looks weird I've tried to search here before but no issues looks like had this issue before so here I am...

So, I've tried to build a very basic image following the quick build yocto guide but instead of using the cyclone5 layer, I've added this meta-tegra layer by following their docs. Besides that, I've downloaded the SDK manager and all the binaries for it and had it set to the NVIDIA_DEVNET_MIRROR to file:///workdir/sdkm_downloads which is where all the .deb and .tbz files that came from SDK manager are. I've also set jetson-nano-qspi-sd to the MACHINE config since I'm targeting Jetson Nano dev kit with the SD card.

So far so good, but as soon as I try to build the image, here is the output I have:

pokyuser@f5ebb91d1769:/workdir/poky/build$ bitbake core-image-minimal
Parsing recipes: 100% |######################################################################################################################################################| Time: 0:00:18
Parsing of 872 .bb files complete (0 cached, 872 parsed). 1469 targets, 62 skipped, 0 masked, 0 errors.
ERROR: No recipes available for:
  /workdir/poky/meta-tegra/recipes-graphics/vulkan/vulkan-loader_1.2.%.bbappend
  /workdir/poky/meta-tegra/recipes-graphics/vulkan/vulkan-tools_1.2.%.bbappend

Can someone shed a light on what am I missing here? Do I need another layer in order to build this?

Thank you! I really appreciate that.

elPrac commented 4 years ago

Hi @galvesribeiro

these meta-tegra append-recipes should apply to poky/meta/recipes-graphics/vulkan/vulkan-* are you sure your conf/bblayers.conf has the correct path to poky/meta?

here is a small tutorial that you can use as reference, they also use meta-tegra as BSP layer.

BR.

galvesribeiro commented 4 years ago

Hey! Thanks for the quick reply!

Here is the config""

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /workdir/poky/meta \
  /workdir/poky/meta-poky \
  /workdir/poky/meta-yocto-bsp \
  /workdir/poky/meta-tegra \
  "

And here is the poky repo I just cloned and inside of its root, the meta-tegra:

Anything wrong with it?

Thanks!

lfdmn commented 4 years ago

Look into your ./poky/meta/recipes-graphics/vulkan/ folder and check the vulkan version you have there.

This kind of issues often happen when the git branch for the repos are not in sync: all should be zeus, or dunfel, etc.

galvesribeiro commented 4 years ago

That came with the clone of polky repo, I didn't clone it myself.

Looks like the files where modified as part of the build by patches from the meta-tegra.

The polky branch I've cloned was the 3.1.2 dunfel which is what I believe the master branch of meta-tegra is targeting, right?

Thanks!

madisongh commented 4 years ago

The polky branch I've cloned was the 3.1.2 dunfel which is what I believe the master branch of meta-tegra is targeting, right?

The branch names in meta-tegra follow the branch names in poky/OE-Core. Consult this wiki page for information on which branch to use, based on the OE-Core branch and version of L4T/JetPack you want to target.

galvesribeiro commented 4 years ago

@madisongh progress! Affter re-clone, set the appropriated branches, and 1h building, it fail but with a diff error:

| aarch64-poky-linux-libtool: link: aarch64-poky-linux-gcc -march=armv8-a+crc -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0=/usr/src/debug/binutils/2.34-r0 -fdebug-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0=/usr/src/debug/binutils/2.34-r0 -fdebug-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/recipe-sysroot= -fdebug-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/recipe-sysroot-native= --sysroot=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/recipe-sysroot -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Wstack-usage=262144 -DELF_LIST_OPTIONS=TRUE -DELF_SHLIB_LIST_OPTIONS=TRUE -DELF_PLT_UNWIND_LIST_OPTIONS=FALSE -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0=/usr/src/debug/binutils/2.34-r0 -fdebug-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0=/usr/src/debug/binutils/2.34-r0 -fdebug-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/recipe-sysroot= -fdebug-prefix-map=/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/recipe-sysroot-native= -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fstack-protector-strong -Wl,-z -Wl,relro -Wl,-z -Wl,now -o .libs/ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o ldbuildid.o eaarch64linux.o eaarch64elf.o eaarch64elf32.o eaarch64elf32b.o eaarch64elfb.o earmelf.o earmelfb.o eaarch64linuxb.o eaarch64linux32.o eaarch64linux32b.o earmelfb_linux_eabi.o earmelf_linux_eabi.o ldelf.o ldelfgen.o  ../bfd/.libs/libbfd.so ../libctf/.libs/libctf.so -L/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/git/build.aarch64-poky-linux.aarch64-poky-linux/libctf/../libiberty/pic -liberty ../libiberty/libiberty.a -lz -ldl
| make[4]: Leaving directory '/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/git/build.aarch64-poky-linux.aarch64-poky-linux/ld'
| make[3]: Leaving directory '/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/git/build.aarch64-poky-linux.aarch64-poky-linux/ld'
| make[2]: Leaving directory '/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/git/build.aarch64-poky-linux.aarch64-poky-linux/ld'
| make[1]: Leaving directory '/workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/git/build.aarch64-poky-linux.aarch64-poky-linux'
| Makefile:852: recipe for target 'all' failed
| make: *** [all] Error 2
| WARNING: /workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/temp/run.do_compile.59606:1 exit 1 from 'exit 1'
| 
ERROR: Task (/workdir/poky/meta/recipes-devtools/binutils/binutils_2.34.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2778 tasks of which 152 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /workdir/poky/meta/recipes-devtools/binutils/binutils_2.34.bb:do_compile
Summary: There were 4 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
pokyuser@ef3aba02439b:/workdir/poky/build$

Any ideas?

madisongh commented 4 years ago

The log snippet doesn't include the error reported. Take a look at /workdir/poky/build/tmp/work/aarch64-poky-linux/binutils/2.34-r0/temp/log.do_compile.59606 and look for error: to see what part of the compilation is failing. If you need help deciphering the error, you can attach the log here and please include details on your build setup, such as the build configuration output that bitbake prints out at the start of the build, and info about your build host.

galvesribeiro commented 4 years ago

Hey @madisongh here is the log log.do_compile.59606.txt

Regarding the environment: I've basically setup the CROPS as documented by Setting Up to Use CROss PlatformS (CROPS).

Then git clone git://git.yoctoproject.org/poky, checkout tags/3.1.2, clone this repo, checkout the branch to the correct one from the wiki as you suggested, added the layer with bitbake, then changed the MACHINE and NVIDIA_DEVNET_MIRROR on the build config file bellow.

local.conf.txt

And then bitbake core-image-minimal. Waited for around 1 hour and I got that error message.

Please let me know if you need more info that may help.

Thanks!

elPrac commented 4 years ago

Take a look in here i think you are missing the GCC compiler setup in your local.conf. You can use this as reference. Bellow is what its working for me:

GCCVERSION = "7.%"
require contrib/conf/include/gcc-compat.conf
DEBUG_PREFIX_MAP_remove = "-fmacro-prefix-map=${WORKDIR}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR}"
CUDA_VERSION="10.0"

BR,

madisongh commented 4 years ago

I've looked at the log file, and from the errors reported it looks like there's an environmental problem - not enough memory, perhaps, to run the build. A compilation happens in stages with different processes running each stage and output piped from one stage to the next... it looks like the C++ first-pass is getting terminated early and/or the output from one pass is getting corrupted while being passed to the next.

@elPrac is right that you're probably going to want to use an older version of GCC (either 7 or 8, depending on which branch you're using) than the default that poky selects for you. I don't think that's the root cause of this particular problem, though. I don't know anything about CROPS, unfortunately, so I'm not sure what to advise in terms of allocating sufficient resources to the container. For running on bare metal or in VMs I typically use 8 CPUs with 2GiB of RAM per CPU (so 16GiB for 8 CPUs) at a minimum.

galvesribeiro commented 4 years ago

Thanks folks I'll make both attempts (increase memory on the container and GCC workaround) and report back.

I've just changed the Max memory on the Docker container and asked it to rebuild again and here is the initial output:

pokyuser@214f5dab7cc2:/workdir/poky/build$ bitbake core-image-minimal
Loading cache: 100% |########################################################################################################################| Time: 0:00:00
Loaded 1468 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "jetson-nano-qspi-sd"
DISTRO               = "poky"
DISTRO_VERSION       = "3.1.2"
TUNE_FEATURES        = "aarch64 armv8a crc"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "yocto-test-3.1.2:569b1f5d67c57de957e243997c53ec2f81dc8dfe"
meta-tegra           = "dunfell-l4t-r32.4.3:3b16d67ca372f899dc6603a2e75ae48af60f97a1"

Initialising tasks: 100% |###################################################################################################################| Time: 0:00:02
Checking sstate mirror object availability: 100% |###########################################################################################| Time: 0:00:38
Sstate summary: Wanted 247 Found 0 Missed 247 Current 959 (0% match, 79% complete)
NOTE: Executing Tasks
WARNING: The free space of /workdir/poky/build/tmp (/dev/vda1) is running low (0.992GB left)
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
WARNING: The free space of /workdir/poky/build/downloads (/dev/vda1) is running low (0.992GB left)
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!
WARNING: The free space of /workdir/poky/build/sstate-cache (/dev/vda1) is running low (0.992GB left)
ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!

Second Keyboard Interrupt, stopping...

Summary: There were 3 WARNING messages shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.

I'll fix the volume size issue and will get back here with the outcome.

Thanks!

galvesribeiro commented 4 years ago

Okey, progress!

All I had todo was to increase the disk space in the volume! Great!

Now, among this lots of things, what is the image that I should use Etcher to put in the SD Card? o.O

galvesribeiro commented 4 years ago

Okey folks, I've extracted the core-image-minimal-jetson-nano-qspi-sd.tegraflash.tar.gz and ran the dosdcard.sh and then we get this:

pokyuser@c6bc1168cb35:/workdir/sdcard/img$ ./dosdcard.sh 
Board ID(3448) version(200) SKU() revision() from environment
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands

[   0.0075 ] tegrasign --key None --getmode mode.txt
[   0.0088 ] Assuming zero filled SBK key
[   0.0091 ] 
[   0.0096 ] Generating RCM messages
[   0.0105 ] tegrarcm --listrcm rcm_list.xml --chip 0x21 0 --download rcm nvtboot_recovery.bin 0 0
[   0.0120 ] RCM 0 is saved as rcm_0.rcm
[   0.0128 ] RCM 1 is saved as rcm_1.rcm
[   0.0129 ] List of rcm files are saved in rcm_list.xml
[   0.0138 ] 
[   0.0139 ] Signing RCM messages
[   0.0153 ] tegrasign --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0165 ] Assuming zero filled SBK key
[   0.0229 ] 
[   0.0230 ] Copying signature to RCM mesages
[   0.0241 ] tegrarcm --chip 0x21 0 --updatesig rcm_list_signed.xml
[   0.0261 ] 
[   0.0261 ] Parsing partition layout
[   0.0277 ] tegraparser --pt flash.xml.tmp
[   0.0290 ] 
[   0.0291 ] Creating list of images to be signed
[   0.0312 ] tegrahost --chip 0x21 0 --partitionlayout flash.xml.bin --list images_list.xml
[   0.0368 ] 
[   0.0368 ] Generating signatures
[   0.0381 ] tegrasign --key None --list images_list.xml --pubkeyhash pub_key.key
[   0.0391 ] Assuming zero filled SBK key
[   0.0904 ] 
[   0.0905 ] Generating br-bct
[   0.0919 ] tegrabct --bct jetson-nano-qspi-sd.cfg --chip 0x21 0
[   0.0930 ] Copying Sdram info from 0 to 1 set
[   0.0941 ] Copying Sdram info from 1 to 2 set
[   0.0941 ] Copying Sdram info from 2 to 3 set
[   0.0941 ] 
[   0.0941 ] Updating boot device parameters
[   0.0957 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatedevparam flash.xml.bin
[   0.0966 ] Warning: No sdram params
[   0.0968 ] 
[   0.0969 ] Updating bl info
[   0.0979 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updateblinfo flash.xml.bin --updatesig images_list_signed.xml
[   0.0996 ] 
[   0.0996 ] Updating secondary storage information into bct
[   0.1011 ] tegraparser --pt flash.xml.bin --chip 0x21 0 --updatecustinfo jetson-nano-qspi-sd.bct
[   0.1022 ] 
[   0.1023 ] Updating Odmdata
[   0.1032 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatefields Odmdata =0x94000
[   0.1042 ] Warning: No sdram params
[   0.1044 ] 
[   0.1044 ] Get Signed section of bct
[   0.1059 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --listbct bct_list.xml
[   0.1070 ] 
[   0.1071 ] Signing BCT
[   0.1093 ] tegrasign --key None --list bct_list.xml --pubkeyhash pub_key.key
[   0.1103 ] Assuming zero filled SBK key
[   0.1117 ] 
[   0.1118 ] Updating BCT with signature
[   0.1128 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatesig bct_list_signed.xml
[   0.1145 ] 
[   0.1146 ] Copying signatures
[   0.1156 ] tegrahost --chip 0x21 0 --partitionlayout flash.xml.bin --updatesig images_list_signed.xml
[   0.1196 ] 
[   0.1197 ] Updating BFS information on BCT
[   0.1207 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatebfsinfo flash.xml.bin
[   0.1219 ] 
[   0.1219 ] Copying signed file in /workdir/sdcard/img/signed
Init...ERR: could not initialize GPT on core-image-minimal.sdcard

Any ideas?

Thanks!

madisongh commented 4 years ago

You should make sure you've got the sgdisk command available in your PATH. If it's there and you still get a failure, try adding a set -x to the top of the make-sdcard script to see more detail on what it's doing when the error occurs.

galvesribeiro commented 4 years ago

This is what I got with the -x set:

pokyuser@dbda77fc2c8a:/workdir/sdcard/img$ ./dosdcard.sh 
Board ID(3448) version(200) SKU() revision() from environment
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands

[   0.0051 ] tegrasign --key None --getmode mode.txt
[   0.0063 ] Assuming zero filled SBK key
[   0.0066 ] 
[   0.0069 ] Generating RCM messages
[   0.0081 ] tegrarcm --listrcm rcm_list.xml --chip 0x21 0 --download rcm nvtboot_recovery.bin 0 0
[   0.0090 ] RCM 0 is saved as rcm_0.rcm
[   0.0098 ] RCM 1 is saved as rcm_1.rcm
[   0.0098 ] List of rcm files are saved in rcm_list.xml
[   0.0099 ] 
[   0.0099 ] Signing RCM messages
[   0.0111 ] tegrasign --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0125 ] Assuming zero filled SBK key
[   0.0181 ] 
[   0.0182 ] Copying signature to RCM mesages
[   0.0198 ] tegrarcm --chip 0x21 0 --updatesig rcm_list_signed.xml
[   0.0220 ] 
[   0.0221 ] Parsing partition layout
[   0.0235 ] tegraparser --pt flash.xml.tmp
[   0.0253 ] 
[   0.0253 ] Creating list of images to be signed
[   0.0264 ] tegrahost --chip 0x21 0 --partitionlayout flash.xml.bin --list images_list.xml
[   0.0320 ] 
[   0.0321 ] Generating signatures
[   0.0334 ] tegrasign --key None --list images_list.xml --pubkeyhash pub_key.key
[   0.0347 ] Assuming zero filled SBK key
[   0.0897 ] 
[   0.0897 ] Generating br-bct
[   0.0911 ] tegrabct --bct jetson-nano-qspi-sd.cfg --chip 0x21 0
[   0.0921 ] Copying Sdram info from 0 to 1 set
[   0.0936 ] Copying Sdram info from 1 to 2 set
[   0.0937 ] Copying Sdram info from 2 to 3 set
[   0.0937 ] 
[   0.0937 ] Updating boot device parameters
[   0.0953 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatedevparam flash.xml.bin
[   0.0969 ] Warning: No sdram params
[   0.0974 ] 
[   0.0974 ] Updating bl info
[   0.0996 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updateblinfo flash.xml.bin --updatesig images_list_signed.xml
[   0.1031 ] 
[   0.1034 ] Updating secondary storage information into bct
[   0.1050 ] tegraparser --pt flash.xml.bin --chip 0x21 0 --updatecustinfo jetson-nano-qspi-sd.bct
[   0.1070 ] 
[   0.1071 ] Updating Odmdata
[   0.1082 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatefields Odmdata =0x94000
[   0.1091 ] Warning: No sdram params
[   0.1095 ] 
[   0.1095 ] Get Signed section of bct
[   0.1108 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --listbct bct_list.xml
[   0.1125 ] 
[   0.1126 ] Signing BCT
[   0.1151 ] tegrasign --key None --list bct_list.xml --pubkeyhash pub_key.key
[   0.1165 ] Assuming zero filled SBK key
[   0.1173 ] 
[   0.1173 ] Updating BCT with signature
[   0.1186 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatesig bct_list_signed.xml
[   0.1202 ] 
[   0.1203 ] Copying signatures
[   0.1215 ] tegrahost --chip 0x21 0 --partitionlayout flash.xml.bin --updatesig images_list_signed.xml
[   0.1266 ] 
[   0.1267 ] Updating BFS information on BCT
[   0.1283 ] tegrabct --bct jetson-nano-qspi-sd.bct --chip 0x21 0 --updatebfsinfo flash.xml.bin
[   0.1295 ] 
[   0.1295 ] Copying signed file in /workdir/sdcard/img/signed
+ set -e
++ basename /workdir/sdcard/img/make-sdcard
+ me=make-sdcard
+++ dirname /workdir/sdcard/img/make-sdcard
++ readlink -f /workdir/sdcard/img
+ here=/workdir/sdcard/img
+ declare -a PARTS
+ FINALPART=
+ DEVNAME=
+ PARTSEP=
+ OUTSYSBLK=
+ HAVEBMAPTOOL=
+ SUDO=
++ id -u
+ '[' 1000 -eq 0 ']'
+ SUDO=sudo
++ getopt -o yhs:b: -n make-sdcard -- -s 16G -b core-image-minimal signed/flash.xml
+ ARGS=' -s '\''16G'\'' -b '\''core-image-minimal'\'' -- '\''signed/flash.xml'\'''
+ '[' 0 -ne 0 ']'
+ eval set -- ' -s '\''16G'\'' -b '\''core-image-minimal'\'' -- '\''signed/flash.xml'\'''
++ set -- -s 16G -b core-image-minimal -- signed/flash.xml
+ unset ARGS
+ preconfirmed=
+ outsize=
+ basename=
+ true
+ case "$1" in
++ compute_size 16G
++ local s=16G
++ local sfx=G
++ '[' G = G -o G = K -o G = M ']'
++ s=16
++ case "$sfx" in
+++ expr 16 '*' 1000 '*' 1000 '*' 1000
++ s=16000000000
++ expr '(' 16000000000 '*' 99 / 100 + 511 ')' / 512
++ return 0
+ outsize=30937500
+ shift 2
+ true
+ case "$1" in
+ basename=core-image-minimal
+ shift 2
+ true
+ case "$1" in
+ shift
+ break
+ '[' '!' -e /workdir/sdcard/img/nvflashxmlparse ']'
+ cfgfile=signed/flash.xml
+ output=
+ '[' -z signed/flash.xml ']'
+ '[' -z '' ']'
+ '[' -z core-image-minimal ']'
+ output=core-image-minimal.sdcard
+ '[' -c core-image-minimal.sdcard ']'
+ '[' -b core-image-minimal.sdcard ']'
+ '[' -e core-image-minimal.sdcard ']'
+ '[' -z 30937500 ']'
+ mapfile PARTS
++ /workdir/sdcard/img/nvflashxmlparse -t sdcard signed/flash.xml
+ '[' 14 -eq 0 ']'
+ echo -n Init...
Init...+ '[' -b core-image-minimal.sdcard ']'
+ dd if=/dev/zero of=core-image-minimal.sdcard bs=512 count=0 seek=30937500 status=none
+ sgdisk core-image-minimal.sdcard --clear --mbrtogpt
+ echo 'ERR: could not initialize GPT on core-image-minimal.sdcard'
ERR: could not initialize GPT on core-image-minimal.sdcard
+ exit 1

madisongh commented 4 years ago

Did you verify that the sgdisk command is in your PATH? What happens if you try these two commands manually?

$ dd if=/dev/zero of=core-image-minimal.sdcard bs=512 count=0 seek=30937500 status=none
$ sgdisk core-image-minimal.sdcard --clear --mbrtogpt

galvesribeiro commented 4 years ago

I was just writing to you here but you were faster! 😄

So, the big problem is that this script requires you to run as sudo which is not possible directly on the poky container. I had to keep the container running and then run docker exec from another terminal passing -u root so I can get a shell as a root (thankfully the container didn't blocked that on its Dockerfile). I was able to run it now.

Gimme a few moments and I'll update here the outcome. Thanks!

galvesribeiro commented 4 years ago

Ok, here are more updates folks.

I was able to boot from that original/initial image but since it doesn't have anything like nvidia libs or even a package manager, I had to get back and chagne some things on the config:

PACKAGE_CLASSES ?= "package_deb" we would like to use apt instead of rpm
EXTRA_IMAGE_FEATURES ?= "debug-tweaks package-management" added package-management as we found on some docs

After that, just build core image again and it fail with this:

pokyuser@48eb097bec16:/workdir/poky/build$ bitbake core-image-minimal
Loading cache: 100% |####################################################################################################################################################################| Time: 0:00:00
Loaded 1468 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "jetson-nano-qspi-sd"
DISTRO               = "poky"
DISTRO_VERSION       = "3.1.2"
TUNE_FEATURES        = "aarch64 armv8a crc"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "yocto-test-3.1.2:569b1f5d67c57de957e243997c53ec2f81dc8dfe"
meta-tegra           = "dunfell-l4t-r32.4.3:3b16d67ca372f899dc6603a2e75ae48af60f97a1"

Initialising tasks: 100% |###############################################################################################################################################################| Time: 0:00:03
Sstate summary: Wanted 1 Found 0 Missed 1 Current 1213 (0% match, 99% complete)
NOTE: Executing Tasks
ERROR: core-image-minimal-1.0-r0 do_image_tegraflash: Execution of '/workdir/poky/build/tmp/work/jetson_nano_qspi_sd-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_tegraflash.41174' failed with exit code 1:
cp: cannot stat '/workdir/poky/build/tmp/deploy/images/jetson-nano-qspi-sd/tegra210-p3448-0000-p3449-0000-a02.dtb': No such file or directory
WARNING: /workdir/poky/build/tmp/work/jetson_nano_qspi_sd-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_tegraflash.41174:1 exit 1 from 'cp -L "/workdir/poky/build/tmp/deploy/images/jetson-nano-qspi-sd/$dtbf" ./'

ERROR: Logfile of failure stored in: /workdir/poky/build/tmp/work/jetson_nano_qspi_sd-poky-linux/core-image-minimal/1.0-r0/temp/log.do_image_tegraflash.41174
Log data follows:
| DEBUG: Executing python function set_image_size
| DEBUG: 91925.600000 = 70712 * 1.300000
| DEBUG: 14680064.000000 = max(91925.600000, 14680064)[14680064.000000] + 0
| DEBUG: 14680064.000000 = int(14680064.000000)
| DEBUG: 14680064 = aligned(14680064)
| DEBUG: returning 14680064
| DEBUG: Python function set_image_size finished
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['virtual:native:/workdir/poky/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-extended/bc/bc_1.07.1.bb:do_populate_sysroot', '/workdir/poky/meta-tegra/recipes-bsp/tegra-binaries/tegra-redundant-boot-base_32.4.3.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/perl/perl_5.30.1.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.2.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/gcc/libgcc_9.3.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/gcc/gcc-cross_9.3.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/apt/apt-native_1.2.31.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/dpkg/dpkg_1.19.7.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/mklibs/mklibs-native_0.1.44.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-connectivity/openssl/openssl_1.1.1g.bb:do_populate_sysroot', '/workdir/poky/meta-tegra/recipes-kernel/linux/linux-tegra_4.9.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/prelink/prelink_git.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/binutils/binutils-cross_2.34.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/python/python3_3.8.2.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-kernel/kmod/kmod-native_git.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/swig/swig_3.0.12.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/quilt/quilt-native_0.66.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-kernel/dtc/dtc_1.6.0.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-support/lzop/lzop_1.04.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/makedevs/makedevs_1.0.1.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/qemu/qemuwrapper-cross_1.0.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-extended/pigz/pigz_2.4.bb:do_populate_sysroot', '/workdir/poky/meta-tegra/recipes-bsp/tegra-binaries/tegra-bootfiles_32.4.3.bb:do_populate_sysroot', '/workdir/poky/meta-tegra/recipes-bsp/tegra-binaries/tegra210-flashtools-native_32.4.3.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-kernel/kmod/depmodwrapper-cross_1.0.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/bison/bison_3.5.3.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-core/glibc/glibc_2.31.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-core/glibc/cross-localedef-native_2.31.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.45.4.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/fdisk/gptfdisk_1.0.4.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-extended/xz/xz_5.2.4.bb:do_populate_sysroot', '/workdir/poky/meta/recipes-devtools/gcc/gcc-runtime_9.3.bb:do_populate_sysroot', 'virtual:native:/workdir/poky/meta/recipes-devtools/flex/flex_2.6.4.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['patch-native', 'bc-native', 'tegra-redundant-boot-base', 'perl-native', 'opkg-utils-native', 'libgcc', 'gcc-cross-aarch64', 'apt-native', 'dpkg-native', 'mklibs-native', 'openssl-native', 'linux-tegra', 'update-rc.d-native', 'prelink-native', 'binutils-cross-aarch64', 'python3-native', 'kmod-native', 'swig-native', 'quilt-native', 'dtc-native', 'ldconfig-native', 'lzop-native', 'makedevs-native', 'pseudo-native', 'qemuwrapper-cross', 'pigz-native', 'tegra-bootfiles', 'tegra210-flashtools-native', 'depmodwrapper-cross', 'bison-native', 'glibc', 'cross-localedef-native', 'kern-tools-native', 'e2fsprogs-native', 'gptfdisk-native', 'xz-native', 'gcc-runtime', 'flex-native', 'libtool-native', 'gnu-config-native', 'autoconf-native', 'automake-native', 'attr-native', 'texinfo-dummy-native', 'readline-native', 'shared-mime-info-native', 'tegra-binaries-patches', 'gdbm-native', 'db-native', 'zlib-native', 'debianutils-native', 'linux-libc-headers', 'mpfr-native', 'libmpc-native', 'gmp-native', 'curl-native', 'gettext-native', 'pkgconfig-native', 'gettext-minimal-native', 'bzip2-native', 'shadow-native', 'elfutils-native', 'binutils-native', 'libtirpc-native', 'sqlite3-native', 'libffi-native', 'util-linux-native', 'libnsl2-native', 'qemu-native', 'gtk-doc-native', 'libpcre-native', 'lzo-native', 'tegra-flashvars', 'tegra-helper-scripts-native', 'popt-native', 'ncurses-native', 'm4-native', 'itstool-native', 'libxml2-native', 'glib-2.0-native', 'libpcre2-native', 'libcap-ng-native', 'ninja-native', 'meson-native', 're2c-native', 'python3-setuptools-native', 'unzip-native']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_image_tegraflash
| cp: cannot stat '/workdir/poky/build/tmp/deploy/images/jetson-nano-qspi-sd/tegra210-p3448-0000-p3449-0000-a02.dtb': No such file or directory
| WARNING: /workdir/poky/build/tmp/work/jetson_nano_qspi_sd-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_tegraflash.41174:1 exit 1 from 'cp -L "/workdir/poky/build/tmp/deploy/images/jetson-nano-qspi-sd/$dtbf" ./'
| ERROR: Execution of '/workdir/poky/build/tmp/work/jetson_nano_qspi_sd-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_tegraflash.41174' failed with exit code 1:
| cp: cannot stat '/workdir/poky/build/tmp/deploy/images/jetson-nano-qspi-sd/tegra210-p3448-0000-p3449-0000-a02.dtb': No such file or directory
| WARNING: /workdir/poky/build/tmp/work/jetson_nano_qspi_sd-poky-linux/core-image-minimal/1.0-r0/temp/run.do_image_tegraflash.41174:1 exit 1 from 'cp -L "/workdir/poky/build/tmp/deploy/images/jetson-nano-qspi-sd/$dtbf" ./'
| 
ERROR: Task (/workdir/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_tegraflash) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3128 tasks of which 3127 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /workdir/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_tegraflash
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Any word on what may be failing? Thanks!!!

madisongh commented 4 years ago

It's trying to copy the device tree files that were built by the kernel recipe into the working area for composing the tegraflash package. Did you by any chance remove files from under $BUILDDIR/tmp/deploy after running one build and before running this build? That would confuse the build system and lead to an error like this.

Try removing everything under $BUILDDIR/tmp and run your build again. It shouldn't take too long if you've still got the shared-state cache left from your prior build(s).

galvesribeiro commented 4 years ago

Yeah! That did the trick thanks.

Well, it is booting and I was able to enable openssh and Docker on it but, looks like we're missing pieces of the nvidia libraries in meta-tegra like for example, the nvidia Container Runtime which allow me to use the GPU inside containers. Is there a flag on meta-tegra that I need to set in order to install it? Or is it really missing from this yocto layer?

Again, thanks for the patience and all help!

galvesribeiro commented 4 years ago

Also, after trying to switch to poky-tiny distro, it looks like meta-tegra and the nano machine doesn't like it even thought I've se the PREFERRED_PROVIDER_virtual/kernel ?= "linux-tegra" as suggested from the error:

pokyuser@5467f0b608c9:/workdir/poky/build$ bitbake core-image-minimal
Loading cache: 100% |##############################################################################################################################################################################################################################| Time: 0:00:01
Loaded 3486 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
ERROR: Nothing PROVIDES 'virtual/kernel'
linux-dummy PROVIDES virtual/kernel but was skipped: PREFERRED_PROVIDER_virtual/kernel set to linux-yocto-tiny, not linux-dummy
linux-yocto-dev PROVIDES virtual/kernel but was skipped: incompatible with machine jetson-nano-qspi-sd (not in COMPATIBLE_MACHINE)
linux-yocto-rt PROVIDES virtual/kernel but was skipped: incompatible with machine jetson-nano-qspi-sd (not in COMPATIBLE_MACHINE)
linux-yocto-tiny PROVIDES virtual/kernel but was skipped: incompatible with machine jetson-nano-qspi-sd (not in COMPATIBLE_MACHINE)
linux-yocto PROVIDES virtual/kernel but was skipped: incompatible with machine jetson-nano-qspi-sd (not in COMPATIBLE_MACHINE)
linux-tegra PROVIDES virtual/kernel but was skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-tegra to enable it
ERROR: Required build target 'core-image-minimal' has no buildable providers.
Missing or unbuildable dependency chain was: ['core-image-minimal', 'virtual/kernel']

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

madisongh commented 4 years ago

Where did you set PREFERRED_PROVIDER_virtual/kernel? If it was in your local.conf file, then that setting is getting overridden by the setting in poky-tiny.conf, since that is read later. You could try

PREFERRED_PROVIDER_virtual/kernel_forcevariable = "linux-tegra"

which should force-override the later setting.

However, the binary-only components from NVIDIA's BSP are linked against glibc, so it's unlikely you'll be able to build any sort of useful image using the poky-tiny distro, which sets musl for the C runtime.

galvesribeiro commented 4 years ago

Ah ok, nevermind then.

What about the missing container runtime libraries?

Get Outlook for iOShttps://aka.ms/o0ukef

From: Matt Madison notifications@github.com Sent: Sunday, August 30, 2020 8:34:46 AM To: OE4T/meta-tegra meta-tegra@noreply.github.com Cc: Gutemberg Ribeiro gutemberg@outlook.com; Mention mention@noreply.github.com Subject: Re: [OE4T/meta-tegra] Fail to build image (#411)

Where did you set PREFERRED_PROVIDER_virtual/kernel? If it was in your local.conf file, then that setting is getting overridden by the setting in poky-tiny.conf, since that is read later. You could try

PREFERRED_PROVIDER_virtual/kernel_forcevariable = "linux-tegra"

which should force-override the later setting.

However, the binary-only components from NVIDIA's BSP are linked against glibc, so it's unlikely you'll be able to build any sort of useful image using the poky-tiny distro, which sets musl for the C runtime.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOE4T%2Fmeta-tegra%2Fissues%2F411%23issuecomment-683409621&data=02%7C01%7C%7C6f2ca2539e4344e486a908d84cd8b2cd%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637343840871903408&sdata=iUQnEqxUGK7omhThVPcAzmK50RCxVwDIaF8FYIYOq9o%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABD64OEJUDVL5YIJDL6QKADSDI2NNANCNFSM4QMPW4KA&data=02%7C01%7C%7C6f2ca2539e4344e486a908d84cd8b2cd%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637343840871903408&sdata=1tGCI9IOxHxBHwSDq0NX7HX%2BI5SwdSrH8fKYovxa0po%3D&reserved=0.

madisongh commented 4 years ago

What about the missing container runtime libraries?

Make sure you have set up your layers and configuration as described on this wiki page.

galvesribeiro commented 4 years ago

Yeah I have those layers. Docker is installed and fully functional. The problem is that it doesn't have nvidia container runtime loaded.

To run any containers, add nvidia-docker

Add that where?

madisongh commented 4 years ago

Add that package to your image. For example

CORE_IMAGE_EXTRA_INSTALL_append = " nvidia-docker"

in your local.conf.

galvesribeiro commented 4 years ago

Okey, progress:

root@jetson-nano-qspi-sd:~# dpkg --get-selections | grep nvidia
libnvidia-container-tools           install
nvidia-container-runtime            install
nvidia-container-toolkit            install
nvidia-docker                   install
root@jetson-nano-qspi-sd:~# docker info | grep nvidia
 Runtimes: nvidia runc

However, there still missing pieces there as it fail to start the container with this:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:413: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled

If I try to debug looks like the driver fail to load after the device boots up:

root@jetson-nano-qspi-sd:~# nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0101 01:00:49.487012 2391 nvc.c:281] initializing library context (version=0.9.0+beta1, build=09df17a2424704c1e4af310d3ce51fe475ba292a)
I0101 01:00:49.487119 2391 nvc.c:255] using root /
I0101 01:00:49.487148 2391 nvc.c:256] using ldcache /etc/ld.so.cache
I0101 01:00:49.487167 2391 nvc.c:257] using unprivileged user 65534:65534
I0101 01:00:49.487504 2392 driver.c:134] starting driver service
E0101 01:00:49.487943 2392 driver.c:196] could not start driver service: load library failed: libcuda.so.1: cannot open shared object file: no such file or directory
I0101 01:00:49.488229 2391 driver.c:229] driver service terminated successfully
nvidia-container-cli: initialization error: driver error: failed to process request

And looks like the CUDA driver libs are in fact not loaded:

root@jetson-nano-qspi-sd:~# ldconfig -p | grep nvidia
    libnvidia-container.so.0 (libc6,AArch64, OS ABI: Linux 3.10.0) => /usr/lib/libnvidia-container.so.0
root@jetson-nano-qspi-sd:~# ldconfig -p | grep cuda  
root@jetson-nano-qspi-sd:~#

Am I missing another config parameter to enable this? Thanks!

madisongh commented 4 years ago

Try adding cuda-libraries to your image. Depending on which container you're trying to load, you may need to include other packages as well.

galvesribeiro commented 4 years ago

I was trying to run just the base image from nvidia...

How I know which packages your layer offer?

I need to make sure Mesa with DRI from nvidia is also available...

Get Outlook for iOShttps://aka.ms/o0ukef

From: Matt Madison notifications@github.com Sent: Monday, August 31, 2020 7:26:47 AM To: OE4T/meta-tegra meta-tegra@noreply.github.com Cc: Gutemberg Ribeiro gutemberg@outlook.com; Mention mention@noreply.github.com Subject: Re: [OE4T/meta-tegra] Fail to build image (#411)

Try adding cuda-libraries to your image. Depending on which container you're trying to load, you may need to include other packages as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOE4T%2Fmeta-tegra%2Fissues%2F411%23issuecomment-683696847&data=02%7C01%7C%7C231010bad256452560d108d84d985db6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344664077608222&sdata=lTOvqV8DD7Jm9B6NlDXWIsHuc8oyEUk0D7BhOgGo5k4%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABD64OEXVYAPK5Q6TMGEDSDSDN3GPANCNFSM4QMPW4KA&data=02%7C01%7C%7C231010bad256452560d108d84d985db6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637344664077608222&sdata=KIYiW2iYH9YpNWSJ2mwu0IVL9SCdpt3ewseVeBgCing%3D&reserved=0.

madisongh commented 4 years ago

I was trying to run just the base image from nvidia...

Unlike the stock L4T/JetPack from NVIDIA, with Yocto-based builds you have more control over what you include in the image.

Unfortunately, NVIDIA's containers assume you're using NVIDIA's distro, so you generally need to pull in more than just a minimal set of packages.

How I know which packages your layer offer?

Just look through the recipes.

I need to make sure Mesa with DRI from nvidia is also available...

I'm not sure what you mean by this, as NVIDIA does not support DRI and does not use mesa for OpenGL. There are recipes in the layer to include their OpenGL and libdrm support, however.

galvesribeiro commented 4 years ago

Unlike the stock L4T/JetPack from NVIDIA, with Yocto-based builds you have more control over what you include in the image.

I meant the container base image.

Just look through the recipes.

I've tried to but for example, I can't find any references to cuda-libraries on it, so I couldn't guess that was the right name from the recipes.

Now we have this:

root@jetson-nano-qspi-sd:~# nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0831 16:13:02.109758 3381 nvc.c:281] initializing library context (version=0.9.0+beta1, build=09df17a2424704c1e4af310d3ce51fe475ba292a)
I0831 16:13:02.109858 3381 nvc.c:255] using root /
I0831 16:13:02.109885 3381 nvc.c:256] using ldcache /etc/ld.so.cache
I0831 16:13:02.109904 3381 nvc.c:257] using unprivileged user 65534:65534
I0831 16:13:02.110190 3382 driver.c:134] starting driver service
I0831 16:13:02.122954 3381 nvc_info.c:585] requesting driver information with ''
I0831 16:13:02.123370 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-tls.so.32.4.3
I0831 16:13:02.123480 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-ptxjitcompiler.so.32.4.3
I0831 16:13:02.123583 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-glsi.so.32.4.3
I0831 16:13:02.123653 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-glcore.so.32.4.3
I0831 16:13:02.123736 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-fatbinaryloader.so.32.4.3
I0831 16:13:02.123805 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-eglcore.so.32.4.3
I0831 16:13:02.123969 3381 nvc_info.c:154] selecting /usr/lib/libcuda.so.1.1
I0831 16:13:02.124087 3381 nvc_info.c:154] selecting /usr/lib/libGLX_nvidia.so.0
I0831 16:13:02.124152 3381 nvc_info.c:154] selecting /usr/lib/libGLESv2_nvidia.so.2
I0831 16:13:02.124216 3381 nvc_info.c:154] selecting /usr/lib/libGLESv1_CM_nvidia.so.1
I0831 16:13:02.124280 3381 nvc_info.c:154] selecting /usr/lib/libEGL_nvidia.so.0
W0831 16:13:02.124320 3381 nvc_info.c:305] missing library libnvidia-ml.so
W0831 16:13:02.124338 3381 nvc_info.c:305] missing library libnvidia-cfg.so
W0831 16:13:02.124350 3381 nvc_info.c:305] missing library libnvidia-opencl.so
W0831 16:13:02.124361 3381 nvc_info.c:305] missing library libnvidia-compiler.so
W0831 16:13:02.124372 3381 nvc_info.c:305] missing library libvdpau_nvidia.so
W0831 16:13:02.124386 3381 nvc_info.c:305] missing library libnvidia-encode.so
W0831 16:13:02.124406 3381 nvc_info.c:305] missing library libnvcuvid.so
W0831 16:13:02.124421 3381 nvc_info.c:305] missing library libnvidia-fbc.so
W0831 16:13:02.124433 3381 nvc_info.c:305] missing library libnvidia-ifr.so
W0831 16:13:02.124445 3381 nvc_info.c:309] missing compat32 library libnvidia-ml.so
W0831 16:13:02.124459 3381 nvc_info.c:309] missing compat32 library libnvidia-cfg.so
W0831 16:13:02.124475 3381 nvc_info.c:309] missing compat32 library libcuda.so
W0831 16:13:02.124494 3381 nvc_info.c:309] missing compat32 library libnvidia-opencl.so
W0831 16:13:02.124514 3381 nvc_info.c:309] missing compat32 library libnvidia-ptxjitcompiler.so
W0831 16:13:02.124534 3381 nvc_info.c:309] missing compat32 library libnvidia-fatbinaryloader.so
W0831 16:13:02.124553 3381 nvc_info.c:309] missing compat32 library libnvidia-compiler.so
W0831 16:13:02.124571 3381 nvc_info.c:309] missing compat32 library libvdpau_nvidia.so
W0831 16:13:02.124589 3381 nvc_info.c:309] missing compat32 library libnvidia-encode.so
W0831 16:13:02.124608 3381 nvc_info.c:309] missing compat32 library libnvcuvid.so
W0831 16:13:02.124627 3381 nvc_info.c:309] missing compat32 library libnvidia-eglcore.so
W0831 16:13:02.124646 3381 nvc_info.c:309] missing compat32 library libnvidia-glcore.so
W0831 16:13:02.124665 3381 nvc_info.c:309] missing compat32 library libnvidia-tls.so
W0831 16:13:02.124685 3381 nvc_info.c:309] missing compat32 library libnvidia-glsi.so
W0831 16:13:02.124703 3381 nvc_info.c:309] missing compat32 library libnvidia-fbc.so
W0831 16:13:02.124721 3381 nvc_info.c:309] missing compat32 library libnvidia-ifr.so
W0831 16:13:02.124740 3381 nvc_info.c:309] missing compat32 library libGLX_nvidia.so
W0831 16:13:02.124760 3381 nvc_info.c:309] missing compat32 library libEGL_nvidia.so
W0831 16:13:02.124782 3381 nvc_info.c:309] missing compat32 library libGLESv2_nvidia.so
W0831 16:13:02.124805 3381 nvc_info.c:309] missing compat32 library libGLESv1_CM_nvidia.so
W0831 16:13:02.125673 3381 nvc_info.c:331] missing binary nvidia-smi
W0831 16:13:02.125699 3381 nvc_info.c:331] missing binary nvidia-debugdump
W0831 16:13:02.125721 3381 nvc_info.c:331] missing binary nvidia-persistenced
W0831 16:13:02.125742 3381 nvc_info.c:331] missing binary nvidia-cuda-mps-control
W0831 16:13:02.125759 3381 nvc_info.c:331] missing binary nvidia-cuda-mps-server
W0831 16:13:02.125825 3381 nvc_info.c:280] missing ipc /var/run/nvidia-persistenced/socket
W0831 16:13:02.125920 3381 nvc_info.c:280] missing ipc /tmp/nvidia-mps
I0831 16:13:02.126471 3381 nvc_info.c:154] selecting /etc/vulkan/icd.d/nvidia_icd.json
I0831 16:13:02.126569 3381 nvc_info.c:154] selecting /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0
I0831 16:13:02.126639 3381 nvc_info.c:154] selecting /usr/lib/libEGL_nvidia.so.0
I0831 16:13:02.126709 3381 nvc_info.c:154] selecting /usr/lib/libGLESv1_CM_nvidia.so.1
I0831 16:13:02.126777 3381 nvc_info.c:154] selecting /usr/lib/libGLESv2_nvidia.so.2
I0831 16:13:02.126837 3381 nvc_info.c:154] selecting /usr/lib/libGLX_nvidia.so.0
I0831 16:13:02.126902 3381 nvc_info.c:154] selecting /usr/lib/libcuda.so.1.1
I0831 16:13:02.126970 3381 nvc_info.c:154] selecting /usr/lib/libnvapputil.so
I0831 16:13:02.127036 3381 nvc_info.c:375] missing library /usr/lib/libnvargus.so
I0831 16:13:02.127095 3381 nvc_info.c:375] missing library /usr/lib/libnvargus_socketclient.so
I0831 16:13:02.127152 3381 nvc_info.c:375] missing library /usr/lib/libnvargus_socketserver.so
I0831 16:13:02.127218 3381 nvc_info.c:154] selecting /usr/lib/libnvavp.so
I0831 16:13:02.127290 3381 nvc_info.c:154] selecting /usr/lib/libnvbuf_fdmap.so.1.0.0
I0831 16:13:02.127356 3381 nvc_info.c:154] selecting /usr/lib/libnvbuf_utils.so.1.0.0
I0831 16:13:02.127426 3381 nvc_info.c:154] selecting /usr/lib/libnvbufsurface.so.1.0.0
I0831 16:13:02.127496 3381 nvc_info.c:154] selecting /usr/lib/libnvbufsurftransform.so.1.0.0
I0831 16:13:02.127585 3381 nvc_info.c:154] selecting /usr/lib/libnvcam_imageencoder.so
I0831 16:13:02.127653 3381 nvc_info.c:154] selecting /usr/lib/libnvcameratools.so
I0831 16:13:02.127721 3381 nvc_info.c:154] selecting /usr/lib/libnvcamerautils.so
I0831 16:13:02.127790 3381 nvc_info.c:154] selecting /usr/lib/libnvcamlog.so
I0831 16:13:02.127858 3381 nvc_info.c:154] selecting /usr/lib/libnvcamv4l2.so
I0831 16:13:02.127924 3381 nvc_info.c:154] selecting /usr/lib/libnvcolorutil.so
I0831 16:13:02.127990 3381 nvc_info.c:154] selecting /usr/lib/libnvdc.so
I0831 16:13:02.128059 3381 nvc_info.c:154] selecting /usr/lib/libnvddk_2d_v2.so
I0831 16:13:02.128128 3381 nvc_info.c:154] selecting /usr/lib/libnvddk_vic.so
I0831 16:13:02.128198 3381 nvc_info.c:154] selecting /usr/lib/libnvdsbufferpool.so.1.0.0
I0831 16:13:02.128267 3381 nvc_info.c:154] selecting /usr/lib/libnveglstream_camconsumer.so
I0831 16:13:02.128335 3381 nvc_info.c:154] selecting /usr/lib/libnveglstreamproducer.so
I0831 16:13:02.128404 3381 nvc_info.c:154] selecting /usr/lib/libnveventlib.so
I0831 16:13:02.128470 3381 nvc_info.c:154] selecting /usr/lib/libnvexif.so
I0831 16:13:02.128538 3381 nvc_info.c:154] selecting /usr/lib/libnvfnet.so
I0831 16:13:02.128606 3381 nvc_info.c:154] selecting /usr/lib/libnvfnetstoredefog.so
I0831 16:13:02.128673 3381 nvc_info.c:154] selecting /usr/lib/libnvfnetstorehdfx.so
I0831 16:13:02.128739 3381 nvc_info.c:154] selecting /usr/lib/libnvgbm.so
I0831 16:13:02.128810 3381 nvc_info.c:154] selecting /usr/lib/libnvid_mapper.so.1.0.0
I0831 16:13:02.128877 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-eglcore.so.32.4.3
I0831 16:13:02.128946 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-fatbinaryloader.so.32.4.3
I0831 16:13:02.129018 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-glcore.so.32.4.3
I0831 16:13:02.129084 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-glsi.so.32.4.3
I0831 16:13:02.129152 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-glvkspirv.so.32.4.3
I0831 16:13:02.129224 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-ptxjitcompiler.so.32.4.3
I0831 16:13:02.129294 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-rmapi-tegra.so.32.4.3
I0831 16:13:02.129370 3381 nvc_info.c:154] selecting /usr/lib/libnvidia-tls.so.32.4.3
I0831 16:13:02.129439 3381 nvc_info.c:154] selecting /usr/lib/libnvimp.so
I0831 16:13:02.129506 3381 nvc_info.c:154] selecting /usr/lib/libnvjpeg.so
I0831 16:13:02.129574 3381 nvc_info.c:154] selecting /usr/lib/libnvll.so
I0831 16:13:02.129641 3381 nvc_info.c:154] selecting /usr/lib/libnvmedia.so
I0831 16:13:02.129709 3381 nvc_info.c:154] selecting /usr/lib/libnvmm.so
I0831 16:13:02.129776 3381 nvc_info.c:154] selecting /usr/lib/libnvmm_contentpipe.so
I0831 16:13:02.129844 3381 nvc_info.c:154] selecting /usr/lib/libnvmm_parser.so
I0831 16:13:02.129914 3381 nvc_info.c:154] selecting /usr/lib/libnvmm_utils.so
I0831 16:13:02.129980 3381 nvc_info.c:154] selecting /usr/lib/libnvmmlite.so
I0831 16:13:02.130048 3381 nvc_info.c:154] selecting /usr/lib/libnvmmlite_image.so
I0831 16:13:02.130117 3381 nvc_info.c:154] selecting /usr/lib/libnvmmlite_utils.so
I0831 16:13:02.130185 3381 nvc_info.c:154] selecting /usr/lib/libnvmmlite_video.so
I0831 16:13:02.130253 3381 nvc_info.c:154] selecting /usr/lib/libnvodm_imager.so
I0831 16:13:02.130320 3381 nvc_info.c:154] selecting /usr/lib/libnvofsdk.so
I0831 16:13:02.130387 3381 nvc_info.c:154] selecting /usr/lib/libnvomx.so
I0831 16:13:02.130455 3381 nvc_info.c:154] selecting /usr/lib/libnvomxilclient.so
I0831 16:13:02.130524 3381 nvc_info.c:154] selecting /usr/lib/libnvos.so
I0831 16:13:02.130581 3381 nvc_info.c:375] missing library /usr/lib/libnvosd.so
I0831 16:13:02.130645 3381 nvc_info.c:154] selecting /usr/lib/libnvparser.so
I0831 16:13:02.130710 3381 nvc_info.c:154] selecting /usr/lib/libnvphs.so
I0831 16:13:02.130778 3381 nvc_info.c:154] selecting /usr/lib/libnvrm.so
I0831 16:13:02.130845 3381 nvc_info.c:154] selecting /usr/lib/libnvrm_gpu.so
I0831 16:13:02.130912 3381 nvc_info.c:154] selecting /usr/lib/libnvrm_graphics.so
I0831 16:13:02.130983 3381 nvc_info.c:154] selecting /usr/lib/libnvscf.so
I0831 16:13:02.131048 3381 nvc_info.c:154] selecting /usr/lib/libnvtestresults.so
I0831 16:13:02.131115 3381 nvc_info.c:154] selecting /usr/lib/libnvtnr.so
I0831 16:13:02.131185 3381 nvc_info.c:154] selecting /usr/lib/libnvtracebuf.so
I0831 16:13:02.131252 3381 nvc_info.c:154] selecting /usr/lib/libnvtvmr.so
I0831 16:13:02.131318 3381 nvc_info.c:154] selecting /usr/lib/libnvvulkan-producer.so
I0831 16:13:02.131387 3381 nvc_info.c:154] selecting /usr/lib/libnvwinsys.so
I0831 16:13:02.131457 3381 nvc_info.c:154] selecting /usr/lib/libsensors.hal-client.nvs.so
I0831 16:13:02.131543 3381 nvc_info.c:154] selecting /usr/lib/libsensors.l4t.no_fusion.nvs.so
I0831 16:13:02.131613 3381 nvc_info.c:154] selecting /usr/lib/libsensors_hal.nvs.so
I0831 16:13:02.131679 3381 nvc_info.c:154] selecting /usr/lib/libtegrav4l2.so
I0831 16:13:02.131740 3381 nvc_info.c:375] missing library /usr/lib/libv4l/plugins/libv4l2_nvvidconv.so
I0831 16:13:02.131795 3381 nvc_info.c:375] missing library /usr/lib/libv4l/plugins/libv4l2_nvvideocodec.so
I0831 16:13:02.131863 3381 nvc_info.c:154] selecting /usr/lib/libv4l2_nvvidconv.so
I0831 16:13:02.131930 3381 nvc_info.c:154] selecting /usr/lib/libv4l2_nvvideocodec.so
I0831 16:13:02.132004 3381 nvc_info.c:154] selecting /usr/lib/libvulkan.so.1.2.132
I0831 16:13:02.132093 3381 nvc_info.c:154] selecting /usr/share/glvnd/egl_vendor.d/10-nvidia.json
I0831 16:13:02.132302 3381 nvc_info.c:154] selecting /lib/firmware/tegra21x_xusb_firmware
I0831 16:13:02.132546 3381 nvc_info.c:154] selecting /usr/lib/libcublas.so.10.2.2.89
I0831 16:13:02.132620 3381 nvc_info.c:154] selecting /usr/lib/libcublasLt.so.10.2.2.89
I0831 16:13:02.132689 3381 nvc_info.c:154] selecting /usr/lib/libnvblas.so.10.2.2.89
W0831 16:13:02.132720 3381 nvc_info.c:431] missing symlink /usr/lib/libcublas.so
W0831 16:13:02.132751 3381 nvc_info.c:431] missing symlink /usr/lib/libcublasLt.so
W0831 16:13:02.132783 3381 nvc_info.c:431] missing symlink /usr/lib/libnvblas.so
W0831 16:13:02.133259 3381 nvc_info.c:459] missing device /dev/nvhost-nvdec1
W0831 16:13:02.133364 3381 nvc_info.c:459] missing device /dev/nvhost-ctrl-nvdla0
W0831 16:13:02.133420 3381 nvc_info.c:459] missing device /dev/nvhost-ctrl-nvdla1
W0831 16:13:02.133473 3381 nvc_info.c:459] missing device /dev/nvhost-nvdla0
W0831 16:13:02.133526 3381 nvc_info.c:459] missing device /dev/nvhost-nvdla1
W0831 16:13:02.133578 3381 nvc_info.c:459] missing device /dev/nvidiactl
W0831 16:13:02.133751 3381 nvc_info.c:459] missing device /dev/nvhost-nvenc1
I0831 16:13:02.133917 3381 nvc_info.c:154] selecting /usr/lib/tegra/libdrm.so.2
I0831 16:13:02.134131 3381 nvc_info.c:154] selecting /etc/ld.so.conf.d/cuda-10-2.conf
I0831 16:13:02.134183 3381 nvc_info.c:642] requesting device information with opts: ''
I0831 16:13:02.134406 3381 nvc_info.c:660] listing device (null) ((null) at (null))
NVRM version:   (null)
CUDA version:   10.2

Device Index:   0
Device Minor:   0
Model:          NVIDIA Tegra X1
Brand:          (null)
GPU UUID:       (null)
Bus Location:   (null)
Architecture:   5.3
I0831 16:13:02.134512 3381 nvc.c:314] shutting down library context
I0831 16:13:02.134574 3382 driver.c:191] terminating driver service
I0831 16:13:02.135573 3381 driver.c:229] driver service terminated successfully

Way better, but there still missing libraries and the device info is not detected as expected, so I guess the driver is not 100% ☹️

madisongh commented 4 years ago

Some of the missing pass-through mappings are normal, or at least OK to ignore unless an application running in the container needs them. For instance, /dev/nvhost-dla0 is only actually present on DLA-capable platforms (i.e., Xavier family).

Some of the device information looks like it's specific to a discrete GPU setup (i.e., PC + GPU card) and isn't applicable to the Jetsons. The model and architecture look correct, at least... have you tried running an app in the container?

elPrac commented 4 years ago

How I know which packages your layer offer?

You can look what recipes meta-tegra offers with $bitbake-layers show-recipes this will list all the available recipes from all your meta-layers, i'm not sure if there is a bitbake option that lists packages from a specific layer but, as you are looking for a specific library you can go to meta-tegra and grep ^FILES_ this will give you a better idea of which recipe is providing what... e.g.

grep -rnw './' -e ^FILES_

./recipes-devtools/cuda/cuda-shared-binaries-10.0.326-1.inc:91:FILES_${PN} += "${prefix}/local/cuda-10.0/${baselib}/*${SOLIBS} ${prefix}/local/cuda-10.0/bin"
./recipes-devtools/cuda/cuda-shared-binaries-10.0.326-1.inc:92:FILES_${PN}-dev += "${prefix}/local/cuda-10.0/extras ${prefix}/local/cuda-10.0/include ${prefix}/local/cuda-10.0/${baselib}/stubs \
./recipes-devtools/cuda/cuda-shared-binaries-10.0.326-1.inc:94:FILES_${PN}-staticdev += "${prefix}/local/cuda-10.0/${baselib}/*.a"

just by looking the above i can tell that cuda-shared-binaries-${PV}.inc is providing some ${SOLIBS} i.e. so files and because it is an .inc recipe it is probably included/required by another recipe like ./meta-tegra/recipes-devtools/cuda/cuda-cublas_%.bb so it looks like using the recipe name cuda-cublas will build and install the libraries associated with this ${PN} you can build this recipe and look what will be installed in your image if you add it e.g bitbake cuda-cublas then look at ${D} of this recipe like:

Try adding cuda-libraries to your image.

I think that @madisongh is refering to these cuda-* ${PN} recipes.

I'm not an Yocto expert so maybe this is not the best way to do it but to me it worked, perhaps @madisongh or anyone else has a better input. just wanted to help!

BR, -Miguel M

galvesribeiro commented 4 years ago

Thank you for the clear explanation @elPrac !

My question was more in the sense that cuda-libraries as @madisongh mentioned, isn't anywhere defined on the files and that is why I couldn't find it. I'm not an Yocto expert either and this is our first project with it.

Our end goal is to build bare minimal image that contains only Docker and NVidia GPU support. We don't even need CUDA. Our application would just require EGL support to render themselves from within a container without have to have a X server.

The eGL sample from NVidia runs on the Host and inside a container without X perfectly on L4T as you may see here (it is a PiP window on my monitor attached to Nano over HDMI):

However, if I run on the Yocto image with meta-tegra added, both at the Host OS or in a Container, it complains that there is no libEGL.so.*...

I tried add to both IMAGE_INSTALL_append and CORE_IMAGE_EXTRA_INSTALL the mesa package, which would give us the eGL lib, but looks like it isn't there. In L4T I see the lib there so it just works...

madisongh commented 4 years ago

Part of the problem is starting with core-image-minimal -- it is a really minimal image, and includes only the barest minimum set of drivers and packages to get the system to boot.

I ran a quick test with the following in my local.conf, building core-image-minimal:

CORE_IMAGE_EXTRA_INSTALL = "l4t-graphics-demos-egldevice kernel-module-hid-logitech-hidpp kernel-module-hid-logitech-dj \
                            tegra-udrm-probeconf tegra-libraries libdrm ${MACHINE_EXTRA_RRECOMMENDS} nvidia-docker"

(The logitech modules are for my wireless USB keyboard, which you may not need.)

I could run the egldevice graphics demos (you'll find them in /usr/bin/l4t-graphics-demos/egldevice) natively. I could also fire up a container with

# docker run --net=host --runtime nvidia --rm --ipc=host -it nvcr.io/nvidia/l4t-base:r32.4.3

Then download NVIDIA's pre-built binary copies of the graphics demos into the container to run there.

galvesribeiro commented 4 years ago

OH!! Awesome! I'll try it right away!

Out of curiosity, what does ${MACHINE_EXTRA_RRECOMMENDS} resolves to?

Again, thanks for helping! :)

dwalkes commented 4 years ago

Out of curiosity, what does ${MACHINE_EXTRA_RRECOMMENDS} resolves to?

You can see how it's set if you use something like: bitbake -e core-image-minimal | grep "^MACHINE_EXTRA_RRECOMMENDS"

as discussed in threads like this one

You can read more about MACHINE_EXTRA_RRECOMMENDS at https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#var-MACHINE_EXTRA_RRECOMMENDS and your favorite search engine likely does a good job of indexing the yocto reference manual. You can substitute the URL portion between docs and ref-manual with latest to make sure you are reading the latest documents.

madisongh commented 4 years ago

Looks like the original issue was resolved, so am closing this now.

OE4T / meta-tegra

Fail to build image #411