Build Fails 1/4/2023

[jlpoolen]

I kept getting failures trying Jenkins and concluded I must have some setting in Jenkins different from the GenPi64's project. I requested access to the project's server so I could compare the configuration settings of the OSU jenkins vs. mine. While I await a response, I decided to try building the project without Jenkins to anticipate any problems and see if i could fix them outside of a Jenkins environment.

On January 4, 2023, at around 8:18 AM, I pulled a copy of my fork which was updated yesterday and confirmed to be in sync this morning and the build failed. Log at https://pastebin.com/Rt0ATBN9

[jlpoolen]

I confirmed now, at 8:43 AM, that my fork is still in sync with the project.

[samip5]

On January 4, 2023, at around 8:18 AM, I pulled a copy of my fork which was updated yesterday and confirmed to be in sync this morning and the build failed. Log at https://pastebin.com/Rt0ATBN9

You seem to be hitting a snag with qemu as Exec format error entails wrong architecture binary is trying to be ran.

[jlpoolen]

I'm testing distcc and, accordingly, modified my make.conf. Here's what is the current configuration:

jenk /home/jlpoole/local/Build.Dist # date; cat /etc/portage/make.conf
Wed Jan  4 09:04:20 AM PST 2023
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
COMMON_FLAGS="-O2 -pipe"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"

# NOTE: This stage was built with the bindist Use flag enabled
PORTDIR="/var/db/repos/gentoo"
DISTDIR="/var/cache/distfiles"
PKGDIR="/var/cache/binpkgs"

# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C
#
# 3/10/22: below are jlpoole
# 1/3/2022: rem'd given distcc below
#MAKEOPTS="-j9"

#
# 3/13/22 per Mike Jones Discord chat
#
QEMU_USER_TARGETS="aarch64 x86_64"
QEMU_SOFTMMU_TARGETS="aarch64 x86_64"

#
# 1/4/2023 jlpoole: for distcc
# Setting for MAKEOPTS calculate per distcc wiki suggestions
# lscpu
#   VM1: 2
#   jenk: 8
# 2+8=10, 10*2 = 20 + 1 = 21

MAKEOPTS="-j21 -l8"
FEATURES="distcc"
jenk /home/jlpoole/local/Build.Dist #

[jlpoolen]

Discussion on the project Discord channel suggests pychroot may be a source of the problem. Here's what I have installed:

[I] dev-python/pychroot
     Available versions:  (~)0.10.4^t **9999*l^t {test PYTHON_TARGETS="python3_8 python3_9 python3_10"}
     Installed versions:  0.10.4^t(09:02:10 PM 01/01/2023)(-test PYTHON_TARGETS="python3_10 -python3_8 -python3_9")
     Homepage:            https://github.com/pkgcore/pychroot
     Description:         a python library and cli tool that simplify chroot handling

There was an investigation into an early failure at environment setup where "arm" registration is made for qemu, turns out the "arm" registration was not necessary and the code line rem'd out and a build attempted thereafter. Result was failure: https://pastebin.com/C1WnM6g1

[jonesmz]

I recommend that you stop attempting to run build.dist's build.sh script until you've completely solved your qemu issue.

Just run the equivelent of the following commands, adjusted for your local environment (and any syntax errors I may have written), after a fresh reboot

echo ":aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\xB7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:F" > /proc/sys/fs/binfmt_misc/register

wget https://mirror.bytemark.co.uk/gentoo/releases/arm64/autobuilds/current-stage3-arm64-openrc/stage3-arm64-openrc-20230101T231658Z.tar.xz
tar xf stage3-arm64-openrc-*.tar.xz
cat <EOF >stage3-arm64-openrc/chroot.sh
#!/bin/bash
##
# Setup the chroot as normal.
##
mount -t proc   proc    ./proc
mount --rbind   /sys    ./sys
mount --rbind   /dev    ./dev

chroot ./ /bin/bash

umount -l ./proc
umount -l ./sys
umount -l ./dev
EOF
chmod +x stage3-arm64-openrc/chroot.sh
cd stage3-arm64-openrc
./chroot.sh

and then see whether or not qemu fails to be engaged as your aarch64 code executor.

if it works, you'll be in the chroot. If it fails, you won't be, and you'll have some kind of error message.

But i really do not recommend continually running the build.sh script for every minor tweak to your environment, because in many cases myself or @samip5 are only telling you the proximate cause to the immediate problem, and not necessarily claiming that the whole script will work with just the one change that we suggest. Better to verify the basics first before attempting to execute everything.

[jlpoolen]

Thank you, Mike.

qemu looks to be working. See log executing your instructions + running nano + compiling a Hello World file and running same in chroot and then in native x86_64 where is failed as expected.

https://pastebin.com/yTupvMpD

What is your suggestion I do next?

[jonesmz]

alright, that's great.

So now that you've got a working aarch64 emulator that works without having to be copied into the chroot environment (the F flag in the binfmt text), the next step is to try re-running the build.dist build.sh script and see what the next thing that breaks is.

[jlpoolen]

I have determined that the problem concerns the command to be executed in the new environment:

 eselect locale set en_US.utf8

To learn what's going on, and to verify that chroot.py is doing what it is supposed to be doing, I created a similar file to the one that parsers/rawcommand/rawcommand [a Bash script] creates and tries to execute for the task "locale". My file is suitably named "jlp_was_here" and was placed in the chroot environment at build/GenPi64OpenRC/chroot/jlp_was_here.

jenk /home/jlpoole/local/Build.Dist # cat -n build/GenPi64OpenRC/chroot/jlp_was_here
     1  #!/usr/bin/env bash
     2  set -evx
     3  source /etc/profile
     4  #eselect locale set en_US.utf8
     5  my_current_dir=`pwd`
     6  echo  my_current_dir = $my_current_dir
     7  ls -lat /
jenk /home/jlpoole/local/Build.Dist #

To simulate the build script, I created another test script:

jenk /home/jlpoole/local/Build.Dist # cat -n test_chroot.sh
     1  # to test pychroot.sh
     2  source ./env.sh
     3  #
     4  #
     5  #
     6  scripts/chroot.py /jlp_was_here
jenk /home/jlpoole/local/Build.Dist #

The results of the session run were successful (note: the eselect command was rem'd out):

jenk /home/jlpoole/local/Build.Dist # bash test_chroot.sh
Skipping arm registration for jlpoole build only
source /etc/profile
+ source /etc/profile
# /etc/profile: login shell setup
#
# That this file is used by any Bourne-shell derivative to setup the
# environment for login shells.
#

# Load environment settings from profile.env, which is created by
# env-update from the files in /etc/env.d
if [ -e /etc/profile.env ] ; then
        . /etc/profile.env
fi
++ '[' -e /etc/profile.env ']'
++ . /etc/profile.env
# THIS FILE IS AUTOMATICALLY GENERATED BY env-update.
# DO NOT EDIT THIS FILE. CHANGES TO STARTUP PROFILES
# GO INTO /etc/profile NOT /etc/profile.env

export CONFIG_PROTECT='/usr/share/gnupg/qualified.txt'
+++ export CONFIG_PROTECT=/usr/share/gnupg/qualified.txt
+++ CONFIG_PROTECT=/usr/share/gnupg/qualified.txt
export CONFIG_PROTECT_MASK='/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf'
+++ export 'CONFIG_PROTECT_MASK=/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf'
+++ CONFIG_PROTECT_MASK='/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf'
export GCC_SPECS=''
+++ export GCC_SPECS=
+++ GCC_SPECS=
export INFOPATH='/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/info:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/info:/usr/share/info'
+++ export INFOPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/info:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/info:/usr/share/info
+++ INFOPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/info:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/info:/usr/share/info
export LANG='C.UTF8'
+++ export LANG=C.UTF8
+++ LANG=C.UTF8
export LESS='-R -M --shift 5'
+++ export 'LESS=-R -M --shift 5'
+++ LESS='-R -M --shift 5'
export LESSOPEN='|lesspipe %s'
+++ export 'LESSOPEN=|lesspipe %s'
+++ LESSOPEN='|lesspipe %s'
export LEX='flex'
+++ export LEX=flex
+++ LEX=flex
export MANPAGER='manpager'
+++ export MANPAGER=manpager
+++ MANPAGER=manpager
export MANPATH='/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/man:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/man:/usr/local/share/man:/usr/share/man'
+++ export MANPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/man:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/man:/usr/local/share/man:/usr/share/man
+++ MANPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/man:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/man:/usr/local/share/man:/usr/share/man
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin'
+++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
+++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
export ROOTPATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin'
+++ export ROOTPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
+++ ROOTPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin

# You should override these in your ~/.bashrc (or equivalent) for per-user
# settings.  For system defaults, you can add a new file in /etc/profile.d/.
export EDITOR=${EDITOR:-/bin/nano}
++ export EDITOR=/bin/nano
++ EDITOR=/bin/nano
export PAGER=${PAGER:-/usr/bin/less}
++ export PAGER=/usr/bin/less
++ PAGER=/usr/bin/less

# 077 would be more secure, but 022 is generally quite realistic
umask 022
++ umask 022

# Set up PATH depending on whether we're root or a normal user.
# There's no real reason to exclude sbin paths from the normal user,
# but it can make tab-completion easier when they aren't in the
# user's PATH to pollute the executable namespace.
#
# It is intentional in the following line to use || instead of -o.
# This way the evaluation can be short-circuited and calling whoami is
# avoided.
if [ "$EUID" = "0" ] || [ "$USER" = "root" ] ; then
        PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:${ROOTPATH}"
else
        PATH="/usr/local/bin:/usr/bin:/bin${PATH:+:}${PATH-}"
fi
++ '[' 0 = 0 ']'
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
export PATH
++ export PATH
unset ROOTPATH
++ unset ROOTPATH

if [ -n "${BASH_VERSION-}" ] ; then
        # Newer bash ebuilds include /etc/bash/bashrc which will setup PS1
        # including color.  We leave out color here because not all
        # terminals support it.
        if [ -f /etc/bash/bashrc ] ; then
                # Bash login shells run only /etc/profile
                # Bash non-login shells run only /etc/bash/bashrc
                # Since we want to run /etc/bash/bashrc regardless, we source it
                # from here.  It is unfortunate that there is no way to do
                # this *after* the user's .bash_profile runs (without putting
                # it in the user's dot-files), but it shouldn't make any
                # difference.
                . /etc/bash/bashrc
        else
                PS1='\u@\h \w \$ '
        fi
else
        # Setup a bland default prompt.  Since this prompt should be useable
        # on color and non-color terminals, as well as shells that don't
        # understand sequences such as \h, don't put anything special in it.
        PS1="${USER:-$(whoami 2>/dev/null)}@$(uname -n 2>/dev/null) \$ "
fi
++ '[' -n '5.1.16(1)-release' ']'
++ '[' -f /etc/bash/bashrc ']'
++ . /etc/bash/bashrc
# /etc/bash/bashrc
#
# This file is sourced by all *interactive* bash shells on startup,
# including some apparently interactive shells such as scp and rcp
# that can't tolerate any output.  So make sure this doesn't display
# anything or bad things will happen !

# Test for an interactive shell.  There is no need to set anything
# past this point for scp and rcp, and it's important to refrain from
# outputting anything in those cases.
if [[ $- != *i* ]] ; then
        # Shell is non-interactive.  Be done now!
        return
fi
+++ [[ ehvxB != *i* ]]
+++ return

for sh in /etc/profile.d/*.sh ; do
        [ -r "$sh" ] && . "$sh"
done
++ for sh in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/gawk.sh ']'
++ . /etc/profile.d/gawk.sh
gawkpath_default () {
        unset AWKPATH
        export AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`
}

gawkpath_prepend () {
        [ -z "$AWKPATH" ] && AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`
        export AWKPATH="$*:$AWKPATH"
}

gawkpath_append () {
        [ -z "$AWKPATH" ] && AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`
        export AWKPATH="$AWKPATH:$*"
}

gawklibpath_default () {
        unset AWKLIBPATH
        export AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`
}

gawklibpath_prepend () {
        [ -z "$AWKLIBPATH" ] && \
                AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`
        export AWKLIBPATH="$*:$AWKLIBPATH"
}

gawklibpath_append () {
        [ -z "$AWKLIBPATH" ] && \
                AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`
        export AWKLIBPATH="$AWKLIBPATH:$*"
}
unset sh
++ unset sh
#eselect locale set en_US.utf8
my_current_dir=`pwd`
++ pwd
+ my_current_dir=/
echo  my_current_dir = $my_current_dir
+ echo my_current_dir = /
my_current_dir = /
ls -lat /
+ ls -lat /
total 88
drwxr-xr-x  31 root root  4096 Jan  6 05:17 etc
dr-xr-xr-x 195 root root     0 Jan  6 05:17 proc
-rwxr-xr-x   1 root root   150 Jan  6 05:17 jlp_was_here
drwxr-xr-x  20 root root  4096 Jan  6 05:04 .
drwxr-xr-x  20 root root  4096 Jan  6 05:04 ..
drwxr-xr-x  15 root root  3960 Jan  6 01:03 dev
dr-xr-xr-x  13 root root     0 Jan  6 01:03 sys
drwxrwxrwt   2 root root  4096 Jan  5 18:23 tmp
-rwxr-xr-x   1 root root   564 Jan  5 18:04 em-config
-rw-r--r--   1 root root 10556 Jan  5 18:04 em-config.json
drwxr-xr-x   2 root root  4096 Jan  2 00:46 sbin
drwxr-xr-x   9 root root  4096 Jan  2 00:46 lib
drwxr-xr-x   2 root root  4096 Jan  2 00:44 bin
drwxr-xr-x   4 root root  4096 Jan  2 00:44 lib64
drwxr-xr-x  12 root root  4096 Jan  2 00:16 usr
drwxr-xr-x   9 root root  4096 Jan  2 00:11 var
drwx------   2 root root  4096 Jan  2 00:11 root
drwxr-xr-x   2 root root  4096 Jan  2 00:11 opt
drwxr-xr-x   2 root root  4096 Jan  2 00:11 mnt
drwxr-xr-x   2 root root  4096 Jan  2 00:11 media
drwxr-xr-x   2 root root  4096 Jan  2 00:11 home
drwxr-xr-x   2 root root  4096 Jan  2 00:11 boot
drwxr-xr-x   2 root root  4096 Jan  2 00:11 run
jenk /home/jlpoole/local/Build.Dist #

I'm not at the finish line, but felt what I have done here might be instructive to others who might encounter the issues I did and want a way to test. Seeing all the output of chroot.py is fascinating and ought to provide further insight into what is going on. I think the next test would be to compile and then run a Hello World program via a script similar to test_chroot.sh, that ought to help answer questions about whether qemu is working or not.

[jlpoolen]

Qemu appears to be working, too.

I created three files, similar to my previous experiment, to test invoking qemu.

jenk /home/jlpoole/local/Build.Dist # cat -n ./test_compile_chroot.sh                1  # to test pychroot.sh
     2  source ./env.sh
     3  #
     4  #
     5  #
     6  scripts/chroot.py /jlp_compile_test
jenk /home/jlpoole/local/Build.Dist # cat -n build/GenPi64OpenRC/chroot/jlp_compile_test
     1  #!/usr/bin/env bash
     2  set -evx
     3  source /etc/profile
     4  gcc -o main /main.c
     5  chmod +x /main.c
     6  echo About to execute main:
     7  my_output=`./main`
     8  echo result :
     9  echo =======================
    10  echo $my_output
    11  echo ========================
    12  mydate =`date`
    13  echo Test completed $mydate
jenk /home/jlpoole/local/Build.Dist # cat -n build/GenPi64OpenRC/chroot/main.c       1  #include <stdio.h>
     2
     3  int main() {
     4      printf("Hello compiled in a chroot environment!\n");
     5      return 0;
     6  }
     7
jenk /home/jlpoole/local/Build.Dist # date; ./test_compile_chroot.sh
Thu Jan  5 09:58:46 PM PST 2023
Skipping arm registration for jlpoole build only
source /etc/profile
+ source /etc/profile
# /etc/profile: login shell setup
#
# That this file is used by any Bourne-shell derivative to setup the
# environment for login shells.
#

# Load environment settings from profile.env, which is created by
# env-update from the files in /etc/env.d
if [ -e /etc/profile.env ] ; then
        . /etc/profile.env
fi
++ '[' -e /etc/profile.env ']'
++ . /etc/profile.env
# THIS FILE IS AUTOMATICALLY GENERATED BY env-update.
# DO NOT EDIT THIS FILE. CHANGES TO STARTUP PROFILES
# GO INTO /etc/profile NOT /etc/profile.env

export CONFIG_PROTECT='/usr/share/gnupg/qualified.txt'
+++ export CONFIG_PROTECT=/usr/share/gnupg/qualified.txt
+++ CONFIG_PROTECT=/usr/share/gnupg/qualified.txt
export CONFIG_PROTECT_MASK='/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf'
+++ export 'CONFIG_PROTECT_MASK=/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf'
+++ CONFIG_PROTECT_MASK='/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf'
export GCC_SPECS=''
+++ export GCC_SPECS=
+++ GCC_SPECS=
export INFOPATH='/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/info:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/info:/usr/share/info'
+++ export INFOPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/info:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/info:/usr/share/info
+++ INFOPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/info:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/info:/usr/share/info
export LANG='C.UTF8'
+++ export LANG=C.UTF8
+++ LANG=C.UTF8
export LESS='-R -M --shift 5'
+++ export 'LESS=-R -M --shift 5'
+++ LESS='-R -M --shift 5'
export LESSOPEN='|lesspipe %s'
+++ export 'LESSOPEN=|lesspipe %s'
+++ LESSOPEN='|lesspipe %s'
export LEX='flex'
+++ export LEX=flex
+++ LEX=flex
export MANPAGER='manpager'
+++ export MANPAGER=manpager
+++ MANPAGER=manpager
export MANPATH='/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/man:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/man:/usr/local/share/man:/usr/share/man'
+++ export MANPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/man:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/man:/usr/local/share/man:/usr/share/man
+++ MANPATH=/usr/share/gcc-data/aarch64-unknown-linux-gnu/11/man:/usr/share/binutils-data/aarch64-unknown-linux-gnu/2.39/man:/usr/local/share/man:/usr/share/man
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin'
+++ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
+++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
export ROOTPATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin'
+++ export ROOTPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
+++ ROOTPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin

# You should override these in your ~/.bashrc (or equivalent) for per-user
# settings.  For system defaults, you can add a new file in /etc/profile.d/.
export EDITOR=${EDITOR:-/bin/nano}
++ export EDITOR=/bin/nano
++ EDITOR=/bin/nano
export PAGER=${PAGER:-/usr/bin/less}
++ export PAGER=/usr/bin/less
++ PAGER=/usr/bin/less

# 077 would be more secure, but 022 is generally quite realistic
umask 022
++ umask 022

# Set up PATH depending on whether we're root or a normal user.
# There's no real reason to exclude sbin paths from the normal user,
# but it can make tab-completion easier when they aren't in the
# user's PATH to pollute the executable namespace.
#
# It is intentional in the following line to use || instead of -o.
# This way the evaluation can be short-circuited and calling whoami is
# avoided.
if [ "$EUID" = "0" ] || [ "$USER" = "root" ] ; then
        PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:${ROOTPATH}"
else
        PATH="/usr/local/bin:/usr/bin:/bin${PATH:+:}${PATH-}"
fi
++ '[' 0 = 0 ']'
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
export PATH
++ export PATH
unset ROOTPATH
++ unset ROOTPATH

if [ -n "${BASH_VERSION-}" ] ; then
        # Newer bash ebuilds include /etc/bash/bashrc which will setup PS1
        # including color.  We leave out color here because not all
        # terminals support it.
        if [ -f /etc/bash/bashrc ] ; then
                # Bash login shells run only /etc/profile
                # Bash non-login shells run only /etc/bash/bashrc
                # Since we want to run /etc/bash/bashrc regardless, we source it
                # from here.  It is unfortunate that there is no way to do
                # this *after* the user's .bash_profile runs (without putting
                # it in the user's dot-files), but it shouldn't make any
                # difference.
                . /etc/bash/bashrc
        else
                PS1='\u@\h \w \$ '
        fi
else
        # Setup a bland default prompt.  Since this prompt should be useable
        # on color and non-color terminals, as well as shells that don't
        # understand sequences such as \h, don't put anything special in it.
        PS1="${USER:-$(whoami 2>/dev/null)}@$(uname -n 2>/dev/null) \$ "
fi
++ '[' -n '5.1.16(1)-release' ']'
++ '[' -f /etc/bash/bashrc ']'
++ . /etc/bash/bashrc
# /etc/bash/bashrc
#
# This file is sourced by all *interactive* bash shells on startup,
# including some apparently interactive shells such as scp and rcp
# that can't tolerate any output.  So make sure this doesn't display
# anything or bad things will happen !

# Test for an interactive shell.  There is no need to set anything
# past this point for scp and rcp, and it's important to refrain from
# outputting anything in those cases.
if [[ $- != *i* ]] ; then
        # Shell is non-interactive.  Be done now!
        return
fi
+++ [[ ehvxB != *i* ]]
+++ return

for sh in /etc/profile.d/*.sh ; do
        [ -r "$sh" ] && . "$sh"
done
++ for sh in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/gawk.sh ']'
++ . /etc/profile.d/gawk.sh
gawkpath_default () {
        unset AWKPATH
        export AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`
}

gawkpath_prepend () {
        [ -z "$AWKPATH" ] && AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`
        export AWKPATH="$*:$AWKPATH"
}

gawkpath_append () {
        [ -z "$AWKPATH" ] && AWKPATH=`gawk 'BEGIN {print ENVIRON["AWKPATH"]}'`
        export AWKPATH="$AWKPATH:$*"
}

gawklibpath_default () {
        unset AWKLIBPATH
        export AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`
}

gawklibpath_prepend () {
        [ -z "$AWKLIBPATH" ] && \
                AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`
        export AWKLIBPATH="$*:$AWKLIBPATH"
}

gawklibpath_append () {
        [ -z "$AWKLIBPATH" ] && \
                AWKLIBPATH=`gawk 'BEGIN {print ENVIRON["AWKLIBPATH"]}'`
        export AWKLIBPATH="$AWKLIBPATH:$*"
}
unset sh
++ unset sh
gcc -o main /main.c
+ gcc -o main /main.c
chmod +x /main.c
+ chmod +x /main.c
echo About to execute main:
+ echo About to execute main:
About to execute main:
my_output=`./main`
++ ./main
+ my_output='Hello compiled in a chroot environment!'
echo result :
+ echo result :
result :
echo =======================
+ echo =======================
=======================
echo $my_output
+ echo Hello compiled in a chroot 'environment!'
Hello compiled in a chroot environment!
echo ========================
+ echo ========================
========================
mydate =`date`
++ date
+ mydate =Fri Jan 6 05:58:48 -00 2023
/jlp_compile_test: line 12: mydate: command not found
jenk /home/jlpoole/local/Build.Dist # ./build/GenPi64OpenRC/chroot/main         qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory
jenk /home/jlpoole/local/Build.Dist #

[jlpoolen]

change root into the environment and executing the command did not trigger any errors:

jenk /home/jlpoole/local/Build.Dist # date
Fri Jan  6 08:46:45 AM PST 2023
jenk /home/jlpoole/local/Build.Dist # export JOHN_CHROOT=/home/jlpoole/local/Build.Dist/build/GenPi64OpenRC/chroot
mount --rbind /dev ${JOHN_CHROOT}/dev
mount --make-rslave ${JOHN_CHROOT}/dev
mount -t proc /proc ${JOHN_CHROOT}/proc
mount --rbind /sys ${JOHN_CHROOT}/sys
mount --make-rslave ${JOHN_CHROOT}/sys
mount --rbind /tmp ${JOHN_CHROOT}/tmp
mount --bind /run ${JOHN_CHROOT}/run
jenk /home/jlpoole/local/Build.Dist # chroot ${JOHN_CHROOT} /bin/bash
jenk / # env-update && . /etc/profile
export PS1="(chroot) $PS1"
>>> Regenerating /etc/ld.so.cache...
(chroot) jenk / #  eselect locale set en_US.utf8
Setting LANG to en_US.utf8 ...
Run ". /etc/profile" to update the variable in your shell.
(chroot) jenk / # . /etc/profile
jenk / # exit
exit
jenk /home/jlpoole/local/Build.Dist # umount -l ${JOHN_CHROOT}/{dev,sys}
jenk /home/jlpoole/local/Build.Dist # umount ${JOHN_CHROOT}/{proc,tmp,run}
jenk /home/jlpoole/local/Build.Dist #

[jonesmz]

That's great that you have it working now!

[jlpoolen]

No. Something in the build script process does not provide me the same environment I just demonstrated below. From a build.sh standpoint, the problem still exists.

jenk /home/jlpoole/local/Build.Dist # date;time ./build.sh
Fri Jan  6 08:58:04 AM PST 2023
Skipping arm registration for jlpoole build only
loading manifest
load complete
Loading status
status not loadable
load complete
[{'name': 'gentoo-base', 'deps': [''], 'parser': 'includejson', 'args': ['subtargets/gentoo-base.json', 'patchtargets/gentoo-aarch64.json', 'patchtargets/gentoo-openrc.json']}, {'name': 'genpi64-lite-finalize', 'deps': ['gentoo-base'], 'parser': 'includejson', 'args': ['subtargets/genpi64-lite-finalize.json']}, {'name': 'echo-lite-banner', 'deps': ['genpi64-lite-finalize'], 'parser': 'printf', 'args': ['############################\n# Build complete\n# To install your image on an SD card, run\n# dd if=/path/to/image of=/path/to/sdcard\n#############################\n']}]
checking  gentoo-base
checking  genpi64-lite-finalize
checking  echo-lite-banner
completed= ['']
manifest= [{'name': 'gentoo-base', 'deps': [], 'parser': 'includejson', 'args': ['subtargets/gentoo-base.json', 'patchtargets/gentoo-aarch64.json', 'patchtargets/gentoo-openrc.json']}, {'name': 'genpi64-lite-finalize', 'deps': ['gentoo-base'], 'parser': 'includejson', 'args': ['subtargets/genpi64-lite-finalize.json']}, {'name': 'echo-lite-banner', 'deps': ['genpi64-lite-finalize'], 'parser': 'printf', 'args': ['############################\n# Build complete\n# To install your image on an SD card, run\n# dd if=/path/to/image of=/path/to/sdcard\n#############################\n']}]
checking  gentoo-base deps []
all deps satisfied for gentoo-base
executing  gentoo-base deps []
checking  echo-lite-banner deps ['genpi64-lite-finalize']
running jobs [(<Popen: returncode: None args: ['/home/jlpoole/local/Build.Dist/parsers/incl...>, 'gentoo-base')]
Using sub-completed file
loading manifest
loading patch patchtargets/gentoo-aarch64.json
loading patch patchtargets/gentoo-openrc.json
load complete
Loading status
steps for completed-gentoo-base ['', 'downloadstage3', 'stage3', 'config', 'overlays', 'git', 'portage', 'etc']
load complete
[{'name': 'locale', 'deps': ['etc'], 'parser': 'rawcommand', 'args': ['eselect locale set en_US.utf8']}, {'name': 'update_portage', 'deps': ['locale'], 'parser': 'emerge', 'args': ['--verbose --oneshot --jobs --tree --ignore-default-opts sys-apps/portage']}, {'name': 'install_ccache', 'deps': ['update_portage'], 'parser': 'emerge', 'args': ['--verbose --jobs --tree --ignore-default-opts --binpkg-respect-use=y --binpkg-changed-deps=y --usepkg=y ccache']}, {'name': 'update_world', 'deps': ['install_ccache'], 'parser': 'emerge', 'args': ['--verbose --update @world']}, {'name': 'rebuild_system', 'deps': ['update_world'], 'parser': 'emerge', 'args': ['--verbose --emptytree @system']}, {'name': 'rebuild_world', 'deps': ['rebuild_system'], 'parser': 'emerge', 'args': ['--verbose --emptytree @world']}, {'name': 'set_profile', 'deps': ['rebuild_world'], 'parser': 'setprofile', 'args': ['']}, {'name': 'emerge_new_profile', 'deps': ['set_profile'], 'parser': 'emerge', 'args': ['--verbose --update @world']}, {'name': 'add_sets', 'deps': ['emerge_new_profile'], 'parser': 'addsets', 'args': ['']}, {'name': 'emerge_new_sets', 'deps': ['add_sets'], 'parser': 'emerge', 'args': ['--verbose --update @world']}, {'name': 'ccache_stats', 'deps': ['emerge_new_sets'], 'parser': 'rawcommand', 'args': ['ccache -d /var/tmp/ccache --show-stats']}, {'name': 'unmerge_ccache', 'deps': ['ccache_stats'], 'parser': 'emerge', 'args': ['--unmerge ccache']}, {'name': 'depclean', 'deps': ['unmerge_ccache'], 'parser': 'emerge', 'args': ['--verbose --depclean']}, {'name': 'eclean-pkg', 'deps': ['depclean'], 'parser': 'rawcommand', 'args': ['eclean-pkg --time-limit=1m']}, {'name': 'eclean-dist', 'deps': ['eclean-pkg'], 'parser': 'rawcommand', 'args': ['eclean-dist --time-limit=1m']}, {'name': 'users', 'deps': ['eclean-dist'], 'parser': 'users', 'args': ['']}, {'name': 'setup_services', 'deps': ['users'], 'parser': 'setupservices', 'args': ['']}, {'name': 'news', 'deps': ['setup_services'], 'parser': 'rawcommand', 'args': ['eselect news read &>/dev/null']}, {'name': 'set_root_password', 'deps': ['news'], 'parser': 'password', 'args': ['']}]
checking  locale
checking  update_portage
checking  install_ccache
checking  update_world
checking  rebuild_system
checking  rebuild_world
checking  set_profile
checking  emerge_new_profile
checking  add_sets
checking  emerge_new_sets
checking  ccache_stats
checking  unmerge_ccache
checking  depclean
checking  eclean-pkg
checking  eclean-dist
checking  users
checking  setup_services
checking  news
checking  set_root_password
completed= ['', 'downloadstage3', 'stage3', 'config', 'overlays', 'git', 'portage', 'etc']
manifest= [{'name': 'locale', 'deps': [], 'parser': 'rawcommand', 'args': ['eselect locale set en_US.utf8']}, {'name': 'update_portage', 'deps': ['locale'], 'parser': 'emerge', 'args': ['--verbose --oneshot --jobs --tree --ignore-default-opts sys-apps/portage']}, {'name': 'install_ccache', 'deps': ['update_portage'], 'parser': 'emerge', 'args': ['--verbose --jobs --tree --ignore-default-opts --binpkg-respect-use=y --binpkg-changed-deps=y --usepkg=y ccache']}, {'name': 'update_world', 'deps': ['install_ccache'], 'parser': 'emerge', 'args': ['--verbose --update @world']}, {'name': 'rebuild_system', 'deps': ['update_world'], 'parser': 'emerge', 'args': ['--verbose --emptytree @system']}, {'name': 'rebuild_world', 'deps': ['rebuild_system'], 'parser': 'emerge', 'args': ['--verbose --emptytree @world']}, {'name': 'set_profile', 'deps': ['rebuild_world'], 'parser': 'setprofile', 'args': ['']}, {'name': 'emerge_new_profile', 'deps': ['set_profile'], 'parser': 'emerge', 'args': ['--verbose --update @world']}, {'name': 'add_sets', 'deps': ['emerge_new_profile'], 'parser': 'addsets', 'args': ['']}, {'name': 'emerge_new_sets', 'deps': ['add_sets'], 'parser': 'emerge', 'args': ['--verbose --update @world']}, {'name': 'ccache_stats', 'deps': ['emerge_new_sets'], 'parser': 'rawcommand', 'args': ['ccache -d /var/tmp/ccache --show-stats']}, {'name': 'unmerge_ccache', 'deps': ['ccache_stats'], 'parser': 'emerge', 'args': ['--unmerge ccache']}, {'name': 'depclean', 'deps': ['unmerge_ccache'], 'parser': 'emerge', 'args': ['--verbose --depclean']}, {'name': 'eclean-pkg', 'deps': ['depclean'], 'parser': 'rawcommand', 'args': ['eclean-pkg --time-limit=1m']}, {'name': 'eclean-dist', 'deps': ['eclean-pkg'], 'parser': 'rawcommand', 'args': ['eclean-dist --time-limit=1m']}, {'name': 'users', 'deps': ['eclean-dist'], 'parser': 'users', 'args': ['']}, {'name': 'setup_services', 'deps': ['users'], 'parser': 'setupservices', 'args': ['']}, {'name': 'news', 'deps': ['setup_services'], 'parser': 'rawcommand', 'args': ['eselect news read &>/dev/null']}, {'name': 'set_root_password', 'deps': ['news'], 'parser': 'password', 'args': ['']}]
checking  locale deps []
all deps satisfied for locale
executing  locale deps []
checking  install_ccache deps ['update_portage']
checking  update_world deps ['install_ccache']
checking  rebuild_system deps ['update_world']
checking  rebuild_world deps ['rebuild_system']
checking  set_profile deps ['rebuild_world']
checking  emerge_new_profile deps ['set_profile']
checking  add_sets deps ['emerge_new_profile']
checking  emerge_new_sets deps ['add_sets']
checking  ccache_stats deps ['emerge_new_sets']
checking  unmerge_ccache deps ['ccache_stats']
checking  depclean deps ['unmerge_ccache']
checking  eclean-pkg deps ['depclean']
checking  eclean-dist deps ['eclean-pkg']
checking  users deps ['eclean-dist']
checking  setup_services deps ['users']
checking  news deps ['setup_services']
checking  set_root_password deps ['news']
running jobs [(<Popen: returncode: None args: ['/home/jlpoole/local/Build.Dist/parsers/rawc...>, 'locale')]
+ source /home/jlpoole/local/Build.Dist/scripts/functions.sh
+ echo 'Running eselect locale set en_US.utf8'
Running eselect locale set en_US.utf8
+ trap finish EXIT
+ cat
+ chmod +x /home/jlpoole/local/Build.Dist/build/GenPi64OpenRC/chroot/em-5803
+ /home/jlpoole/local/Build.Dist/scripts/chroot.py /em-5803
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pychroot/scripts/pychroot.py", line 130, in main
  File "/usr/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1847, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/em-5803'
+ finish
+ ret=1
+ cp /home/jlpoole/local/Build.Dist/build/GenPi64OpenRC/chroot/em-5803 /tmp/rawcommand_em-5803
+ echo copied em-5803 to /tmp/rawcommand_em-5803
copied em-5803 to /tmp/rawcommand_em-5803
+ rm -f /home/jlpoole/local/Build.Dist/build/GenPi64OpenRC/chroot/em-5803
+ exit 1
FATAL: JOBFAILED  locale
FATAL: JOBFAILED  gentoo-base
run complete.

real    0m2.136s
user    0m0.407s
sys     0m0.193s
jenk /home/jlpoole/local/Build.Dist #

So, I need to understand why the disparity of results and fix something.

[jonesmz]

Perhaps the most efficient way to figure this out is to have either myself or @samip5 ssh into this machine and do some diagnostics. You had this working in the past.

[samip5]

Perhaps the most efficient way to figure this out is to have either myself or @samip5 ssh into this machine and do some diagnostics. You had this working in the past.

Unfortunately, as the system is running systemd, it wouldn't be a good idea for me to try to use pychroot there. The same exact step is working fine inside Jenkins.

Unless you're not talking about that but rather to their machine?

[samip5]

I also have a strong feeling that only part of the issue is pychroot which we have had issues with in the past and systemd-nspawn at least is not recommended to even try on openrc as it will screw up things.

EDIT: Our builders use systemd-nspawn and not pychroot so that maybe why you're facing issues as it's not getting tested much.

[jonesmz]

My understanding was that jlpoole used openrc, not systemd ?

[jlpoolen]

Correct. all my Gentoo systems are OpenRC.

[samip5]

My understanding was that jlpoole used openrc, not systemd ?

I'm saying that as our builders use systemd, the pychroot one doesn't get as much testing so could definitely be broken in one way or another.

[jlpoolen]

I may have found the problem, and it may be a misplaced double quotation mark in rawcommand. My question is, I got the locale step?/task? which creates a little file executing:

eselect locale set en_US.utf8

and after I successfully ran it, it looks like the results are cached somewhere so on my next run it skips that step. I want to force the step/task to be reperformed to trigger the error as I want to test a difference in the code for rawcommand. What file stores previous results and what do I do to force that step to be performed again, and again? Obviously, I do not want the 15 minute huge download and processing steps that precede this step.

[jlpoolen]

found the cached file: build/GenPi64/completed-gentoo-base @jonesmz also suggested on Discord just as I found it. I'm getting mixed results with the placement of the double quote on line 22 rawcommand. Still experimenting.

[jlpoolen]

Here I am a year later, attempting to build on a new and faster machine with the current Gentoo image and ran into the same problem. Referencing this comment of this bug, I attempted to chroot into the new environment and my attempt failed:

ryzdesk /home/jlpoole/Build.Dist_Aug2023 # date; chroot ${JOHN_CHROOT} /bin/bash
Thu Jan 18 09:13:55 AM PST 2024
chroot: failed to run command '/bin/bash': Exec format error
ryzdesk /home/jlpoole/Build.Dist_Aug2023 # echo ${JOHN_CHROOT}
/home/jlpoole/Build.Dist_Aug2023/build/GenPi64OpenRC/chroot
ryzdesk /home/jlpoole/Build.Dist_Aug2023 # ls build/GenPi64OpenRC/chroot/bin/bash -la
-rwxr-xr-x 1 root root 868168 Oct 29 18:15 build/GenPi64OpenRC/chroot/bin/bash
ryzdesk /home/jlpoole/Build.Dist_Aug2023 #

A search suggests that problem is that the command '/bin/bash' is not present in the new environment, but the above proves that it is. I'll update this bug when I am able to successfully chroot and overcome this problem.

[samip5]

Here I am a year later, attempting to build on a new and faster machine with the current Gentoo image and ran into the same problem. Referencing this comment of this bug, I attempted to chroot into the new environment and my attempt failed:

ryzdesk /home/jlpoole/Build.Dist_Aug2023 # date; chroot ${JOHN_CHROOT} /bin/bash
Thu Jan 18 09:13:55 AM PST 2024
chroot: failed to run command '/bin/bash': Exec format error
ryzdesk /home/jlpoole/Build.Dist_Aug2023 # echo ${JOHN_CHROOT}
/home/jlpoole/Build.Dist_Aug2023/build/GenPi64OpenRC/chroot
ryzdesk /home/jlpoole/Build.Dist_Aug2023 # ls build/GenPi64OpenRC/chroot/bin/bash -la
-rwxr-xr-x 1 root root 868168 Oct 29 18:15 build/GenPi64OpenRC/chroot/bin/bash
ryzdesk /home/jlpoole/Build.Dist_Aug2023 #

A search suggests that problem is that the command '/bin/bash' is not present in the new environment, but the above proves that it is. I'll update this bug when I am able to successfully chroot and overcome this problem.

It is in fact present (otherwise it would tell you it did not find it rather than the current error), just wrong architecture one. I think you're building on amd64, and the bash binary is arm64, so you should setup qemu properly for user land emulation.

[jlpoolen]

Here I am a year later, attempting to build on a new and faster machine with the current Gentoo image and ran into the same problem. Referencing this comment of this bug, I attempted to chroot into the new environment and my attempt failed:

ryzdesk /home/jlpoole/Build.Dist_Aug2023 # date; chroot ${JOHN_CHROOT} /bin/bash
Thu Jan 18 09:13:55 AM PST 2024
chroot: failed to run command '/bin/bash': Exec format error
ryzdesk /home/jlpoole/Build.Dist_Aug2023 # echo ${JOHN_CHROOT}
/home/jlpoole/Build.Dist_Aug2023/build/GenPi64OpenRC/chroot
ryzdesk /home/jlpoole/Build.Dist_Aug2023 # ls build/GenPi64OpenRC/chroot/bin/bash -la
-rwxr-xr-x 1 root root 868168 Oct 29 18:15 build/GenPi64OpenRC/chroot/bin/bash
ryzdesk /home/jlpoole/Build.Dist_Aug2023 #

A search suggests that problem is that the command '/bin/bash' is not present in the new environment, but the above proves that it is. I'll update this bug when I am able to successfully chroot and overcome this problem.

It is in fact present (otherwise it would tell you it did not find it rather than the current error), just wrong architecture one. I think you're building on amd64, and the bash binary is arm64, so you should setup qemu properly for user land emulation.

That is correct assessment, @samip5. I did not have qemu configured for "aarch64" and will be re-emerging accordingly.

[jlpoolen]

I setup qemu properly:

         ryzdesk /home/jlpoole/Build.Dist_Aug2023 # date; ./scripts/binfmt.sh
        Thu Jan 18 02:08:30 PM PST 2024
        ryzdesk /home/jlpoole/Build.Dist_Aug2023 # ls /proc/sys/fs/binfmt_misc/
        aarch64  arm  register  status
        ryzdesk /home/jlpoole/Build.Dist_Aug2023 # 

and then proceeded to build anew. In other words, I erased the prior build directory and ran the ./build.sh script which started all over again. I ran into this same problem just now:

checking  news deps ['setup_services']
checking  set_root_password deps ['news']
running jobs [(<Popen: returncode: None args: ['/home/jlpoole/Build.Dist_Aug2023/parsers/et...>, 'etc')]
Copying etc config files for GenPi64OpenRC
pychroot: error: failed to run command '/usr/sbin/locale-gen': [Errno 2] No such file or directory: '/usr/sbin/locale-gen'
FATAL: JOB etc FAILED with exit code 1
FATAL: JOB gentoo-base FAILED with exit code 1
run complete.

I recall a problem, before, about white space (tabs vs. spaces) in a script file wreaking havoc by nullifying the command that was intended. This looks a lot like it might be the same issue. I'll have to go back and review my notes.