valid targets do not resolve under pbuilder

[udance4ever]

I found failures due to make targets not found (which might imply wrong working directory):

output/x86_64/pbuilder_logs$ grep "No rule" *
batocera-nvidia.log:make[2]: *** No rule to make target 'batocera-nvidia'.  Stop.
libretro-xrick.log:make[2]: *** No rule to make target 'libretro-xrick'.  Stop.

verified both targets exist:

squatter@lina:~/dev/parallel2$ make x86_64-shell

squatter@5677da9c5081:/x86_64$ make batocera-nvidia-show-depends
host-ccache host-skeleton host-tar skeleton toolchain

squatter@5677da9c5081:/x86_64$ make libretro-xrick-show-depends
host-ccache host-skeleton host-tar skeleton toolchain

impacted targets in this specific build to look for patterns:

no rule

1. batocera-nvidia
2. libretro-xrick
3. libretro-beetle-wswan
4. retrogame
5. openbor4432 (bcm2836) (didn’t happen during x86_64 pbuild)

no target host-glslang (needed by mesa3d)

1. xdriver_xf86-input-synaptics
2. qt5base

[udance4ever]

verified all 5 targets build outside of pbuilder.

example output from 2 pkgs:

squatter@6aad9d9b0b44:/x86_64$ make batocera-nvidia
>>> batocera-nvidia 1.4 Extracting
>>> batocera-nvidia 1.4 Patching
>>> batocera-nvidia 1.4 Configuring
>>> batocera-nvidia 1.4 Building
>>> batocera-nvidia 1.4 Installing to target
install -m 0755 /build/package/batocera/gpu/batocera-nvidia/batocera-nvidia /x86_64/target/usr/bin/
# [Xorg]
mkdir -p /x86_64/target/etc/X11/xorg.conf.d/
ln -sf /userdata/system/99-nvidia.conf /x86_64/target/etc/X11/xorg.conf.d/99-nvidia.conf
# [Blacklist & Modprobe]
mkdir -p /x86_64/target/etc/modprobe.d/
ln -sf /var/run/nvidia/modprobe/blacklist-nouveau.conf /x86_64/target/etc/modprobe.d/blacklist-nouveau.conf
ln -sf /var/run/nvidia/modprobe/nvidia-drm.conf        /x86_64/target/etc/modprobe.d/nvidia-drm.conf

and

squatter@6aad9d9b0b44:/x86_64$ make libretro-xrick
WARNING: no hash file for libretro-xrick-58e544ee8de08ca8421e97c064d52ef6cdd73bd2.tar.gz
>>> libretro-xrick 58e544ee8de08ca8421e97c064d52ef6cdd73bd2 Extracting
gzip -d -c /build/buildroot/dl/libretro-xrick/libretro-xrick-58e544ee8de08ca8421e97c064d52ef6cdd73bd2.tar.gz | /x86_64/host/bin/tar --strip-components=1 -C /x86_64/build/libretro-xrick-58e544ee8de08ca8421e97c064d52ef6cdd73bd2   -xf -
>>> libretro-xrick 58e544ee8de08ca8421e97c064d52ef6cdd73bd2 Patching
>>> libretro-xrick 58e544ee8de08ca8421e97c064d52ef6cdd73bd2 Configuring
>>> libretro-xrick 58e544ee8de08ca8421e97c064d52ef6cdd73bd2 Building
GIT_DIR=. PATH="/x86_64/host/bin:/x86_64/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" AR="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc-ar" AS="/x86_64/host/bin/x86_64-buildroot-linux-gnu-as" LD="/x86_64/host/bin/x86_64-buildroot-linux-gnu-ld" NM="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc-nm" CC="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc" GCC="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc" CPP="/x86_64/host/bin/x86_64-buildroot-linux-gnu-cpp" CXX="/x86_64/host/bin/x86_64-buildroot-linux-gnu-g++" FC="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gfortran" F77="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gfortran" RANLIB="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc-ranlib" READELF="/x86_64/host/bin/x86_64-buildroot-linux-gnu-readelf" STRIP="/x86_64/host/bin/x86_64-buildroot-linux-gnu-strip" OBJCOPY="/x86_64/host/bin/x86_64-buildroot-linux-gnu-objcopy" OBJDUMP="/x86_64/host/bin/x86_64-buildroot-linux-gnu-objdump" AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as" CC_FOR_BUILD="/x86_64/host/bin/ccache /usr/bin/gcc" GCC_FOR_BUILD="/x86_64/host/bin/ccache /usr/bin/gcc" CXX_FOR_BUILD="/x86_64/host/bin/ccache /usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld" CPPFLAGS_FOR_BUILD="-I/x86_64/host/include" CFLAGS_FOR_BUILD="-O2 -I/x86_64/host/include" CXXFLAGS_FOR_BUILD="-O2 -I/x86_64/host/include" LDFLAGS_FOR_BUILD="-L/x86_64/host/lib -Wl,-rpath,/x86_64/host/lib" FCFLAGS_FOR_BUILD="" DEFAULT_ASSEMBLER="/x86_64/host/bin/x86_64-buildroot-linux-gnu-as" DEFAULT_LINKER="/x86_64/host/bin/x86_64-buildroot-linux-gnu-ld" CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1" CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1" LDFLAGS="" FCFLAGS=" -O2 -g0" FFLAGS=" -O2 -g0" PKG_CONFIG="/x86_64/host/bin/pkg-config" STAGING_DIR="/x86_64/host/x86_64-buildroot-linux-gnu/sysroot" INTLTOOL_PERL=/usr/bin/perl /usr/bin/make -j9 CXX="/x86_64/host/bin/x86_64-buildroot-linux-gnu-g++" CC="/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc" -C /x86_64/build/libretro-xrick-58e544ee8de08ca8421e97c064d52ef6cdd73bd2/ -f Makefile.libretro platform="unix"
/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1 -DFRONTEND_SUPPORTS_RGB565 -O2 -DNDEBUG -DWANT_ZLIB -fsigned-char -Wno-strict-prototypes -ffast-math -fomit-frame-pointer -fno-builtin -I./include -I./include/compat -I./src -I./libretro -I./libretro/sdl_wrapp -I./libretro/include -I./libretro-common/include -D__LIBRETRO__ -DINLINE="inline" -fPIC  -c -osrc/unzip.o src/unzip.c
.
.
.
/x86_64/host/bin/x86_64-buildroot-linux-gnu-gcc -o xrick_libretro.so -fPIC -shared -Wl,--version-script=./libretro/core/link.T -Wl,--no-undefined ./src/unzip.o ./src/data.o ./src/scr_xrick.o ./src/scr_pause.o ./src/scr_imain.o ./src/scr_imap.o ./src/scr_gameover.o ./src/scr_getname.o ./src/dat_picsPC.o ./src/dat_picsST.o ./src/dat_screens.o ./src/dat_tilesPC.o ./src/dat_tilesST.o ./src/dat_maps.o ./src/dat_ents.o ./src/dat_spritesST.o ./src/dat_spritesPC.o ./src/ents.o ./src/e_bullet.o ./src/e_bomb.o ./src/e_rick.o ./src/e_sbonus.o ./src/e_them.o ./src/e_bonus.o ./src/e_box.o ./src/rects.o ./src/util.o ./src/game.o ./src/xrick.o ./src/draw.o ./src/maps.o ./src/sysvid.o ./src/control.o ./src/system.o ./src/scroller.o ./src/sysarg.o ./src/dat_snd.o ./libretro/core/soundplay.o ./libretro/sdl_wrapp/sdl_primitives.o ./libretro/core/libretro-core.o ./libretro/core/core-mapper.o ./deps/libz/adler32.o ./deps/libz/crc32.o ./deps/libz/inflate.o ./deps/libz/inftrees.o ./deps/libz/inffast.o ./deps/libz/zutil.o ./libretro-common/compat/compat_posix_string.o ./libretro-common/compat/compat_snprintf.o ./libretro-common/compat/compat_strcasestr.o ./libretro-common/compat/compat_strl.o ./libretro-common/compat/fopen_utf8.o ./libretro-common/encodings/encoding_utf.o ./libretro-common/file/file_path.o ./libretro-common/file/file_path_io.o ./libretro-common/streams/file_stream.o ./libretro-common/streams/file_stream_transforms.o ./libretro-common/string/stdstring.o ./libretro-common/time/rtime.o ./libretro-common/vfs/vfs_implementation.o  -lm
>>> libretro-xrick 58e544ee8de08ca8421e97c064d52ef6cdd73bd2 Installing to target
/usr/bin/install -D /x86_64/build/libretro-xrick-58e544ee8de08ca8421e97c064d52ef6cdd73bd2/xrick_libretro.so /x86_64/target/usr/lib/libretro/xrick_libretro.so

do you believe the most likely cause is pbuilder somehow lost track of its CWD?

how can I help troubleshoot this?

[udance4ever]

ok - this one is actually going to be more challenging to pinpoint as it is intermittent.

during my current pbuild, buildroot couldn’t find the rule for openbor4432 inside the pbuilder environment.

Instead of building the package outside of pbuilder, I just executed the graph again and it built the second time.

I am wondering what is causing pbuilder to run make outside of BR2_EXTERNAL? 🤔

https://github.com/paguilar/br-pbuilder/blob/5b83f07cbaad67657aab234d860fce76f6337dec/src/graph_exec.c#L333

https://github.com/paguilar/br-pbuilder/blob/5b83f07cbaad67657aab234d860fce76f6337dec/src/graph_exec.c#L353

my understanding is each make process is spawned so there’s no way the execution of the child process could modify the PWD.

on quick glance, I don’t see changing directories anywhere either.

perhaps one way to troubleshoot this is to print the CWD before make.

and if that doesn’t work, perhaps change directory to BR2_EXTERNAL before invoking make and see if this changes anything?

just some ideas :)

[paguilar]

Instead of building the package outside of pbuilder, I just executed the graph again and it built the second time.

I've seen that when some br-pbuilder executions give an error and then executing them again doesn't, is related to some Buildroot scripts or makefiles that get executed simultaneously. For example, thread 1 executes makefile A and also thread 2 executes makefile A, but thread 1 is slightly faster and moves/removes files that thread 2 is expecting, but it doesn't find them so it gives an error. The patch 0003 in br-pbuilder/patches is an example of this situation. May be this

I found failures due to make targets not found (which might imply wrong working directory):

is another case that pops up with in this configuration. I'll dig around to check if this is the case.

[udance4ever]

it sounds like you can reproduce the issue on your end?

The bcm2836 build is much smaller if it makes a difference:

git clone https://github.com/udance4ever/batocera.linux.git --branch br-pbuilder --depth 1
git submodule update --init --recursive
make bcm2836-pbuild

good news is the build eventually completes & produces a bootable image 🚀

however, in the bcm2836 case, buildroot's per-package parallel build works so there's less juice with this board (but may be easier to debug pbuilder)

it is the x86_64 config that shows promise as buildroot's per-package introduces a graphical glitch that doesn't allow the UI to come up on boot. By avoiding the complexities of rsync & hard linking packages, pbuilder is able to create a working image despite a few stumbles along the way (which I feel are surmountable!)

let me know if you need any more info from me.

[paguilar]

Thanks for pointing the bcm2836 defconfig, I'll use it to reduce testing time and storage space :)

[udance4ever]

hey I noticed you now pass BR2_EXTERNAL to make in this commit - I can see this making a difference in making sure buildroot is in the correct working directory.

https://github.com/paguilar/br-pbuilder/blob/1b09261da4e9f396d261c9dd76fc174223ef3b53/src/graph_exec.c#L75

I’m firing off a clean build with both your commits to see if it resolves this issue, #2 and #6 🤞🏼 I’ll report back by tomorrow.

[udance4ever]

ah - unfortunately - the build already stopped by a different set of 5 packages with the same issue.

So you are right, the No rule to make target ‘<package>’ is not specific to any particular package.

workaround is to rerun pbuilder and have it build these 5 packages (one package took 3 tries)

[paguilar]

This confirms that there should be some other makefile/script that, when executed twice at the same time, it changes/removes files that is expecting. One way to solve this is by using locks, but first I need to find the makefile/script. I'm working on it, but it takes time...