awk detection is broken on Musl-based Linux distros

Forking the issue https://github.com/NetBSD/pkgsrc/issues/130#issuecomment-2017688580.

Currently I test only the OpenMPTCProuter distro, but OpenWRT probably suffers in the same way.

The distro provides BusyBox version of AWK which seems to have a mix of features from nawk, gawk and probably some other implementations too: https://wiki.alpinelinux.org/wiki/Awk

My idea was that the boostrap/bootstrap script should either pick up the OS-native version (/usr/bin/awk -> ../../bin/busybox) or bootstrap its' own version (/usr/pkg/bin/nawk).

The bootstrapping seems to to ignore the former and require the latter. The problem is that the latter isn't built yet at the moment it gets required:

===> bootstrap command: bootstrap/bootstrap 
===> bootstrap started: Sun Mar 24 21:02:38 UTC 2024
===> running: /bin/sed -e 's|@DEFAULT_INSTALL_MODE@|'0755'|' /root/debug/src/github.com/NetBSD/pkgsrc.git/sysutils/install-sh/files/install-sh.in > /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh
===> running: /bin/chmod +x /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh
===> Creating default mk.conf in /root/debug/src/github.com/NetBSD/pkgsrc.git/work
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/share/mk
===> Bootstrapping mk-files
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/pkgtools/bootstrap-mk-files/files && env CP=/bin/cp  OPSYS=Linux MK_DST=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/share/mk ROOT_GROUP=root ROOT_USER=root SED=/bin/sed SYSCONFDIR=/usr/pkg/etc /bin/sh ./bootstrap.sh)
===> Bootstrapping bmake
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh configure  --prefix=/root/debug/src/github.com/NetBSD/pkgsrc.git/work --with-default-sys-path=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/share/mk --with-machine-arch=x86_64 )
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh make-bootstrap.sh)
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake
===> Building libnbcompat
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /bin/sh ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
===> Bootstrapping pkgtools
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive; env  CONFIG_SHELL=/bin/sh /bin/sh ./configure  --enable-static --disable-shared --disable-bsdtar --disable-bsdcpio --disable-bsdunzip --disable-posix-regex-lib --disable-xattr --disable-maintainer-mode --disable-acl --without-zlib --without-bz2lib --without-iconv --without-lzma --without-lzo2 --without-lz4 --without-nettle --without-openssl --without-xml2 --without-expat --without-zstd MAKE=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install; env  CPPFLAGS=' -I/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat -I/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive/libarchive' LDFLAGS=' -L/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat' LIBS=' -lnbcompat' /bin/sh ./configure  -C --enable-bootstrap --prefix=/usr/pkg --sysconfdir=/usr/pkg/etc --with-pkgdbdir=/usr/pkg/pkgdb --infodir=/usr/pkg/info --mandir=/usr/pkg/man  && STATIC_LIBARCHIVE=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/libarchive/.libs/libarchive.a STATIC_LIBARCHIVE_LDADD='' PKGSRC_MACHINE_ARCH=x86_64 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/add/pkg_add /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_add
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/admin/pkg_admin /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_admin
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/create/pkg_create /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_create
===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/pkg_install/info/pkg_info /root/debug/src/github.com/NetBSD/pkgsrc.git/work/sbin/pkg_info
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: {print $3; exit}: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/tools/gmake.mk" line 43: warning: "/usr/bin/make --version |               '{print $3; exit}'" returned non-zero status
===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/pkgtools/cwrappers && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  MAKE_JOBS=1 PKG_COMPRESSION=none PKGSRC_KEEP_BIN_PKGS=no MAKECONF=/root/debug/src/github.com/NetBSD/pkgsrc.git/work/mk.conf install)
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: -f: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/lang/gcc6/options.mk" line 77: warning: "echo  6.5.0 |   -f ../../mk/scripts/larger_symbol_version.awk" returned non-zero status
sh: {print $3; exit}: not found
bmake[1]: "/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/tools/gmake.mk" line 43: warning: "/usr/bin/make --version |               '{print $3; exit}'" returned non-zero status
/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/pkgformat/pkg/list-dependencies: line 47: /usr/pkg/bin/nawk: not found
===> Installing dependencies for cwrappers-20220403
/root/debug/src/github.com/NetBSD/pkgsrc.git/mk/pkgformat/pkg/list-dependencies: line 47: /usr/pkg/bin/nawk: not found
*** Error code 1

Stop.
bmake: stopped in /root/debug/src/github.com/NetBSD/pkgsrc.git/pkgtools/cwrappers
===> exited with status 1
aborted.

(I trimmed down details from all earlier steps except for the last three ones)

@jperkin responded on IRC on Sunday, March 17th, 2024:

I'd start from first-principles

firstly, does it not exist, or is /usr/pkg/bin/nawk there but somehow built with an invalid interpreter or something?

nawk is only bootstrapped if a native awk isn't found or is unsuitable, so I'd also check to see why your native awk isn't sufficient and if it can be provided/fixed instead

then if it is required and is missing, I'd be looking at the bootstrap log to figure out why it failed and how that wasn't reported as a failure to the bootstrap script

as this is Linux you'll be looking at mk/tools/tools.Linux.mk for how the native awk is found

To be continued.

Just to document my latest finding. It looks like the path for AWK is decided in mk/tools/replace.mk:

    226 .if !defined(TOOLS_IGNORE.awk) && !empty(_USE_TOOLS:Mawk)
    227 .  if ${PKGPATH} == lang/nawk
    228 MAKEFLAGS+=                     TOOLS_IGNORE.awk=
    229 .  elif !empty(_TOOLS_USE_PKGSRC.awk:M[yY][eE][sS])
    230 TOOLS_DEPENDS.awk?=             nawk>=20040207:../../lang/nawk
    231 TOOLS_CREATE+=                  awk
    232 TOOLS_PATH.awk=                 ${TOOLBASE}/bin/nawk
    233 .  endif
    234 .endif

... with TOOLBASE being set in mk/bsd.prefs.mk:

    607 LOCALBASE?=             /usr/pkg
    608 TOOLBASE:=              ${LOCALBASE}

It seems mk/tools/tools.Linux.mk tries to find "gawk" specifically, and otherwise will not assign anything. That whole file basically assumes GNU tooling. (You kind of already partly answered your own question there in https://github.com/NetBSD/pkgsrc/issues/130#issuecomment-2016946196 .)

A dependency may indeed be specified in mk/tools/replace.mk where you quote, in certain contexts only.

(If you're looking for a single place where everything is revealed, well, a lot of the pkgsrc infrastructure isn't like that. It's more complicated.)

Confirmed on the following distros, attaching the logs inline:

Quoting @jperkin's IRC response and inviting for assistance:

firstly, does it not exist, or is /usr/pkg/bin/nawk there but somehow built with an invalid interpreter or something?

Surely the file does not exist yet:

# ls /usr/pkg/
ls: /usr/pkg/: No such file or directory

nawk is only bootstrapped if a native awk isn't found or is unsuitable, so I'd also check to see why your native awk isn't sufficient and if it can be provided/fixed instead

My impression is that nawk was not bootstrapped. That's the root problem I see.

then if it is required and is missing, I'd be looking at the bootstrap log to figure out why it failed and how that wasn't reported as a failure to the bootstrap script

Jonathan, I still doesn't understand that part. Sounds too abstract, maybe even ambiguous to me. A guidance is needed :)

as this is Linux you'll be looking at mk/tools/tools.Linux.mk for how the native awk is found

The only awk references I found in mk/tools/tools.Linux.mk are:

    105 .  if exists(${_path}/gawk)
    106 TOOLS_PLATFORM.gawk?=           ${_path}/gawk
    107 .  endif
    ...
    280
    281 # assuming grep/sed/tar/awk are usually the GNU versions, is this safe?
    282
    283 .if !empty(TOOLS_PLATFORM.bison)
    284 TOOLS_PLATFORM.bison-yacc?=     ${TOOLS_PLATFORM.bison} -y
    285 .endif
    286
    287 TOOLS_PLATFORM.awk?=            ${TOOLS_PLATFORM.gawk}
    288

Which @dhgutteridge explained above:

It seems mk/tools/tools.Linux.mk tries to find "gawk" specifically, and otherwise will not assign anything. That whole file basically assumes GNU tooling.

Debian 12 also provides non-GNU implementation by default, the mawk. Interestingly, pkgsrc decides to bootstrap its' own nawk here just fine resulting in /usr/pkg/bin/nawk.

From 05-bootstrap-builds-nawk-on-Debian-12.log:

--- /root/debug/src/github.com/NetBSD/pkgsrc.git/debug/logs/03-bootstrap-misses-nawk-on-Alpine-Linux.log
+++ /root/debug/src/github.com/NetBSD/pkgsrc.git/debug/logs/05-bootstrap-builds-nawk-on-Debian-12.log
@@ -1,16 +1,16 @@
 ===> bootstrap command: bootstrap/bootstrap
-===> bootstrap started: Fri Apr 19 20:26:12 EEST 2024
+===> bootstrap started: Sat Apr 20 08:29:37 AM EEST 2024
 Working directory is: /root/debug/src/github.com/NetBSD/pkgsrc.git/work
  ...
-===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /bin/sh make-bootstrap.sh)
  ...
-===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake
+===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake && /usr/bin/sh make-bootstrap.sh)
  ...
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bmake/bmake /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake
 ===> Building libnbcompat
-===> running: /bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat
-===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /bin/sh ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat
+===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/libnbcompat; /usr/bin/sh ./configure  -C --prefix=/usr/pkg --infodir=/usr/pkg/info --mandir=/usr/pkg/man --sysconfdir=/usr/pkg/etc --enable-bsd-getopt --enable-db && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1)
 configure: creating cache config.cache
-checking build system type... x86_64-pc-linux-musl
-checking host system type... x86_64-pc-linux-musl
+checking build system type... x86_64-pc-linux-gnu
+checking host system type... x86_64-pc-linux-gnu
 checking for gcc... gcc
 checking whether the C compiler works... yes
 checking for C compiler default output file name... a.out
@@ -265,13 +267,11 @@
 checking whether ln -s works... yes
 checking for ranlib... ranlib
 checking for gawk... no
-checking for mawk... no
-checking for nawk... no
-checking for awk... awk
+checking for mawk... mawk
 checking for ar... ar
 checking for fparseln in -lutil... no
-checking for grep that handles long lines and -e... /bin/grep
-checking for egrep... /bin/grep -E
+checking for grep that handles long lines and -e... /usr/bin/grep
+checking for egrep... /usr/bin/grep -E
 checking for _Bool... yes
 checking for stdbool.h that conforms to C99... yes
 checking for alloca.h... yes
  ...
 ranlib libnbcompat.a
+===> Bootstrapping awk
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -d -o root -g root /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk
+===> running: (cd /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk && /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/bmake  -j1 -f Makefile CC="cc" CFLAGS="")
+cc   -c awkgram.tab.c
+cc   -c b.c
+cc   -c main.c
+cc   -c parse.c
+cc   -c proctab.c
+cc   -c tran.c
+cc   -c lib.c
+cc   -c run.c
+cc   -c lex.c
+cc  awkgram.tab.o b.o main.o parse.o proctab.o tran.o lib.o run.o lex.o   -lm -o a.out
+===> running: /usr/bin/sh /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/install-sh -c -o root -g root -m 755 /root/debug/src/github.com/NetBSD/pkgsrc.git/work/awk/a.out /root/debug/src/github.com/NetBSD/pkgsrc.git/work/bin/nawk
 ===> Bootstrapping pkgtools

I wonder what makes the bootstrap fail to reach (or just skip) this step on the topic distros.

OK, found the answer in bootstrap/bootstrap:

 737         # Debian/Ubuntu's awk is mawk, and mawk does not understand
 738         # some regexps used in pkgsrc/mk.
 739         if [ -f /etc/debian_version ]; then
 740                 need_awk=yes

So it's tied to the hardcoded OS name. Waiting for the suggestions on how to proceed in a clean way :)

So it's tied to the hardcoded OS name. Waiting for the suggestions on how to proceed in a clean way :)

The main thing we would avoid changing (active users and all...) is Enterprise Linux. Perhaps you can tell us if GNU awk is specifically identifiable on distributions like CentOS @sskras (perhaps installed as "gawk" in some directory)? Otherwise we can set need_awk=yes on all Linux...

Thanks @alarixnia. I think I would rather:

collect the needed AWK features,
detect during bootstrap if any on them is missing,
only then fall back to need_awk=yes.

Perhaps you can tell us if GNU awk is specifically identifiable on distributions like CentOS @sskras (perhaps installed as "gawk" in some directory)?

On CentOS 9 it's this:

saukrs@mp-client:~$ type awk gawk
awk is /usr/bin/awk
gawk is /usr/bin/gawk

saukrs@mp-client:~$ ll /usr/bin/gawk /usr/bin/awk
lrwxrwxrwx. 1 root root      4 Feb 16  2022 /usr/bin/awk -> gawk*
-rwxr-xr-x. 1 root root 714976 Feb 16  2022 /usr/bin/gawk*

saukrs@mp-client:~$ . /etc/os-release; echo $PRETTY_NAME
CentOS Stream 9

Not sure if that helps.

The same issue happens on Adelie Linux 1.0-beta5, also Musl-based distro (but no Busybox here): 10-bootstrap-misses-nawk-on-Adelie-Linux.log

NetBSD / pkgsrc

awk detection is broken on Musl-based Linux distros #132