Cannot build busybox with tar --to-command support enabled

[ogregpt]

Followed execellent build instructions for Windows producing an executable (mingw64_defconfig). Found the tar --to-command missing, but implemented in menuconfig: make menuconfig, enable tar --to-command, save make. One compiler error: CC archival/libarchive/data_extract_to_command.o In file included from archival/libarchive/data_extract_to_command.c:5: archival/libarchive/data_extract_to_command.c: In function 'data_extract_to_command': include/libbb.h:1295:18: warning: implicit declaration of function 'vfork'; did you mean 'xfork'? [-Wimplicit-function-declaration] 1295 | #define xvfork() vfork() | ^~~~~ archival/libarchive/data_extract_to_command.c:84:42: note: in expansion of macro 'xvfork'

I'm sure it's trivial, but I'm stuck here...

[avih]

It's not necessarily trivial.

mingw64_defconfig already includes all/most of the build options which can build and work on windows. The disabled options typically are irrelevant on windows or don't (currently) build for windows.

Specifically, I'm guessing this option is disabled because Windows doesn't have fork (mapped from xfork in busybox source), so it doesn't build.

I don't know how much effort it would require to make it work, but it's not necessarily trivial, and possibly even hard or very hard.

[ogregpt]

I understand. My problem was that the menuconfig sort of indicates that the option is available+selectable - I would prefer not to be able to select it instead of a compilation error! I'm not going to make a request for the feature, and BTW: busybox/Windows already is immensely useful... Regards

[avih]

I would prefer not to be able to select it instead of a compilation error!

I'm not an expert on this, but I don't think the build system supports automatic detection of which features would be able to compile (i.e. a "configure" step). The build system is from upstream busybox, and not uniqe to busybox-w32.

That's why there are various pre-configured combinations - like mingw64_defconfig, and a manual config option if you know what you're doing.

In your case, you can probably consider that all the viable options are already enabled in one of the mingw_* configs (you might also want to try the mingw64u_defconfig to add unicode support).

Though as I said, I'm not an expert with this build system (which is based on the linux kernel build system, as far as I know).

[rmyorston]

What @avih says is correct. The build system comes from upstream BusyBox, it's based on the Linux kernel build system (though quite an old version) and it doesn't do any feature detection.

It isn't really practical to keep track of what might or might not work. The only reliable test is to try a build and see what happens.

[ogregpt]

Fair enough - I just made a mistake assuming that a configurable option was implemented. You already do a fabulous job with busybox, I'll code my way around the missing feature... Regards

[avih]

For future rerefence, this is a simplistic script which pipes each file into a command, e.g. to count the bytes in each file:

untar2cmd.sh foo.tar.gz wc -c

The command can use the env vars TAR_ARCHIVE and TAR_FILENAME:

#!/bin/sh

# untar2cmd.sh: simplified tar --to-command functionality, by avih
USAGE="Usage: $0 ARCHIVE CMD...    pipe each (non-dir) file in ARCHIVE into CMD..."
# CMD can use env TAR_ARCHIVE (the archive name) and TAR_FILENAME (current file)

err() { >&2 printf %s\\n "$*"; exit 1; }

[ $# -ge 2 ] || err "$USAGE"

ar=$1
shift

export TAR_ARCHIVE="$ar"

tar -t -f "$ar" | {
    has=
    while IFS= read -r f; do
        has=yes
        case $f in */) continue; esac  # directory

        export TAR_FILENAME="$f"

        case $f in *["[?*"]*)  # escape glob chars
            # there seems to be a bug in busybox tar where if glob chars are
            # escaped, then the file is extracted correctly, but it prints
            # an error message that the file is not found in the archive.
            f=$(printf %s "$f" | sed 's/\([[?*]\)/\\\1/g')
        esac

        tar x -f "$ar" -O -- "$f" | "$@" \
            || >&2 echo "tar: Child returned status $? -- $TAR_FILENAME"
    done

    # no pipefail, so guess fail if no files in ARCHIVE (likely can't untar)
    [ "$has" ]
}

[ogregpt]

Thank you for your effort! I already tried something similar to your script, but more basic - I stopped because busybox/tar complains, like you describe... Now that I know about the messages I can live with them: they're less than 10 in my use case. Your script works fine and fast, except for symlinks: only regular files are handled by tar --to-command, your script treats them as regular files. I use the setup to read-check Restic archives, by md5sum-ming all files in a snapshot and comparing them to a similar list made at backup time.

[avih]

script works fine and fast, except for symlinks: only regular files are handled by tar --to-command, your script treats them as regular files.

I don't think a script like this can detect the type (or date, or many other properties) without actually extracting the file first to the filesystem.

The only thing it can do is list the entries in the archive, and ignore the directories (because then end in /), so that's what it does, and that's what it documents: "pipe each (non-dir) file in ARCHIVE into CMD...".

Feel free to improve it as you see fit.