On Windows, Meson doesn't support the .cmd stubs

[ghost]

Describe the bug In case of we use Meson for a cross-compilation (build machine = Win10, host machine = RISC-V), the .cmd stubs which can be used as the toolchain and created by a xPack/XPM tool are not supported.

During the meson setup --cross-file riscv.crossfile build_folder command through a xpm run prepare command, Meson doesn't find the binaries listed into the Meson cross build definition file (riscv.crossfile).

The problem affects all projects which have dependencies installed with npm/xpm (both!).

To Reproduce

meson setup --cross-file riscv.crossfile 10_Builds\Meson\debug
The Meson build system
Version: 0.63.3
Source dir: C:\dev\riscv-disco\my_project
Build dir: C:\dev\riscv-disco\my_project\10_Builds\Meson\debug
Build type: cross build
Project name: my_project
Project version: 0.1

meson.build:1:0: ERROR: Unknown compiler(s): [['riscv-none-elf-gcc']]
The following exception(s) were encountered:
Running `riscv-none-elf-gcc --version` gave "[WinError 2] The system cannot find the file specified"

A full log can be found at C:\dev\riscv-disco\my_project\10_Builds\Meson\debug\meson-logs\meson-log.txt
error: running 'meson setup --cross-file riscv.crossfile 10_Builds\Meson\debug' failed

Add .cmd extension at the end of all binaries files of the list of the binaries into the Meson cross build definition file.

[binaries]
c       = 'riscv-none-elf-gcc.cmd'
cpp     = 'riscv-none-elf-g++.cmd'
ld      = 'riscv-none-elf-ld.cmd'
ar      = 'riscv-none-elf-ar.cmd'
as      = 'riscv-none-elf-gcc.cmd -x assembler-with-cpp'
size    = 'riscv-none-elf-size.cmd'
objdump = 'riscv-none-elf-objdump.cmd'
objcopy = 'riscv-none-elf-objcopy.cmd'
strip   = 'riscv-none-elf-strip.cmd'
gdb     = 'riscv-none-elf-gdb.cmd'

[host_machine]
system     = 'none'
cpu_family = 'riscv32'
cpu        = 'rv32imc'
endian     = 'little'

Re-launch the Meson setup command:

meson setup --cross-file riscv.crossfile 10_Builds\Meson\debug
The Meson build system
Version: 0.63.3
Source dir: C:\dev\riscv-disco\my_project
Build dir: C:\dev\riscv-disco\my_project\10_Builds\Meson\debug
Build type: cross build
Project name: my_project
Project version: 0.1
C compiler for the host machine: riscv-none-elf-gcc.cmd (gcc 12.2.0 "riscv-none-elf-gcc.exe (xPack GNU RISC-V Embedded GCC x86_64) 12.2.0")
C linker for the host machine: riscv-none-elf-gcc.cmd ld.bfd 2.38
C compiler for the build machine: gcc (gcc 8.1.0 "gcc (x86_64-win32-seh-rev0, Built by MinGW-W64 project) 8.1.0")
C linker for the build machine: gcc ld.bfd 2.30
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: riscv32
Host machine cpu: rv32imc
Target machine cpu family: riscv32
Target machine cpu: rv32imc
Message: C compiler version: 12.2.0
Program riscv-none-elf-objcopy.cmd found: YES
Program riscv-none-elf-objdump.cmd found: YES
Program riscv-none-elf-size.cmd found: YES
Program riscv-none-elf-gdb.cmd found: YES
Build targets in project: 2

my_project 0.1

  User defined options
    Cross files: riscv.crossfile

Found ninja-1.11.1 at C:\dev\riscv-disco\my_project\xpacks\.bin\ninja.CMD

Expected behavior Don't add a .cmd extension at the end of all binary files of the list of the binaries into the Meson cross build definition file to ensure that the build is cross-platform.

system parameters

• Is this a cross build
• Windows 10 22H2
• Python 3.11.3
• Meson v0.63.3
• Ninja v1.11.1
• Original issue: On Windows, meson doesn't support the .cmd stubs to individual programs added in xpacks/.bin

[eli-schwartz]

Don't add a .cmd extension at the end of all binary files of the list of the binaries into the Meson cross build definition file to ensure that the build is cross-platform.

I don't understand this, because cross files by definition aren't cross-platform -- each cross file is specific to a given platform, due to containing e.g.

[host_machine]
system = 'linux'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

[eli-schwartz]

That being said, my understanding is that on Windows, you can run a command as foobar and it will execute foobar.cmd. In fact, python's shutil.which() function explicitly, manually, does this:

>>> shutil.which('foobar')
'path/to/foobar.cmd'

And same for .exe files.

The flip side is that if you don't name the file with a .cmd extension, as far as Microsoft is concerned, it isn't one and Meson has no right to assume it is one.

[xclaesse]

I think the problem is that we don't call shutil.which() on env.lookup_binary_entry() result. That has more implications than just the missing .cmd extension, it also means that if PATH is different at configure and build times, we could be missing those binaries, and ninja will have to lookup in PATH for every invocation since we don't give a fullpath.

[ilg-ul]

First, a bit of background info.

In order to achieve reproducible builds, one solution is to create local build environments with the exact versions of the required tools. On macOS/Linux these can be easily achieved by unpacking multiple versions of the tools in separate folders, and making symbolic links to the desired versions in a folder like <project>/.bin, added in front of the build path. In this way the local tools are preferred to possible system tools with the same names.

Unfortunately this mechanism does not work on Windows, since there are no reliable links to files.

The alternate solution, used by tools like npm and xpm (which complements npm for C/C++ projects), is to use .cmd and .ps1 shims, which are simple scripts for the corresponding shells, to start the actual tools from their location.

Since Windows does not have a mechanism like #!/usr/bin/env bash to automatically identify the program to run a script, invoking such shims require an explicit shell invocation at application level, when child processes are created.

---

For meson, although this ticket reported a failure during the automatic toolchain identification, the problem can also occur when the meson scripts call any third party tools which were installed as npm/xpm dependencies, for example when running tests via an emulator like QEMU; or, in complex builds, for various other tasks, like managing configurations, generating source files, etc.

To summarise, on Windows there is not much that can be done, the shims are called .cmd and .ps1 and must be invoked via cmd.exe or PowerShell.

From my experience, except tools that have an explicit .exe, to achieve an uniform way of creating subprocesses, they must be started via an explicit shell.

Thus I suggest to consider this issue in a broader context, not only for invoking toolchain binaries, which can be worked around with the explicit .cmd in the cross files.

[eli-schwartz]

The alternate solution, used by tools like npm and xpm (which complements npm for C/C++ projects), is to use .cmd and .ps1 shims, which are simple scripts for the corresponding shells, to start the actual tools from their location.

This is in marked contrast to tools like pip, which are very particular about providing a true first-class Windows experience. When pip installs a *.py script, it makes sure that it can be run on Windows, no matter what -- even though .py files aren't really executable there -- and to do that, it explicitly shunned the use of .cmd files on the grounds that they "don't reliably work".

The solution was https://bitbucket.org/vinay.sajip/simple_launcher/

This is a simple launcher for script files under Windows, originally developed for use with Python scripts, though nothing in it is Python- specific. There are two versions of the launcher - console and GUI - built from the same source code.

The launcher is intended to facilitate script execution under Windows by processing shebang lines in an analogous way to how shebang lines are processed on POSIX platforms.

There are two modes to the launcher -- depending on whether APPENDED_ARCHIVE is defined, it either attempts to run a .zip file appended to the executable, or else searches for static wchar_t suffix[] = { L"-script.py" }; as a replacement for the .exe extension of the current executable (i.e. the Windows analogue to /proc/self/exe)

From my experience, except tools that have an explicit .exe, to achieve an uniform way of creating subprocesses, they must be started via an explicit shell.

Python's pip ecosystem indeed decided that having that explicit .exe is the only reliable solution.

[ilg-ul]

Unfortunately the npm/xpm ecosystem uses shims, not launchers, and I doubt there is anything to do about it; starting them via a shell might not be perfect, but it is functional.

[eli-schwartz]

Unfortunately for the npm ecosystem. However you do have influence over the xpm ecosystem, so there is very much something that could be done about it. :)

...

Anyway it is still not clear to me why cross files cannot simply contain the .cmd extension in them.

For meson, although this ticket reported a failure during the automatic toolchain identification, the problem can also occur when the meson scripts call any third party tools which were installed as npm/xpm dependencies

Unless explicitly overridden by cross files, meson will call any third party tools searched for via e.g. find_program(), by using shutil.which to locate them which does resolve .cmd extensions.

So I think this should probably be fine. Unfortunately, running all processes via cmd.exe is not viable, since ninja itself launches all commands with CreateProcess to reduce the expensive overhead of Windows processes.

[xclaesse]

Since Windows does not have a mechanism like #!/usr/bin/env bash to automatically identify the program to run a script, invoking such shims require an explicit shell invocation at application level, when child processes are created.

To summarise, on Windows there is not much that can be done, the shims are called .cmd and .ps1 and must be invoked via cmd.exe or PowerShell.

Meson already do that for *.py files. If they have "python" in their shebang Meson will invoke them via python.exe on Windows. Doing similar trick for .cmd and .ps1 would make sense IMHO. But I thought python's subprocess API was doing that for us already.

[xclaesse]

Also, I think you should instead set PATH pointing to your toolchain directory instead of using shims, because invoking your toolchain via a cmd/ps1 wrapper will make the build significantly slower.

[eli-schwartz]

Meson already does it for any files that have a shebang. The only thing that meson does with .py files specifically, is special case a shebang of "python" and use its own python.exe rather than invoking python.exe /path/to/foo.py, because meson knows it has a python available even if there isn't one on PATH.

[ilg-ul]

... the xpm ecosystem... ... you should instead set PATH ...

For consistency and compatibility reasons, the xpm ecosystem is part of the npm ecosystem, with good and bad.

So, for now, as long as npm uses shims and does not set multiple PATHs, xpm has to follow and do the same.