Subprojects are not included in tarball, when behind a disabled option

[evelikov]

Describe the bug

Consider a project where you have a handful of options, that pull in subprojects as dependencies. Some of those options are disabled by default.

Unless one explicitly enables all relevant options, meson dist --include-subprojects (with or w/o --no-tests) will not bundle some of the subprojects in the distribution tarball.

One can audit and enable each option during meson setup -D foo=true -D bar=enabled, although that is not always possible.

Consider the following example: The project builds on Linux, MacOS, Windows, etc and you are creating the tarball on one platform. Yet running meson setup -D windows=true -D linux=true -D macos=true fails, since not all the required dependencies are available. Hence you cannot create complete, self-contained distribution tarball.

To Reproduce

Note that my reproducer is a little less convoluted than the one described. It merely guards the tests (and hence cmocka subproject) behind a build-tests bool option.

• clone https://gitlab.freedesktop.org/mesa/waffle
• create the tarball meson setup builddir && meson dist --include-subprojects --no-tests -C builddir
• extract the tarball
• run meson setup --wrap-mode nodownload -D build-tests=true builddir
• observe the failure since cmocka is not bundled nor downloaded as per the --wrap-mode settings

Expected behavior

• meson processes all *.wrap files in the well-known subprojects/ folder
• and/or ultimately all the subprojects are included in the dist tarball

system parameters

• Is this a cross build or just a plain native build (for the same computer)? - Happens in both cases
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) - Arch, also happens on Ubuntu 22.04
• what Python version are you using e.g. 3.8.0 - 3.10.8
• what meson --version - 0.64.0
• what ninja --version if it's a Ninja build - 1.11.1

[xclaesse]

I guess we could keep --include-subprojects with no extra argument to only include subprojects that have been used for current setup, and extend the syntax to --include-subprojects foo,bar to only include foo and bar regardless of which subprojects have been used for the setup?

[xclaesse]

Note that the original use-case for --include-subprojects is to make a specific build reproductible. For example when configuring GStreamer for a specific device we want to be able to ship the source code of exactly what is being used on that device. It is for example required for legal reason.

[evelikov]

Thanks Xavier.

Is there a reason why meson cannot traverse across the subprojects directory and add them all?

AFAICT meson subprojects foreach already knows how to do that and is not limited solely to the subprojects pulled in as dependencies.

As you pointed out - this new functionality could be under a new name or toggle. Be that --include-subprojects --all or anything else the meson team is happy with.

[xclaesse]

@evelikov I guess we can add --include-subprojects * or all. PR welcome ;-)