dlang / dub

Package and build management system for D
MIT License
673 stars 230 forks source link

add the build ID in TargetDescription #2644

Closed rtbo closed 1 year ago

rtbo commented 1 year ago

Allow 3rd party programs (e.g. Meson) to locate artifacts in the cache. Following @Geod24 comment in #2642.

@WebFreak001 @Geod24 This would be enough for Meson to locate Dub's artifacts. If you prefer to use this proposal, feel free to revert #2642.

github-actions[bot] commented 1 year ago

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 14

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5280256 bin/dub
-rough build time=77s
+rough build time=76s
Full build output ``` DUB version 1.32.1, built on Jul 23 2023 LDC - the LLVM D compiler (1.33.0): based on DMD v2.103.1 and LLVM 15.0.7 built with LDC - the LLVM D compiler (1.33.0) Default target: x86_64-unknown-linux-gnu Host CPU: icelake-server http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/dub/dub/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/ldc2 for x86_64. Building dub 1.34.0+commit.14.gf621c89c: building configuration [application] source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp` source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp` source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` Linking dub STAT:statistics (-before, +after) STAT:executable size=5280256 bin/dub STAT:rough build time=76s ```
rtbo commented 1 year ago

Instead of cacheBuildId, I could use cacheArtifactPath with the full path to the built artifact in the cache. It would avoid Meson to take any assumption about the internal cache structure of Dub i.e.

$dubhome/cache/$pkg/$version[/+$subpkg]/build/$buildId/$targetFileName

Even with the build Id available, if this structure changes, Meson will be lost again.

PetarKirov commented 1 year ago

A full path would be better, as Meson shouldn't depend on implementation details of Dub. As @Geod24 mentioned, dub describe is what other tools should use and if it doesn't contain all necessary information we should simply update it to fix that.

rtbo commented 1 year ago

There seem to be a bug in ProjectGenerator.configurePackages that prevent this PR to work as intended. For example, when I run dub describe eventcore, the target taggedalgebraic get versions from target eventcore, despite having no dependency towards eventcore:

                "versions": [
                    "EventcoreEpollDriver",
                    "Have_taggedalgebraic"
                ],

However when I run dub build eventcore, taggedalgebraic is not built (because static libs) and when I run dub build taggedalgebraic, it is built without the EventcoreEpollDriver version.

It means that dub describe is currently describing targets that can't actually be built.

This come down to this piece of code, which looks wrong to me. I don't understand why dependencies need versions from their parents.

        // 1. downwards inherits versions, debugVersions, and inheritable build settings
        static void configureDependencies(const scope ref TargetInfo ti, TargetInfo[string] targets, size_t level = 0)
        {
            // do not use `visited` here as dependencies must inherit
            // configurations from *all* of their parents
            logDebug("%sConfigure dependencies of %s, deps:%(%s, %)", ' '.repeat(2 * level), ti.pack.name, ti.dependencies);
            foreach (depname; ti.dependencies)
            {
                auto pti = &targets[depname];
                mergeFromDependent(ti.buildSettings, pti.buildSettings);
                configureDependencies(*pti, targets, level + 1);
            }
        }

        configureDependencies(*roottarget, targets);
WebFreak001 commented 1 year ago

This would be enough for Meson to locate Dub's artifacts. If you prefer to use this proposal, feel free to revert https://github.com/dlang/dub/pull/2642.

actually I think having both would be beneficial, as dub describe will only tell you about a single configuration, but you might want to know what everything in the folder is.

rtbo commented 1 year ago

When I completely remove the "downwards inherits versions etc." section from Dub's code, I still make valid builds. Do you know the purpose of this (dependencies receiving versions from their parents) ?

Right now dub build and dub describe are not fully compatible with each other.

rikkimax commented 1 year ago

Do you know the purpose of this (dependencies receiving versions from their parents) ?

It is used quite heavily, see bindbc projects that use it to specify the version of the library it needs to load.

Very useful.

rtbo commented 1 year ago

It is used quite heavily, see bindbc projects that use it to specify the version of the library it needs to load.

Ah yes, quite obvious. Anyway this PR is ready. It will be fully useful if the --deep flag I've proposed is merged.

rtbo commented 1 year ago

Hi, can this be merged before next Dub release? The change seems to have consensus. I'd like to use it to fix Meson support for Dub dependencies.