Initial work towards a parallel dub building by supporting deferred logging

[rikkimax]

This introduces the deferred logging struct, unless enabled it should act identical to how it has in the past.

The idea is we would swap every function that ProjectGenerator.generate calls to use the DereferredLogState by ref instead of the global functions.

When parallelized we would turn on the deferment and once all the builds have completed for a given parallel set it would print them so that they will not have cross pollution between the builds.

[github-actions[bot]]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 14

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5331080 bin/dub
-rough build time=69s
+executable size=5367944 bin/dub
+rough build time=66s

Full build output

``` DUB version 1.34.0, built on Oct 15 2023 LDC - the LLVM D compiler (1.35.0): based on DMD v2.105.2 and LLVM 16.0.6 built with LDC - the LLVM D compiler (1.35.0) Default target: x86_64-unknown-linux-gnu Host CPU: znver3 http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai loongarch32 - 32-bit LoongArch loongarch64 - 64-bit LoongArch mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 spirv32 - SPIR-V 32-bit spirv64 - SPIR-V 64-bit systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/dub/dub/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/ldc2 for x86_64. Building dub 1.35.0-rc.1+commit.6.ge8a57d4f: building configuration [application] source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp` source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp` source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` Linking dub STAT:statistics (-before, +after) STAT:executable size=5367944 bin/dub STAT:rough build time=66s ```

[PetarKirov]

This design decision looks odd to me. I can't think of any build system (all of which build independent jobs in parallel) that defer logging until after a job is finished. Typically you'd have the output from different jobs running parallel prefixed with the name of the job and often also colorized differently to make the it easy to follow visually.

[PetarKirov]

E.g. nix build with nix-output-monitor, which I use almost every day: https://asciinema.org/a/7hJXH2iFLEkKxG1lL25lspqNn

[rikkimax]

That sounds like it has good tradeoffs too.

I'm happy to see that implemented, although I feel I've done my part for the initial work on parallelization.