Cache json db

[rtbo]

As discussed in #2640, proposal to write Json files in the cache for 3rd party software like Meson to assess compatibility of artifacts in the Dub cache. I've made the choice to write small Json files per package version rather than a gigantic one for the whole cache. This comes at the price of stabilizing the directory structure within the cache (up to where the json file is written).

The PR add the following files in the cache

~/.dub/cache/vibe-d/0.9.6/+http/build-db.json
~/.dub/cache/vibe-d/0.9.6/+http/build-db.lock

To ensure interoperability, the following stabilization apply:

~/.dub/cache/vibe-d/0.9.6/+http/build-db.json
~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      1             2

1. $DUB_HOME/cache default location is stable (env-vars and settings customization still apply)
2. Json file path relative to $DUB_HOME/cache is stable

The final artifact location remains up to Dub developers.

fixes #2640

[github-actions[bot]]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 15

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5280736 bin/dub
-rough build time=98s
+executable size=5284832 bin/dub
+rough build time=94s

Full build output

``` DUB version 1.31.1, built on May 12 2023 LDC - the LLVM D compiler (1.32.2): based on DMD v2.102.2 and LLVM 15.0.7 built with LDC - the LLVM D compiler (1.32.2) Default target: x86_64-unknown-linux-gnu Host CPU: skylake-avx512 http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/dub/dub/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/ldc2 for x86_64. Building dub 1.33.0-beta.1+commit.13.g88f50e80: building configuration [application] source/dub/internal/dyaml/node.d(2513,9): Deprecation: scope variable `this` assigned to non-scope parameter `_param_0` calling `match` source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp` source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp` source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): which would be `@system` because of: /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` Linking dub STAT:statistics (-before, +after) STAT:executable size=5284832 bin/dub STAT:rough build time=94s ```

[WebFreak001]

since you want to ensure that the output doesn't change, do you want to add a test in the tests folder that checks the file to match exactly using diff?

[rtbo]

since you want to ensure that the output doesn't change, do you want to add a test in the tests folder that checks the file to match exactly using diff?

The Json file contains the build ID, which varies with compiler version. So I can't predict in a test what the exact content is. At best I can check if all the Json fields are there and the value of the immutable ones.

[WebFreak001]

thanks, seems to be a well designed and useful feature!

[rtbo]

Thanks for the quick review !

[Geod24]

I've missed the discussion here, but why are we adding JSON files for other tools to depend on ? dub describe was meant as the way to do this, why is it not enough ?

[Geod24]

I'm not sure what "stabilization" means in this context, either. The commit just adds comments.

[rtbo]

dub describe doesn't give any info about cached builds. e.g. on my system

$ tree ~/.dub/cache/urld
/home/remi/.dub/cache/urld
└── 2.1.1
    ├── build
    │   ├── library-debug-cu9jbmcFOeHiBceMMo2VbA
    │   │   └── liburld.a
    │   └── library-debug-Z7qINYX4IxM8muBSlyNGrw
    │       └── liburld.a
    ├── db.json
    └── db.lock

Meson needs to know what these 2 archives are (compiler, compiler version etc.) in order to use them properly. Meson uses dub describe for everything else (import paths, sub-dependencies etc.), but the cache info is not available. I have to say that enriching dub describe output would have been a viable option as well (e.g. add the buildId).

[rtbo]

I'm not sure what "stabilization" means in this context, either. The commit just adds comments.

What is stable is what should not be changed in future Dub versions.

[Geod24]

What is stable is what should not be changed in future Dub versions.

The only thing that should not be changed in future dub version is whatever is part of the API. And the API to access dub's internal data is dub describe, hence my comment.

I have to say that enriching dub describe output would have been a viable option as well (e.g. add the buildId).

I think that might be a better option. But I need to do some reading on the issue(s).