dlang / dub

Package and build management system for D
MIT License
673 stars 230 forks source link

Leave dependencies results in tmp dir if they can't get writen to their target directory #2713

Closed grillo-delmal closed 1 month ago

grillo-delmal commented 10 months ago

When you try to build an app with dub and one of their dependencies is located in a read-only directory, the whole process fails with the following message (when running with --vverbose):

Error <target file>: Read-only file system
Full exception: std.file.FileException@std/file.d(4332): <target file>: Read-only file system
----------------
??:? @trusted void std.file.copyImpl(scope const(char)[], scope const(char)[], scope const(char)*, scope const(char)*, std.typecons.Flag!("preserveAttributes").Flag) [0x7efe6d875c59]
source/dub/compilers/buildsettings.d:4251 [0x516daf]
source/dub/internal/vibecompat/core/file.d:78 [0x4d4b12]
source/dub/internal/vibecompat/core/file.d:193 [0x51722e]
source/dub/generators/build.d:509 [0x67f30c]
source/dub/generators/build.d:299 [0x67e01d]
source/dub/generators/build.d:234 [0x679e50]
source/dub/generators/build.d:174 [0x67aadc]
source/dub/generators/build.d:128 [0x67a3cf]
source/dub/generators/build.d:180 [0x678fa6]
source/dub/generators/generator.d:185 [0x4b798d]
source/dub/commandline.d:1463 [0x7b4891]
source/dub/commandline.d [0x7b4d5d]
source/dub/commandline.d:509 [0x6f9b64]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x7efe6d520b8c]
??:? _d_run_main2 [0x7efe6d5209a2]
??:? _d_run_main [0x7efe6d5207ec]
??:? [0x7efe6d17b149]
??:? __libc_start_main [0x7efe6d17b20a]
??:? [0x47d914]

This patch adds code to catch that exception and stop the dependency from being copied to the target directory, letting the build process to continue using the dependency results directly from the tmp-dir.

This might help to solve #1473

github-actions[bot] commented 10 months ago

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 14

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5331080 bin/dub
 rough build time=64s
Full build output ``` DUB version 1.34.0, built on Oct 15 2023 LDC - the LLVM D compiler (1.35.0): based on DMD v2.105.2 and LLVM 16.0.6 built with LDC - the LLVM D compiler (1.35.0) Default target: x86_64-unknown-linux-gnu Host CPU: znver3 http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai loongarch32 - 32-bit LoongArch loongarch64 - 64-bit LoongArch mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 spirv32 - SPIR-V 32-bit spirv64 - SPIR-V 64-bit systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/dub/dub/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/ldc2 for x86_64. Building dub 1.35.0-beta.1+commit.4.g9fb664fe: building configuration [application] source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp` source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp` source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals` source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` Linking dub STAT:statistics (-before, +after) STAT:executable size=5331080 bin/dub STAT:rough build time=64s ```
WebFreak001 commented 9 months ago

I don't think this will use the dependency from tmp-dir, right? It rather looks like it would just reuse the already present binary, if any, in that case and cause build failure due to file system issues to use old binaries, which is a no-go.

If the DUB packages are in a read-only filesystem, I think you need to rather specify --cache=local|system|user to the correct value. Ideally this should be possible to be settable using the dub settings.json, but isn't yet

grillo-delmal commented 9 months ago

I think you need to rather specify --cache=local|system|user to the correct value.

Even if you set the --cache parameter, running dub build will still fail if the dependency library sources are located in a read-only folder. This is because the process tries to move the resulting binaries from the build cache into the source folder during the build process (and thus why the exception is triggered). I have tested this on the use cases described in this repository: https://github.com/grillo-delmal/dub-system-wide-lib-tests .

What I'm trying to patch here is that, when the process correctly builds a dependency and fails to move the resulting binaries to the source code path, stop the process from crashing and continue the build process, skipping the part where the resulting dependency binaries are moved to the source directory.

In the piece of code I modified, the settings.tempBuild variable is only true if you are using dub run instead of dub build. The only thing that gets skipped by applying the patch is a call to updateCacheDatabase, which I assume its there to update the new built dependency path after it was moved correctly.

In this context, the only thing that I'm assuming is that if you don't call the updateCacheDatabase method, the build process will search for the built binary in the library build cache. I assume this considering that dub run works fine even if it doesn't need to run this code block. Please correct me if I'm wrong :)

Do you mean that, in the case where you have an older binary available in the source directory, the cache database will point to that file even if a new one was just built and is available in the build cache? Wouldn't this still be a problem if you try to run dub run without any changes?

edit: Trying to make sense x100. Sometimes even I can't understand what I write

grillo-delmal commented 1 month ago

I'll close this and approach it from another perspective