dlang / dub

Package and build management system for D
MIT License
677 stars 227 forks source link

Add betterC probing for performance and for applications without drun… #2753

Closed etcimon closed 10 months ago

etcimon commented 10 months ago

This improves performance and also allows me to compile a custom druntime with my application. (post-switches = [] in the ldc2.conf)

github-actions[bot] commented 10 months ago

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 11

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5371736 bin/dub
-rough build time=62s
+executable size=5375832 bin/dub
+rough build time=63s
Full build output ``` DUB version 1.35.1, built on Jan 6 2024 LDC - the LLVM D compiler (1.36.0): based on DMD v2.106.1 and LLVM 17.0.6 built with LDC - the LLVM D compiler (1.36.0) Default target: x86_64-unknown-linux-gnu Host CPU: znver3 http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai loongarch32 - 32-bit LoongArch loongarch64 - 64-bit LoongArch mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 spirv32 - SPIR-V 32-bit spirv64 - SPIR-V 64-bit systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/dub/dub/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/ldc2 for x86_64. Building dub 1.36.0-beta.1+commit.30.ga4abc842: building configuration [application] /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/algorithm/iteration.d(475,42): Deprecation: function `dub.dub.getRegistryPackageSupplier` is deprecated - This function wasn't intended for public use - open an issue with Dub if you need it source/dub/dub.d(67,57): instantiated from here: `map!(immutable(string)[])` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/algorithm/iteration.d(582,19): Deprecation: function `dub.dub.getRegistryPackageSupplier` is deprecated - This function wasn't intended for public use - open an issue with Dub if you need it /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/algorithm/iteration.d(479,16): instantiated from here: `MapResult!(getRegistryPackageSupplier, immutable(string)[])` source/dub/dub.d(67,57): instantiated from here: `map!(immutable(string)[])` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/algorithm/iteration.d(594,23): Deprecation: function `dub.dub.getRegistryPackageSupplier` is deprecated - This function wasn't intended for public use - open an issue with Dub if you need it /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/algorithm/iteration.d(479,16): instantiated from here: `MapResult!(getRegistryPackageSupplier, immutable(string)[])` source/dub/dub.d(67,57): instantiated from here: `map!(immutable(string)[])` source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/write.d(537,34): which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec` /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): which wouldn't be `@safe` because of: /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/../import/std/format/spec.d(258,33): scope variable `this` assigned to non-scope parameter `e` calling `put` Linking dub STAT:statistics (-before, +after) STAT:executable size=5375832 bin/dub STAT:rough build time=63s ```
rikkimax commented 10 months ago

Duplicate of: https://github.com/dlang/dub/pull/2699

etcimon commented 10 months ago

My solution involves minor changes and reverts to old behaviour, easier to approve.

Geod24 commented 10 months ago

Just to be sure I follow: Do we need the -betterC flag if we make it -betterC-compliant ? As in, if you get it to a point it does not pull anything from the runtime, the flag becomes superfluous, doesn't it ?

rikkimax commented 10 months ago

Just to be sure I follow: Do we need the -betterC flag if we make it -betterC-compliant ? As in, if you get it to a point it does not pull anything from the runtime, the flag becomes superfluous, doesn't it ?

No, it doesn't.

You will still be linking against druntime, and that has cost.

etcimon commented 10 months ago

The probePlatform routine takes ~25ms (vs ~38ms previously) for me using a StopWatch when building with these improvements. It's now also compatible with projects that don't include druntime.

If I benchmark only the first run on windows arch x86_64, it goes from 52ms to 24ms

etcimon commented 10 months ago

On dmd removing the -betterC flag improves it to 18ms, but with ldc it worsens it to 40ms.

Geod24 commented 10 months ago

You will still be linking against druntime, and that has cost.

Why do we need to link ?

etcimon commented 10 months ago

It wouldn't link, it doesn't write an object file, compile flag -o-. It's mostly dmd.dinterpret in the compiler doing most of the work and returning after.

Geod24 commented 10 months ago

It's mostly dmd.dinterpret in the compiler doing most of the work and returning after.

So why is -betterC relevant here ? That's what I'm confused about. I would like to solve your issue, but I'm not sure I fully understand it.

etcimon commented 10 months ago

Originally it's because this project https://github.com/etcimon/libwasm implements its own druntime and for it you must remove the includes in ldc2.conf, but probing fails on that configuration if it depends on druntime.

Geod24 commented 10 months ago

but probing fails on that configuration if it depends on druntime.

I assume that probing fails because it tries to import druntime, rather than link it ?

etcimon commented 10 months ago

It complains that there is no module object

Geod24 commented 10 months ago

@kinke : Any chance you could have a look at this ? I think the use case is good, but I would like to make sure it keeps on working with newer versions of DMD/LDC with minimal maintenance.

etcimon commented 10 months ago

So why is -betterC relevant here ?

It works without -betterC, but it does help with both performance and guarantees.

etcimon commented 10 months ago

Are we waiting on @kinke for a follow-up?

kinke commented 10 months ago

AFAICT after a quick glance, the main change here is that the generated probe module becomes the special object module, only defining the string alias (not even size_t), so that the (fat) druntime one isn't imported. -betterC then shouldn't really make a difference, as its effects should be enabled automatically without TypeInfo class etc. in object.

While such a minimalistic, fully self-contained D module is surely fast to analyze with -o- and working for ~all exotic targets too, the cost is the ugly duplication in source/dub/platform.d. I guess that could be improved though by post-processing the existing string, e.g., replacing ret ~= <literal>; by pragma(msg, ´ <literal>´); and stripping off string[] ret; and return ret;.

etcimon commented 10 months ago

I guess that could be improved though by post-processing the existing string

The post-processing using regex, matchAll and replace makes it impossible to evaluate the string with ctfe, but I doubt this would make it slower.

etcimon commented 10 months ago

nvm I got it working in ctfe

etcimon commented 10 months ago

Should I squash it?