Add runtests subcommand

shoo commented 3 months ago

As with many official tools, projects often require integration testing for CI/CD. In such cases, it is useful to run integration tests in a unified format. The runtests subcommand I added is similar to test/run-unittest.sh or test/run-unittest.d so integration tests of dub projects can be run dub itself. To test dub, run the following command:

dub run -- runtests --skip-build

github-actions[bot] commented 3 months ago

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 8

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5293368 bin/dub
-rough build time=62s
+executable size=5364928 bin/dub
+rough build time=63s

Full build output

``` DUB version 1.38.0, built on Jul 4 2024 LDC - the LLVM D compiler (1.39.0): based on DMD v2.109.1 and LLVM 18.1.6 built with LDC - the LLVM D compiler (1.39.0) Default target: x86_64-unknown-linux-gnu Host CPU: znver3 http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai loongarch32 - 32-bit LoongArch loongarch64 - 64-bit LoongArch mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 spirv - SPIR-V Logical spirv32 - SPIR-V 32-bit spirv64 - SPIR-V 64-bit systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/dub/dub/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.39.0/x64/ldc2-1.39.0-linux-x86_64/bin/ldc2 for x86_64. Building dub 1.39.0-beta.1+commit.9.gc8c3cbb0: building configuration [application] source/dub/internal/dyaml/composer.d(210,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/composer.d(232,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/composer.d(336,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/event.d(196,5): Deprecation: cannot access overlapped field `Event.explicitDocument` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/event.d(214,5): Deprecation: cannot access overlapped field `Event.explicitDocument` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/event.d(241,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/event.d(148,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code source/dub/internal/dyaml/event.d(148,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code Linking dub STAT:statistics (-before, +after) STAT:executable size=5364928 bin/dub STAT:rough build time=63s ```

rikkimax commented 3 months ago

Ideally these two would be merged together by looking for the json/sdl comment at start of file.

- `*.d` : The D language source code is executed by rdmd.
- `*.script.d` : Runs the D language source code as a file as a single-file DUB project.

shoo commented 3 months ago

Ideally these two would be merged together by looking for the json/sdl comment at start of file.
- `*.d` : The D language source code is executed by rdmd.
- `*.script.d` : Runs the D language source code as a file as a single-file DUB project.
Although not impossible, only the minimum necessary functions have been implemented in this PR. This change would be able to be enhanced later without breaking changes.

Here is a list of others that I can think of that would be nice to have but have not yet been implemented:

.fail_build / .no_build / .no_test / .no_run / .min_frontend / .testignore / --testdir These could be better configured, for example integrated into the project settings.
When running tests, it would be useful to have an environment variable that contains the path to the binary file under test.
A way to output the test results in CSV format, etc., for easier handling by other tools would be helpful.
It would be useful to have an option to specify the directory where the coverage results are stored.
An option to run each test in parallel may be useful to speed up the process.

If you can think of any very important enhancements, glitches, or future enhancements that might require breaking changes, please let me know.

rikkimax commented 3 months ago

Support directory based dub projects.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

shoo commented 3 months ago

Support directory based dub projects.

If you are talking about test cases, they are already implemented. The added test test/runtests/test/test01 is a directory based dub project.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

It is difficult to detect with certainty which directory contains test cases. For example, dub includes three directories (test, examples, scripts) with similar file characteristics, but it is difficult to mechanically identify which directory is for integration testing. Furthermore, if this were to be automatically deduced, it would be confusing to the user, unless it were deduced with fairly simple logic.

rikkimax commented 3 months ago

Support directory based dub projects.

If you are talking about test cases, they are already implemented. The added test test/runtests/test/test01 is a directory based dub project.

No, I'm talking about test/runtests/dub.sdl.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

It is difficult to detect with certainty which directory contains test cases. For example, dub includes three directories (test, examples, scripts) with similar file characteristics, but it is difficult to mechanically identify which directory is for integration testing. Furthermore, if this were to be automatically deduced, it would be confusing to the user, unless it were deduced with fairly simple logic.

I didn't suggest automatic detection. I suggested a directive in your dub file to specify the directory to use.

shoo commented 3 months ago

No, I'm talking about test/runtests/dub.sdl .

More information, please. Are you saying that you want to recognize the directory as being for the dub project without the dub.sdl?

I didn't suggest automatic detection. I suggested a directive in your dub file to specify the directory to use.

I see. However, I considered this but decided not to include it in this PR. I believe that adding dub project configuration directives with a higher priority than the default but lower than command line arguments can be added in a future update, not this time.

rikkimax commented 3 months ago

No, I'm talking about test/runtests/dub.sdl .

More information, please. Are you saying that you want to recognize the directory as being for the dub project without the dub.sdl?

Below the tests directory, below that is either files that you're running. I would like it to also support dub projects at that level.

It could be a misunderstanding on my part, but that changelog doesn't imply it.

shoo commented 3 months ago

Below the tests directory, below that is either files that you're running. I would like it to also support dub projects at that level.

It could be a misunderstanding on my part, but that changelog doesn't imply it.

When you run dub runtests at the root of a dub project, it will execute directory based dub projects contained within the test directory. This behavior should align with your expectations. However, in the case of the test/runtests directory, it is excluded from the tests because it contains a .no_build file. It might be confusing, but the test case I created runs test/runtests.script.d with test/runtests as the current directory. This test executes the test cases in test/runtests/test/* by running the dub runtests command.

Geod24 commented 3 months ago

First thing that comes to mind: We already have a special configuration for unittest, why not do that for integration tests ? Why a special command ?

shoo commented 3 months ago

First thing that comes to mind: We already have a special configuration for unittest, why not do that for integration tests ? Why a special command ?

The idea of having a special configuration selected when running runtests subcommand might be a good one. By having a directive to replace --testdir, such as allowing --config to switch test directories, we could achieve good synergy. On the other hand, I still think a special subcommand is necessary. Just as there is a dub test subcommand instead of dub run --config unittest -- --DRT-testmode=test-only .

dlang / dub

Add runtests subcommand #2952