search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
28.46k stars 11.77k forks source link

check-all misses 3200+ compiler-rt tests in runtimes build #72511

Open rorth opened 11 months ago

rorth commented 11 months ago

As has been reported for a long time (probably since the LLVM 14 release cycle when the runtimes build was first made the reltest.sh default), building with -DLLVM_ENABLE_RUNTIMES=compiler-rt (which is the default these days) misses a large number of compiler-rt tests that are run with -DLLVM_ENABLE_PROJECTS=compiler-rt. The issue came up again in Issue #71971 and has apparently never been noticed during the development of the runtimes build (which doesn't speak particularly highly for the diligence of the testing at that point) and completely ignored despite the release tester reports since, massively reducing compiler-rt test coverage and missing all sorts of issues in new patches that would be noticed immedidately otherwise.

To provide detail, I've now run two x86_64-pc-linux-gnu builds, one projects build and one runtimes build. Here are the differences in overall test results from just that change (which should be a no-op for test results):

-Total Discovered Tests: 110511
-  Skipped          :     84 (0.08%)
-  Unsupported      :   3723 (3.37%)
-  Passed           : 106411 (96.29%)
-  Expectedly Failed:    292 (0.26%)
+Total Discovered Tests: 107217
+  Skipped          :     48 (0.04%)
+  Unsupported      :   3591 (3.35%)
+  Passed           : 103287 (96.33%)
+  Expectedly Failed:    290 (0.27%)
   Failed           :      1 (0.00%)

As one can see, more than 3200 tests are no longer run in a runtimes build. I've then extracted the names of the tests run from both build's ninja check-all output. Here are the test categrories that are missing in the runtimes build:

    836 AddressSanitizer-Unit
      3 Builtins
    215 Builtins-i386-linux
    215 Builtins-x86_64-linux
     70 GwpAsan-Unittest
     12 Interception-Unit
     58 LLVMFuzzer-Unittest
    194 SanitizerCommon-Unit
    275 ScudoStandalone-Unit
    275 ScudoStandalone-Unit-GwpAsanTorture
     79 ThreadSanitizer-Unit
     15 XRay-x86_64-linux       15 out of 29 tests turn UNSUPPORTED

In most cases, all tests from a category are missing, while for the xray tests about half of the tests changed from PASS to UNSUPPORTED.

I then looked for the missing tests: in the case of builtins, the lit.* files that controlled the tests used to live in

-test/builtins/lit.site.cfg.py
-test/builtins/Unit/I386LinuxConfig/lit.site.cfg.py
-test/builtins/Unit/X86_64LinuxConfig/lit.site.cfg.py

while they are now missing completely: they are neither present in runtimes-bins nor in builtins-bins where they should be after splitting the object directories for builtins from the rest of compiler-rt. Without the lit.* files, it's no wonder the builtins tests aren't run at all.

For the other missing tests, the lit.* files are present, but I don't yet see why the tests aren't run.

vitalybuka commented 11 months ago

Interesting. Can you please provide full cmake command? I recently, about month ago, switched our bots from DLLVM_ENABLE_PROJECTS to DLLVM_ENABLE_RUNTIMES and compared full verbose output. There were small lld related difference, which I fixed, but nothing big.

rorth commented 10 months ago

I just saw it when looking at a LLVM 17 release build with reltest.sh, which always does a runtimes build. When looking for Builtins-x86_64-linux :: divtc3_test.c, I came up blank.

The main builds I'm referring to above were 2-stage builds. The exact invocations are:

vitalybuka commented 10 months ago

I can reproduce this. DLLVM_ENABLE_RUNTIMES check-all is missing all/most unittests and all builtin tests.

However check-compiler-rt is missing only builtin tests.

On our sanitizer bots we run check-compiler-rt, and builtins are not in the scope of the group, so I probably was not concerned switching to DLLVM_ENABLE_RUNTIMES.

petrhosek commented 10 months ago

I have managed to reproduce this and I think I understand the issue, I'll try to come up with a solution.

petrhosek commented 10 months ago

I have confirmed that most of these will be addressed by #72955, the only ones that's not covered by that pull request are builtins tests which need a different solution.

rorth commented 8 months ago

When building main on sparc64-unknown-linux-gnu as a runtimes build in preparation for LLVM 18.1.0 rc1, I noticed that the builtins tests are still not run with check-all. I couldn't even find any target to run them at all.

rorth commented 2 months ago

Unfortunately, the lack of check-builtins support is still an issue after 9 months. IIUC this is (or isn't) handled in the terrible maze that is llvm/runtimes/CMakeLists.txt: while the runtimes support in there does add check-runtimes* targets, the (sort of) equivalent builtins support does not.