search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
28.28k stars 11.68k forks source link

[llvm-cov][MC/DC][Qualification] Too High MCDC coverage for C++ #109940

Open escherle-validas opened 1 week ago

escherle-validas commented 1 week ago

Too High MCDC coverage for C++

Criticality: HIGH

During qualification of MCDC coverage at Validas we found that the computation of MCDC coverage for the term "if ((TRUE ||TRUE) && v2)" is too high. The result is 100%, but should be 33% (or 50% or 66% depending on the interpretation of folded constants). This is highly critical, since tester might think they have covered everything and do not optimize the code. When we run the same example using Rust, we see 50% coverage, which is different and safe.

Rust example: constant_folding_rust Source Code and generated Reports: Test_000005.zip

C++ example: constant_folding_cpp Source Code and generated Reports: Test_000005.zip

evodius96 commented 1 week ago

This may be addressed by: https://github.com/llvm/llvm-project/pull/94137

In short, with Rust, there is a desire for more granular results for constant conditions, including those conditions that are uncoverable or unreachable and whether they ought to be included in the metrics. This is presently not the default for standard clang, in which constant conditions are simply ignored, which also corresponds to the behavior of other vendors. With the above pull request, this can be changed for clang.

escherle-validas commented 1 week ago

For me it is not clear why C++ and Rust should be handled differently here.

Furthermore, it is task of MCDC coverage to find weaknesses in the Code and Constants in the term might be a problem. In this case there should be reported a coverage value less than 100 % MCDC.

evodius96 commented 6 days ago

HI @escherle-validas MC/DC for Rust is presently a work-in-progress with some issues outstanding. What I can say is that clang folkds the conditions for C/C++, and folded conditions are not included in the metrics because they aren't influenced by variable test inputs. In looking at this further, I don't think this is the same issue as is addressed by https://github.com/llvm/llvm-project/pull/94137 which deals with non-constant conditions that may be uncoverable or unreachable due to constant conditions.

But if there is a desire to include constant folded conditions in the metrics, it seems feasible that could be added as an option.