Cannot find a project file to assign the valgrind error when using meson

bartlibert commented 5 years ago

Description

I have a moderately complex meson setup that is reused by several projects and I find the valgrind (memcheck) results are not fed into sonarqube, giving the error "Cannot find a project file to assign the valgrind error".

I cannot share everything because some proprietary code is involved, but I'll try to sketch the situation as well as possible.

Repository layout

Main repository is in /home/username/repo/git/projectname
Sources are in subdirectories /home/username/repo/git/projectname/src
Includes are in subdirectories of /home/username/repo/git/projectname/include
Unit tests are in subdirectories of /home/username/repo/git/projectname/units
Builds are made in /home/username/repo/git/projectname/build

Sonar properties file

Located at /home/username/repo/git/projectname/build/meson_tools/sonarqube

Contents:

sonar.projectKey=projecname
sonar.projectName=projectname
sonar.projectVersion=3c9295e9+
sonar.sources=/home/username/repo/git/projectname/src,/home/username/repo/git/projectname/include,/home/username/repo/git/projectname/unit
sonar.cxx.valgrind.reportPath=/home/username/repo/git/projectname/build/meson_tools/sonarqube/memcheck.xml
sonar.cxx.includeDirectories=/home/username/repo/git/projectname/include,/home/username/repo/git/projecname/build,/usr/include/c++/8,/usr/include/x86_64-linux-gnu/c++/8,/usr/include/c++/8/backward,/usr/lib/gcc/x86_64-linux-gnu/8/include,/usr/lib/gcc/x86_64-linux-gnu/8/include-fixed,/usr/include/x86_64-linux-gnu,/usr/include
sonar.projectBaseDir=/home/username/repo/git/projectname
sonar.sourceEncoding=UTF-8

Valgrind

Valgrind output is generated in /home/username/repo/git/projectname/meson_tools/sonarqube/memcheck.xml
Paths in the xml are relative to the /home/username/repo/git/projectname

Sample valgrind error frame output

<error>
  <unique>0x0</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>23 bytes in 1 blocks are definitely lost in loss record 1 of 9</text>
    <leakedbytes>23</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x483577F</ip>
      <obj>/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>malloc</fn>
      <dir>./coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>299</line>
    </frame>
    <frame>
      <ip>0x5136FF9</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.28.so</obj>
      <fn>strdup</fn>
      <dir>/build/glibc-d2N3Ld/glibc-2.28/string</dir>
      <file>strdup.c</file>
      <line>42</line>
    </frame>
    <frame>
      <ip>0x57BBF0</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>PdfFormFixture::getTestPixelColour(PoDoFo::PdfMemDocument*)</fn>
      <dir>unit/utils</dir>
      <file>test_pdfForm.cpp</file>
      <line>177</line>
    </frame>
    <frame>
      <ip>0x572F46</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0::test()</fn>
      <dir>unit/utils</dir>
      <file>test_pdfForm.cpp</file>
      <line>366</line>
    </frame>
    <frame>
      <ip>0x577545</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::TestInvokerAsMethod&lt;(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0&gt;::invoke() const</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>823</line>
    </frame>
    <frame>
      <ip>0x326A82</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::TestCase::invoke() const</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>11742</line>
    </frame>
    <frame>
      <ip>0x31EAB6</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::RunContext::invokeActiveTestCase()</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>10601</line>
    </frame>
    <frame>
      <ip>0x31E66E</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::RunContext::runCurrentTest(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;&amp;)</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>10574</line>
    </frame>
    <frame>
      <ip>0x31C8FB</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::RunContext::runTest(Catch::TestCase const&amp;)</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>10344</line>
    </frame>
    <frame>
      <ip>0x32085D</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::(anonymous namespace)::runTests(std::shared_ptr&lt;Catch::Config&gt; const&amp;)</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>10903</line>
    </frame>
    <frame>
      <ip>0x322707</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::Session::runInternal()</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>11098</line>
    </frame>
    <frame>
      <ip>0x32239E</ip>
      <obj>/home/username/repo/git/projectname/build/units</obj>
      <fn>Catch::Session::run()</fn>
      <dir>unit/include/lib</dir>
      <file>catch.hpp</file>
      <line>11055</line>
    </frame>
  </stack>
</error>

Sonar log snippet

11:53:21.074 INFO: Sensor C++ (Community) ValgrindSensor [cxx]
11:53:21.074 INFO: Searching reports by relative path with basedir '/home/username/repo/git/projectname' and search prop 'sonar.cxx.valgrind.reportPath'
11:53:21.074 DEBUG: Normalized report includes to '[/home/username/repo/git/projectname/build/meson_tools/sonarqube/memcheck.xml]'
11:53:21.074 DEBUG: Scanner uses normalized report path(s): '/home/username/repo/git/projectname/build/meson_tools/sonarqube/memcheck.xml'
11:53:21.075 INFO: Parser will parse '1' report file(s)
11:53:21.075 INFO: Processing report '/home/username/repo/git/projectname/build/meson_tools/sonarqube/memcheck.xml'
11:53:21.075 DEBUG: Parsing 'Valgrind' format
11:53:21.082 WARN: Cannot find a project file to assign the valgrind error 'ValgrindError [kind=Leak_DefinitelyLost, text=23 bytes in 1 blocks are definitely lost in loss record 4 of 9, stacks=[ ValgrindStack=[0x483577F: malloc (vg_replace_malloc.c:299)
0x5136FF9: strdup (strdup.c:42)
0x57BBF0: PdfFormFixture::getTestPixelColour(PoDoFo::PdfMemDocument*) (test_pdfForm.cpp:177)
0x573AA0: (anonymous namespace)::____C_A_T_C_H____T_E_S_T____0::test() (test_pdfForm.cpp:427)
0x577545: Catch::TestInvokerAsMethod<(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0>::invoke() const (catch.hpp:823)
0x326A82: Catch::TestCase::invoke() const (catch.hpp:11742)
0x31EAB6: Catch::RunContext::invokeActiveTestCase() (catch.hpp:10601)
0x31E66E: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:10574)
0x31C8FB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:10344)
0x32085D: Catch::(anonymous namespace)::runTests(std::shared_ptr<Catch::Config> const&) (catch.hpp:10903)
0x322707: Catch::Session::runInternal() (catch.hpp:11098)
0x32239E: Catch::Session::run() (catch.hpp:11055)] ] ]' to
11:53:21.082 WARN: Cannot find a project file to assign the valgrind error 'ValgrindError [kind=Leak_DefinitelyLost, text=23 bytes in 1 blocks are definitely lost in loss record 1 of 9, stacks=[ ValgrindStack=[0x483577F: malloc (vg_replace_malloc.c:299)
0x5136FF9: strdup (strdup.c:42)
0x57BBF0: PdfFormFixture::getTestPixelColour(PoDoFo::PdfMemDocument*) (test_pdfForm.cpp:177)
0x572F46: (anonymous namespace)::____C_A_T_C_H____T_E_S_T____0::test() (test_pdfForm.cpp:366)
0x577545: Catch::TestInvokerAsMethod<(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0>::invoke() const (catch.hpp:823)
0x326A82: Catch::TestCase::invoke() const (catch.hpp:11742)
0x31EAB6: Catch::RunContext::invokeActiveTestCase() (catch.hpp:10601)
0x31E66E: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:10574)
0x31C8FB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:10344)
0x32085D: Catch::(anonymous namespace)::runTests(std::shared_ptr<Catch::Config> const&) (catch.hpp:10903)
0x322707: Catch::Session::runInternal() (catch.hpp:11098)
0x32239E: Catch::Session::run() (catch.hpp:11055)] ] ]' to
11:53:21.082 WARN: Cannot find a project file to assign the valgrind error 'ValgrindError [kind=Leak_DefinitelyLost, text=23 bytes in 1 blocks are definitely lost in loss record 2 of 9, stacks=[ ValgrindStack=[0x483577F: malloc (vg_replace_malloc.c:299)
0x5136FF9: strdup (strdup.c:42)
0x57BBF0: PdfFormFixture::getTestPixelColour(PoDoFo::PdfMemDocument*) (test_pdfForm.cpp:177)
0x573392: (anonymous namespace)::____C_A_T_C_H____T_E_S_T____0::test() (test_pdfForm.cpp:383)
0x577545: Catch::TestInvokerAsMethod<(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0>::invoke() const (catch.hpp:823)
0x326A82: Catch::TestCase::invoke() const (catch.hpp:11742)
0x31EAB6: Catch::RunContext::invokeActiveTestCase() (catch.hpp:10601)
0x31E66E: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:10574)
0x31C8FB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:10344)
0x32085D: Catch::(anonymous namespace)::runTests(std::shared_ptr<Catch::Config> const&) (catch.hpp:10903)
0x322707: Catch::Session::runInternal() (catch.hpp:11098)
0x32239E: Catch::Session::run() (catch.hpp:11055)] ] ]' to
11:53:21.083 WARN: Cannot find a project file to assign the valgrind error 'ValgrindError [kind=Leak_DefinitelyLost, text=23 bytes in 1 blocks are definitely lost in loss record 3 of 9, stacks=[ ValgrindStack=[0x483577F: malloc (vg_replace_malloc.c:299)
0x5136FF9: strdup (strdup.c:42)
0x57BBF0: PdfFormFixture::getTestPixelColour(PoDoFo::PdfMemDocument*) (test_pdfForm.cpp:177)
0x5736D3: (anonymous namespace)::____C_A_T_C_H____T_E_S_T____0::test() (test_pdfForm.cpp:399)
0x577545: Catch::TestInvokerAsMethod<(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0>::invoke() const (catch.hpp:823)
0x326A82: Catch::TestCase::invoke() const (catch.hpp:11742)
0x31EAB6: Catch::RunContext::invokeActiveTestCase() (catch.hpp:10601)
0x31E66E: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:10574)
0x31C8FB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:10344)
0x32085D: Catch::(anonymous namespace)::runTests(std::shared_ptr<Catch::Config> const&) (catch.hpp:10903)
0x322707: Catch::Session::runInternal() (catch.hpp:11098)
0x32239E: Catch::Session::run() (catch.hpp:11055)] ] ]' to
11:53:21.083 WARN: Cannot find a project file to assign the valgrind error 'ValgrindError [kind=Leak_DefinitelyLost, text=23 bytes in 1 blocks are definitely lost in loss record 5 of 9, stacks=[ ValgrindStack=[0x483577F: malloc (vg_replace_malloc.c:299)
0x5136FF9: strdup (strdup.c:42)
0x57BBF0: PdfFormFixture::getTestPixelColour(PoDoFo::PdfMemDocument*) (test_pdfForm.cpp:177)
0x573D66: (anonymous namespace)::____C_A_T_C_H____T_E_S_T____0::test() (test_pdfForm.cpp:441)
0x577545: Catch::TestInvokerAsMethod<(anonymous namespace)::____C_A_T_C_H____T_E_S_T____0>::invoke() const (catch.hpp:823)
0x326A82: Catch::TestCase::invoke() const (catch.hpp:11742)
0x31EAB6: Catch::RunContext::invokeActiveTestCase() (catch.hpp:10601)
0x31E66E: Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (catch.hpp:10574)
0x31C8FB: Catch::RunContext::runTest(Catch::TestCase const&) (catch.hpp:10344)
0x32085D: Catch::(anonymous namespace)::runTests(std::shared_ptr<Catch::Config> const&) (catch.hpp:10903)
0x322707: Catch::Session::runInternal() (catch.hpp:11098)
0x32239E: Catch::Session::run() (catch.hpp:11055)] ] ]' to
11:53:21.083 DEBUG: CXX-VALGRIND processed = 0
11:53:21.083 INFO: CXX-VALGRIND processed = 0

guwirth commented 5 years ago

Hi @bartlibert,

I don't use the sensor so I can only guess:

The source code is here: https://github.com/SonarOpenCommunity/sonar-cxx/blob/master/cxx-sensors/src/main/java/org/sonar/cxx/sensors/valgrind/CxxValgrindSensor.java#L77
As you can see you get the message when the last frame from the stack is not in own code. Own means it must be a file which is below root and indexed.

In your sample it seams that files are from a library or in test code?

Regards,

bartlibert commented 5 years ago

Hi @guwirth, I already had a look at the code and I assumed that the "getLastOwnFrame" call would try to find a file within the source folder(s), hence it should find test_pdfForm.cpp and try to assign to that. Am I interpreting this wrong and is "getLastOwnFrame" badly named?

guwirth commented 5 years ago

Hi @bartlibert,

is test_pdfForm.cpp in/below source or test?

I'm not sure but test files are not handled as source files?
Please verify in the LOG file that file test_pdfForm.cpp is indexed.

Regards,

bartlibert commented 5 years ago

It is inside the unit folder (so the tests), which is also in the "sonar.sources" config variable, so I don't know if it should be handled differently? The file is indexed, although it has some syntax errors, related to the catch2 testing framework.

shua27 commented 3 years ago

tl;dr: Valgrind output MUST contain absolute paths for \<dir> entries and they MUST start with whatever you put as projectBaseDir.

@guwirth Been debugging this for a few days... had basically the same issue as @bartlibert and I think I finally found the issue.

Inside getLastOwnFrame we loop over every frame in the valgrind stack trying to find one that's "owned" by our project (Bart was right about that part):https://github.com/SonarOpenCommunity/sonar-cxx/blob/4f9280bd1d0a114606914733dd71e0c1b5a6d87d/cxx-sensors/src/main/java/org/sonar/cxx/sensors/valgrind/ValgrindStack.java#L91-L98

You might get this far and think the issue of "ownership" has to do with whether or not the file is indexed or being excluded for some reason but you'd be wrong in chasing that lead. Instead it's this nasty snippet:

https://github.com/SonarOpenCommunity/sonar-cxx/blob/4f9280bd1d0a114606914733dd71e0c1b5a6d87d/cxx-sensors/src/main/java/org/sonar/cxx/sensors/valgrind/ValgrindStack.java#L38-L40

path.startsWith(folder) is checking against baseDir, which in my case was the source of the issue because the valgrind report that I generated was actually generated from a binary that was created in another directory and this sensor (woefully) has no support for relative paths in valgrind output.

guwirth commented 3 years ago

Hi @shua27,

thanks for your feedback.

First of all I like to ask you to read:

Does this make sense for Valgrind?

path.startsWith(folder) is checking against baseDir, which in my case was the source of the issue because the valgrind report that I generated was actually generated from a binary that was created in another directory and this sensor (woefully) has no support for relative paths in valgrind output.

If you create the reports on another computer, you need to adjust the paths in the report files to match the paths on the computer running the scanner.

Or is the problem the “has no support for relative paths in valgrind output“?

Do you have a sample report file / snippet showing the issue? Or even better: a patch for the issue?

I’m not familiar with Valgrind and don’t have the possibility to make own tests… but we are for sure interested to improve the sensor.

Regards,

guwirth commented 2 years ago

To be fixed: … sensor (woefully) has no support for relative paths in valgrind output.

tienthanhnt commented 2 weeks ago

I also have same issue and fixed by recompiling program with option -g. Here is example: gcc -g -o use_after_free use_after_free.c. Include the -g flag for debugging symbols, which helps tools like Valgrind provide more useful output.

SonarOpenCommunity / sonar-cxx

Cannot find a project file to assign the valgrind error when using meson #1714

Description