search

atlaste / CPPCoverage

Visual Studio Extension for measuring C++ Code Coverage and profiling data
78 stars 19 forks source link

Would this work with Node.js native modules? #35

Open cinderblock opened 3 years ago

cinderblock commented 3 years ago

In Node.js JavaScript, when one wants to link to native code, there are many build tools that compile C/C++ into native .node binaries that are imported/required by the Node.js JavaScript runtime.

I've been using GCC to compile projects with gcov enabled to get great coverage reports for Linux and macOS hosts. However Windows hosts are proving to be more reticent with MSBuild.exe doing the compilation.

https://github.com/serialport/node-serialport/pull/2314

I've been searching for a project that might help us generate coverage reports for Windows binaries. That this package has a free CLI tool gives me hope that integrating would be possible.

Looking into it more, it seems this tool is targeting being run as a Visual Studio extension, which I imagine gives easier GUI instructions on configuration. In our Node.js environment, node-gyp (a fork of Chrome's defunct build tool GYP) uses a file binding.gyp to descript required build flags and variations for alternate platforms. It looks like I'll need to try running Coverage-x64.exe in binding.gyp on Windows hosts, but I'm not seeing a 100% clear example of the commands needed to properly use this CLI tool.

I'm going to poke at this myself some more but I thought asking here early might save me some trouble.

Cheers.

atlaste commented 3 years ago

Will it work? I don't know. I see little reason why it wouldn't.

Basically the coverage application enumerates the pdb files for points that need to be covered. As long as they are there, it should be fine.

Just use '-help' on the coverage.exe for info on how to use it.

cinderblock commented 3 years ago

I can see that .pdb files are being generated in debug mode, however it's unclear that your coverage tool is finding those. With your confidence that it shouldn't really matter, I'll poke around with it some more.

Would this tool be sensitive to having an intermediate executable between it and the real one? It's very common for one node program to launch a subsequent node program. If the program I want to cover is a grandchild process of coverage.exe, will it still work?

cinderblock commented 3 years ago

So, I'm trying to run Coverage-x64.exe (and x86/d variants produce the same results), with a subsequent program that generates side effects (creates a file). The files are not created. No errors are displayed. So it seems my program is not executing. I'm not sure what I'm doing wrong.

C:\Users\camer\git\node-serialport>Coverage-x64d.exe -- npm run cov 
--- Arguments --- 
0: Coverage-x64d.exe
1: --
2: npm
3: run
4: cov
Executable: npm
Arguments: npm run cov
cinderblock commented 3 years ago

Ah ha! npm is not an executable on Windows. I needed to call npm.cmd. Now we're getting something.

.\Coverage-x64.exe -p packages\bindings -- npm.cmd run test-win

Still not getting any output in my .cov file.

I am seeing this in the output:

Loading: C:\Users\camer\git\node-serialport\packages\bindings\build\Debug\bindings.node... 
[Symbols loaded]

Which indicates that the .node binary that I care about is being loaded. It's not clear if it's finding the .pdb next to it.

I notice at the end of the log:

Gathering profile data of 0 sources...
Filtering post-process notifications...
Writing coverage report...done.

"0 sources...", checking the sources here, it looks like no profiling data was captured maybe?

I'm not seeing any similar issues to the above "0 sources".

cinderblock commented 3 years ago

Well, I've moved onto using OpenCppCoverage since it worked without issue pretty much out of the box on first try, had more clear debugging outputs/error messages, and has self explanatory cli options.

atlaste commented 3 years ago

Hmm that's a lot of ticket info I just noticed. I don't know why I don't get github notifications... Anyways, I get why it's frustrating, but please know that to my knowledge we haven't had any people running it for nodejs before. That being said, I see no reason why it shouldn't work.

Yes you can use OpenCPPCoverage, it works just fine for small projects; we used it in the past as well, but was in the end too simple for our purposes. If it does the job for you, by all means.

The coverage tool can also gather some performance profiling data by default, because we just get that for free ("profile data in the log") - but you might not need it. Anyhow, one reason is that we use a bit of code analysis (assembler) to get rid of things what C++ compilers incorrectly marks as "uncovered". I'm actually wondering if that gets in the way here. Child process loading and uploading works, I'm sure, because we use it with vstest - which runs C# code through C++, and then spawns another C++ process, while loading and unloading dll's...

Regardless, I'm still wondering why you don't get any data. If you get 'symbols loaded', it means that the PDB is picked up, the assembler data is processed and breakpoints were set.

The normal way we start Coverage from the VS IDE is as follows:

Coverage-x64.exe -o resultFile.cov -p "rootSolutionFolder" -w "workingFolder" -- [command line]

The solution folder is used as a filter for the debug symbols here. If the source files cannot be found as a child of the solution folder, they won't be used for coverage.

I'm no node.js expert by any means, but in your case, I guess it should be something like:

Coverage-x64.exe -o resultFile.cov -p "." -w "packages\bindings" -- npm.cmd run test-win

Could you try that and post the results please?

If it doesn't work, please try this, because it might give a hint where things go wrong:

Coverage-x64.exe -codeanalysis -o resultFile.cov -p "." -w "packages\bindings" -- npm.cmd run test-win

Stefan.

cinderblock commented 3 years ago

Thank you. The example commands were helpful.

-w doesn't seem to be correct. I'm running in my current directory. The C++ sources are in packages/bindings/src.

I had not seen any examples of -codeanalysis in any documentation. With that, I get slightly more info:

[...]
Loading: C:\Windows\System32\bcryptprimitives.dll... 
[No symbols available: The request is not supported.]
Loading: C:\Users\camer\git\node-serialport\packages\bindings\build\Debug\bindings.node... 
[Symbols loaded]
[0 breakpoints total, 0 are reachable]
Loading: C:\Windows\System32\ole32.dll... 
[No symbols available: The request is not supported.]
Loading: C:\Windows\System32\setupapi.dll... 
[No symbols available: The request is not supported.]
hello sent
hello received
Process exited with code: 0.
Gathering profile data of 0 sources...
Filtering post-process notifications...
Writing coverage report...done.

and resultFile.cov is created, but empty.

If you'd like to try this environment our yourself:

  1. Install Node.js
  2. Maybe optional: Install Windows build tools needed by node-gyp (MSVC & Python)
    • Automatic easy mode, that is marked deprecated, but idk: npm install --global windows-build-tools
  3. git clone --branch simple-test https://github.com/cinderblock/node-serialport.git
  4. cd node-serialport
  5. npm ci
  6. npm run rebuild -- -- --debug
  7. Simple test: Coverage-x64.exe -codeanalysis -o resultFile.cov -p "." -w "." -- node test.js
  8. Full test suite: Coverage-x64.exe -codeanalysis -o resultFile.cov -p "." -w "." -- npm.cmd test
atlaste commented 3 years ago

I think this has to do with that it is originally designed for c++, but I currently lack the time to do a proper deep analysis.

Let's just leave this issue open for now, until it can be properly resolved.

cinderblock commented 3 years ago

The sources I'm trying to cover are C++.