Questions about usage, as a novice user

[claremacrae]

Hi, I'm going to be doing a demo early next week, showing coverage in CLion... And I'm hoping to show this plugin's branch coverage...

So I really want to check my understanding of it. And I figured that others may find the answers useful too, so I hope it's OK if I create an issue to ask for info....

Background:

• I'm running the plugin in CLion 2020.3 - on macOS
• Compiler is:

  clang version 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d457092ed932768724a6f06)
  Target: x86_64-apple-darwin19.6.0
  Thread model: posix
  InstalledDir: /Users/clare/Documents/develop/tools/clang/clang+llvm-11.0.0-x86_64-apple-darwin/bin

• My CLion run-profile was set up by clicking CLion's Coverage button, and then allowing CLion to create a new configuration, with adjusted configuration, to enable coverage. This added these CMake args: -DCMAKE_CXX_FLAGS="-fprofile-instr-generate -fcoverage-mapping" -DCMAKE_C_FLAGS="-fprofile-instr-generate -fcoverage-mapping". This is really handy, as it saves me having to work out what options to add - CLion just takes care of it.

Questions

1. As far as I can tell, this plugin only updates its results display when I use CLion's Run button - and not when I do Debug or Coverage. Is that correct? (It makes it a big complicated to explain how to set up - as I think I have to say "Click to Coverage button to get CLion to set up the build config - but after that, use the Run button, to get branch coverage)
2. I'm happy that I've collected enough coverage data now, and I want to go back to developing, with the normal CLion display. I've used Run -> "Hide Coverage" to hide CLion's coverage display. My screen looks like the following - how can I turn off this plugin's display of background colour and icons, without disabling the plugin and restarting CLion?

[zero9178]

Hello Claire!

I saw the announcement on the CLion blog and have registered already. Very much looking forward to it!

As to your questions: 1) There is two ways to tell the plugin to process coverage data at the moment. By default it will simply detect the compiler flags and then update if run. There is also the option in the settings (Settings -> Language & Frameworks -> C/C++ Coverage) the option to instead use a Run Coverage Action. This adds a separate button up in the run menu which looks like the coverage button from CLion but with a C++ Logo in the lower left corner.

2) After coverage has been gathered the toolwindow on the right called C/C++ Coverage should be populated and contain statistics for both region and branch coverage: You can sort the columns and also expand to check the coverage of functions. The Clear Button above should clear all line markings and branch coverage markers left by the plugin. If it doesn't then it's a bug.

I am aware that both issues are rather inconvenient, specifically since it duplicates functionality. The plugin predates any coverage functionality in CLion. After they added Coverage functionality they naturally used common APIs that are shared between all IntelliJ based IDEs, that last I checked does not support branch coverage nor the precise region coverage as the plugin does currently. I am personally a big fan of the precise region coverage clang generates where it's even possible to tell if an expression within a boolean expression has not been executed. CLions coverage even when using clang is a simple line coverage. I am also currently unaware of any way for me to hook into CLions APIs that would allow my plugin to replace it. For now I try my best to live alongside it.

Ideally CLion would someday fully incorporate the functionality of the plugin but until then I'll try to maintain the plugin to the best my time allows. While I won't be able to add major feature additions, I will try to stay up to date with CLion updates and EAPs as well as Compiler changes. On Windows not only MinGW but clang-cl is supported as well for example.

On that topic: Clang 12 will be capable of generating branch coverage as well so I will be trying to implement that as soon as possible. That will remove the expensive and time consuming post processing step after the executable has been run.

[claremacrae]

I saw the announcement on the CLion blog and have registered already. Very much looking forward to it!

Oh wow, that's great. Now you know why I was asking about coverage a few weeks ago... :-)

As to your questions: 1) There is two ways to tell the plugin to process coverage data at the moment. By default it will simply detect the compiler flags and then update if run. There is also the option in the settings (Settings -> Language & Frameworks -> C/C++ Coverage) the option to instead use a Run Coverage Action. This adds a separate button up in the run menu which looks like the coverage button from CLion but with a C++ Logo in the lower left corner.

I saw that settings page but ignored it - I think I assumed that it a CLion built-in thing...

I wonder if it would be worth putting some kind of wording in that Settings page to mention the plugin..

Perhaps something like:

Settings for "C/C++ Coverage" Plugin. For more information, see https:....

I turned on the Run Action, and I see it in the toolbar... When I click it, I get:

It seems that doing a clean build may have fixed that.

2. After coverage has been gathered the toolwindow on the right called C/C++ Coverage should be populated and contain statistics for both region and branch coverage: You can sort the columns and also expand to check the coverage of functions. The Clear Button above should clear all line markings and branch coverage markers left by the plugin. If it doesn't then it's a bug.

Brilliant - I think I didn't look closely at that view - assuming again it was CLion functionality... The Clear button works perfectly.

Thank you for all the other info too... this is really helpful.

This is the CLion ticket I'm tracking, for a better branch coverage implementation.... https://youtrack.jetbrains.com/issue/CPP-22202

I'll add a link there to this issue, so they are aware of all the useful info in it. Thank you!

[claremacrae]

[Continuing to abuse GitHub issues, for a general discussion - I hope that's OK]

Thinking about my demo, I was going to show downloading the plugin - but I'm already way over the time limit for the webinar... so I think it's probably better to point people at a step-by-step guide to the things you said above. And that guide can then be updated over time.

(I see now that some of them were covered in the README - but I'll try to have a think about if there's anything I could add to the wording to help people with the basics)

[zero9178]

Don't worry about using the issues page at all, communication is important!

After reading your feedback, is something like this what you had in mind?

I'd change the name on the left to indicate it's the Plugins settings and functionality and not CLions and I added a label with a hyperlink up top linking to the repository.

If you do find any odd wording or feel like there should be any clarifications in the README or elsewhere do feel free to make a pull request and I'll merge it in. While my English is pretty good, I am no native speaker so you may find odd wording.

[claremacrae]

Both those changes really help - thank you!

[claremacrae]

Hi Markus,

It's looking like the CLion demo might be more effective on Windows than Mac - but only if I can get effective code coverage working...

I am not sure from the README if this is expected to show branch coverage on Windows.

Can you advise on the steps involved, please, if it's expected to work on Windows?

[zero9178]

Supported compilers and needed tools on Windows are almost the same as on every other platform. Using GCC e.g. is 100% the same and also still requires gcov which is generally shipped by basically all GCC distributions on Windows.

The compiler you'll want to be using though is Clang as it's coverage is worlds better. Here you got two options. The most obvious and probably best suited option for most is clang-cl. It supports 100% of the same features, even with the same compiler flags. Problem is usually that many clang-cl distributions seem to not ship any of the required tools (llvm-cov and llvm-profdata). I am not sure about what tools Visual Studio installs along side clang-cl.

What does ship and work is clang-cl that you can download on https://llvm.org/ for Windows. Configuring it in CLion is quite easy; you simply need to create a Visual Studio compiler profile but then choose clang-cl as C and C++ Compiler. It should pick up the environment.

One subtle difference between clang-cl and other versions of clang is that since it uses the Microsoft ABI it has different name mangling. All other platforms use Itanium ABI which can be demangled using either c++filt or llvm-cxxfilt. When using clang-cl, llvm-undname is required as demangler. This is sadly not currently shipped by LLVMs binary distribution which would sadly lead to listing the mangled names in the statistics Window.

Alternatively to clang-cl and Visual Studio you could use a version of clang using MinGW as libc. I myself use and test against a build similar to this one: https://github.com/mstorsjo/llvm-mingw. Usage with it, as well as tools required are 100% identical to Unix systems.

Lastly, compiling clang on Windows yourself would also be an option to get all needed tooling and is what I do, but would require some familiarity with LLVMs CMake options.

MSVC does not offer any coverage functionality at all and is therefore not supported.

[claremacrae]

Thank you for that comprehensive answer. Time is tight, but I’ll give it a go!

[claremacrae]

Closing, as all my questions have been comprehensively answered - thank you!!!!