Add coverage metrics for unit tests in CI

[lockshaw]

Add coverage metrics for the unit tests currently in CI in repo-refactor, most likely along the lines described here. If generating and viewing the code coverage output will require any additional commands from FF developers, document them in the PR.

Relevant files:

• CI workflow: https://github.com/flexflow/FlexFlow/blob/repo-refactor/.github/workflows/per-lib-check.yml
• cmake code can be found here. Try to avoid just dumping all of the cmake code in the top-level CMakeLists.txt as it makes it impossible to read
• third party dependencies (which would include any third-party cmake code) are managed via submodules in https://github.com/flexflow/FlexFlow/tree/repo-refactor/deps

Next steps:

• investigate integrating the output of code coverage with reviewable: https://docs.reviewable.io/repositories.html?highlight=coverage#code-coverage
• create a PR to https://github.com/lockshaw/proj to add a flag for coverage.

[Bob-Chen222]

Hi @lockshaw Colin, I am not sure what PR I need to make to https://github.com/lockshaw/proj because, based on my understanding of tutorial, I only need to add a few lines of code in the top-level CMakeLists.txt to enable the code coverage. (i.e. as long as adding flags when compile the whole project, I can somehow generate the code coverage report later)

After doing so, I will add some extra lines of code in the CI workflow to generate code coverage report. For example, add a script to run lcov -c -d . -o main_coverage.info \ > && genhtml main_coverage.info --output-directory out

And after that, figuring out ways to integrate output with reviewable.

Could you point out if there are any misunderstandings regarding the procedures? Thanks!

[lockshaw]

@Bob-Chen222 The main thing is to make sure that it's easy for developers to get the coverage information: if it's a matter of remembering to run lcov -c -d . -o main_coverage.info > && genhtml main_coverage.info --output-directory out in the right directory, tweak it a bit to find which libraries coverage you actually want to see, open a browser in the right output directory, etc. then in practice no one will use it. If it's a matter of running proj coverage and having a browser immediately open with the coverage metrics of the library being focused on, then it will be used.

tldr: the exact integration with proj, etc. doesn't really matter, the key is to make it incredibly easy for people not familiar with the system to get the coverage numbers and see them.

[Bob-Chen222]

I see @lockshaw. Could you tell me how to build the project? I want to build the project and actually generate coverage report to make sure that the implementation is correct but I am not very familiar with cmake and I got a lot of errors when I use something like cd _build && cmake ..

[lockshaw]

@Bob-Chen222 Without more information about what system/environment you're running on I can't provide more specific direction. One option that's pretty well-tested is to use nix (or nix-portable where you don't want to install system-wide) as done in https://github.com/flexflow/sapling-guide. It's also how the CI environment on repo-refactor is configured. Happy to meet over zoom etc. if you want some help getting things set up.

[lockshaw]

Closed by #1396 and #1380