search

near / near-one-project-tracking

A repository for tracking work items that NEAR One is working on.
0 stars 0 forks source link

[ProjectTracking] Report code coverage for all our changes #12

Open Ekleog-NEAR opened 9 months ago

Ekleog-NEAR commented 9 months ago

Goals

Background

We have been having issues with debugging integration tests, that often fail without unit tests failing alongside them. In order to fix this, we need to increase our code coverage for unit tests specifically.

In addition, the project was kickstarted by the Hacken report, that was suggesting we implement some bare minimum of code coverage.

Why should NEAR One work on this

Nearcore should see a change in the way of thinking about tests. If each new change added relevant tests, then we would be able to develop much more confidently than we are right now, and fear breakage less. We would also spend less time debugging failures in production.

All this, together, would lead to much improved developer productivity.

What needs to be accomplished

We need to:

  1. report numbers on code coverage
  2. change our culture, so that all our new code at least is properly tested
  3. enforce the testing culture, by rejecting PRs that do not have enough tests

Work on 1 has already started for the Hacken report remediation, and I worked on it on-and-off depending on other issues that took priority. We currently report overall numbers, but should get more precise numbers depending on the part of the codebase affected, as we do not need full coverage for eg. the tools folder.

Main use case

Reliability is the main use case. However, security will also benefit indirectly, as fewer bugs usually means fewer vulnerabilities.

Links to external documentations and discussions

None

Estimated effort

The security team, and in particular @Ekleog-NEAR, is expected to work on this project. The estimated time is 2-3 weeks-person, that will need to be spread out to let enough time for culture changes to percolate.

Assumptions

None

Pre-requisites

None

Out of scope

Catching up on our backlog of untested code base is out of the scope of this project, but it will be enabled by proper measurement of code coverage.

Task list

### Tasks
- [x] report numbers on code coverage
- [ ] make it easy to find which test lines are not covered by unit tests
- [ ] change our culture, so that all our new code at least is properly tested
- [ ] enforce the testing culture, by rejecting PRs that do not have enough tests
Ekleog-NEAR commented 9 months ago

No progress on this project yet, since creating it last week I only had time to work on code reviews and publishing crates for external users.

Ekleog-NEAR commented 8 months ago

Since the last report, I configured code coverage properly. Most of the technical changes are done here, now remains what’s most important: changing the mentality so that people do not land PRs that do not meet the requirements for code coverage.

The main thing missing is this missing feature request I sent to codecov, that I noticed while writing this comment: it’s currently hard to find which lines are missing unit tests in particular.

This being said, the mentality change is going to be nontrivial: over the last week (during which code coverage was configured properly), excluding automated PRs:

Ekleog-NEAR commented 7 months ago

This is basically done, and I implemented the unit test coverage reporting.

I’ll leave this open for now just to not forget making a proper announcement for it, and then we can close: the only thing that’s missing will be incrementally ramping up our code coverage :)