[POC] per test code coverage using TracePoint.new(:line)

[anmarchenko]

Proof of concept for per test code coverage using TracePoint class to attach to line events. Works in files mode (tracking only source files executed) and lines mode (tracking files together with executed lines).

Performance evaluation

Projects used to evaluate performance:

• sidekiq: very small and fast test suite
• rubocop: very big but quite fast test suite (20 000 tests under 2 minutes)
• vagrant: very big and slow test suite (5 000 tests in about 14 minutes)

In the following table are durations of the whole test session in seconds, in brackets overhead percentage compared to run with datadog-ci.

Coverage type	Sidekiq	Rubocop	Vagrant
No coverage	~6,8	111	827
Simplecov (total coverage)	~7	137	864
Per test coverage, files	~11 (61%)	617 (455%)	1044 (20%)
Per test coverage, lines	~11 (61%)	617 (455%)	1094 (26%)
Per test coverage, lines AND total coverage with simplecov*	~11 (57%)	660 (380%)	1122 (29%)

* overhead here is compared to "Simplecov (total coverage)"

Relevant code

mode = :files # or :lines
regex = /\A#{Regexp.escape(Utils::Git.root + File::SEPARATOR)}/i.freeze
coverage = {}

@tracepoint = TracePoint.new(:line) do |tp|
  # filter only files that are under the git repository root
  next unless tp.path =~ regex

  if mode == :files
    coverage[tp.path] = true
  elsif mode == :lines
    coverage[tp.path] ||= Set.new
    coverage[tp.path] << tp.lineno
  end
end

# when starting test
@tracepoint.enable

# when ending test
@tracepoint.disable

[ioquatix]

If you are finding the coverage is not what you expect, you might like to check this: https://github.com/ioquatix/covered/blob/v0.18.5/lib/covered/capture.rb#L23-L34

I found that you sometimes need to look at the call stack to figure out the coverage of sub-expressions.

[anmarchenko]

@ioquatix thanks, I am keeping this in mind

for the current stage files coverage would be actually enough for the use case, then I'll have more time to dive deeper into lines coverage