ysugimoto / falco

falco is a VCL parser and linter dedicated to Fastly
MIT License
105 stars 29 forks source link

Add feature to collect function/statement/branch coverage to `test` command #343

Open nodaguti opened 3 months ago

nodaguti commented 3 months ago

This PR implements a new feature to collect test coverage data while running tests with -c / --coverage option enabled and report them in the output (either in human-friendly way or in JSON).

Approach

There are roughly two ways to implement coverage measurement: 1) collecting coverage in the interpreter side (like V8 JavaScript code coverage), and 2) collecting coverage in the userland side (like Istanbul test coverage)

For option 1, adding support for code coverage to our interpreter may result in tight coupling between the interpreter and coverage measurement, which I think would not a great idea especially when considering a fact that falco uses the interpreter not only in tester but also in simulator and console features.

For option 2, since we already have a special function to mutate an existing table in tester mode, testing.table_set, tables can be used to store coverage data.

This PR adds a new process called "instrumentation" to the preparation phase of testing: traversing an AST tree to find subroutines, statements, and branches to initialise coverage tables, and injecting testing.table_set statements to record executions of each subroutine, statement, or branch.

These injected statements will be evaluated along with original VCL code and produce coverage information for each test case.

After running all tests, each coverage information will be merged so that falco can calculate overall coverage percentages.

Screenshot

Screenshot 2024-07-21 at 4 08 02 PM

Related Issues

nodaguti commented 3 months ago

@ysugimoto I'm sorry for the PR being relatively large, but I look forward to hearing your impressions. Thank you!