Feature request: Semantic production code coverage information in sqlite database

[fkromer]

The sqlite database table "context" already contains semantic information "context". As far as I know the type of the context is limited to "test" right now.

context_id	context
1	test_some_method
2	test_some_function

The sqlite database tables "line" and "arc" contain line/branch based production code coverage information.

Could we add production code coverage information with semantic meaning? This could be done either as (option a) a separate table "semantic" or as (option b) part of the line and arc table as additional info?

option a (separate "semantic" table):

file_id	context_id	module	class	method	function
1	1	some_module	SomeClass	some_method
2	2	some_module			some_function

option b (line table with additional info):

file_id	context_id	module	class	method	function	lineno
1	1	some_module	SomeClass	some_method		17
1	1	some_module	SomeClass	some_method		18
...
2	2	some_module			some_function	34
2	2	some_module			some_function	35
...

For some consistent, exemplary database table content refer to python-tia/coveragepy_database_tables.ipynb.

[nedbat]

My goal with "context" was for it to be open-ended. The code in the alpha uses the test name as the context, but eventually, dynamic contexts will be able to record whatever information they want as the context. ClassName.function_name would be fine as a context.

[fkromer]

It's a bit off-topic in this issue but because you mentioned dynamic contexts: For advanced test impact analysis it would be helpful to track fixtures, setup functions, teardown functions, etc. means everything which relates to test code, which is executed as part of a single test execution and which would in case of a change imply the need for a re-execution of one or several affected test(s). Let's call it kind of "test code coverage". Is something like that in the scope of dynamics contexts?

[nedbat]

My plan is that the dynamic context could be determined by plugins, so someone could write as complex a determination as they want. I'm sure pytest (for example) has enough information to set the context based on fixtures, setup, etc.

My theory is that a plugin that can set a string as the context will be sufficient. Do you agree? Or is something more elaborate needed?

[fkromer]

My theory is that a plugin that can set a string as the context will be sufficient. Do you agree? Or is something more elaborate needed?

I agree.

It should be possible to map test code specific dynamic contexts to single tests of course. I don't know how coveragepy interacts with test runners exactly. All information should be there: The test runner knows the executed test and coveragepy knows what's executed as part of this tests setup (test code), actual test execution (test code like utilities or fixtures, production code under test) and teardown (test code).

[nedbat]

Coverage.py 5.0a5 now supports context plugins and an API entry point for test runners to use to set the context. There's a branch of pytest-cov that sets the pytest test id, and phase (setup, call, teardown) as the context. Is that getting closer to what you want? https://nedbatchelder.com/blog/201905/coveragepy_50a5_pytest_contexts.html

[fkromer]

Wow, that looks nice. According to your blog post the context table contains entries which allow to link test specific setup

5 test_it.py::test_prod2|setup

to file id(s) and line regions in the arc table (file_id, context_id, fromno, tono)

1 5 25 46

, right? (The file id to file mapping is contained in the file table as usual.) Do you know what the setup context covers when used with pytest in detail (files and lines refering to: fixtures, etc.)?

I'd assume that context table entries of the call context type cover every production code (class init functions, class methods, functions, etc.) executed in a specific test I guess.

9 test_it.py::test_prod3[1-1]|call

I'll have a look into the data.

[nedbat]

The pytest-cov plugin now provides good support for tracking test ids: https://nedbatchelder.com/blog/201910/pytestcov_support_for_whotestswhat.html . I'm going to close this for now, let me know what you find when you try the latest code.