Using Coverage API to cover tests from .pyc file, and with separate folders for source, test and .pyc

bensarthou commented 1 month ago

Hi,

I'm trying to use coverage to cover tests written in unittest in a production framework, with several constraints, but it fails with this error: raise NoSource(f"No source for code: '{filename}'.") coverage.exceptions.NoSource: No source for code: 'E:\adele\BSU3PythonCoverage\win_b64\code\python\lib\metrics.py'.

I've looked online and saw this SO post with a similar issue, however the proposed fix does not seems to work here.

I assume the issue is mainly because I run coverage (and the tests) through .pyc, and it breaks the link between source codes at runtime.

Is there a way to make coverage work in this setting ?

You can find below more detailed info

Detailed description of the issue:

Source code, test code, and "run" code are all at separate locations
Coverage (and tests) are run from .pyc files
I need to run coverage through the python API for parametrization (multiple repos to cover with different path relationships)

My code is structured as such (that cannot be changed unfortunately)

folder_code\
    src\
        file1.py
 ...
folder_test\
    src\
        test_file1.py
...
folder_run\
    file1.pyc
    test_file1.pyc
    coverage_main.pyc

folder_code contains the code to be tested and covered folder_test contains the source code of the test (using unittest) folder_run contains the .pyc files of the test and the source code It also contains the file calling coverage through the python API (see below)

coverage_main is written as such:

import unittest
import coverage

if __name__ == '__main__':

    ## Path to source code (src in module of code fw)
    SRC_PATH = ... (path to folder_code\src)
    ## Path to test code (technically not useful)
    TEST_PATH = ... (path to folder_test\src)
    ## Path to run .pyc
    RTV_PATH = ... (path to runtime .pyc)

    # Path to save .coverage 
    OUTPUT_DIR = ...

    cov = coverage.Coverage(source=[SRC_PATH, RTV_PATH], # <-- I suspect the issue is here 
                                            data_file=os.path.join(OUTPUT_DIR, '.coverage')) 
    cov.start()

    ## LAUNCH TESTS
    loader = unittest.TestLoader()
    tests = loader.discover(TEST_PATH)
    testRunner = unittest.runner.TextTestRunner(buffer=True)
    testRunner.run(tests)

    cov.stop()
    cov.save()
    cov.html_report(directory=OUTPUT_DIR)

While calling this API through the .pyc file with the following command: $ python -u -m coverage_main.pyc

I obtain the following log and error:

----------------------------------------------------------------------
Ran 2 tests in 0.081s

OK
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "ROOT_DIR\PythonIntegration.tst\PythonCoverage.m\src\python_coverage.py", line 80, in <module>
    cov.html_report(directory=OUTPUT_DIR)
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\control.py", line 1177, in html_report
    ret = reporter.report(morfs)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\html.py", line 331, in report
    for fr, analysis in get_analysis_to_report(self.coverage, morfs):
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\report_core.py", line 100, in get_analysis_to_report
    analysis = coverage._analyze(morf)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\control.py", line 948, in _analyze
    return analysis_from_file_reporter(data, self.config.precision, file_reporter, filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\results.py", line 31, in analysis_from_file_reporter
    statements = file_reporter.lines()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\python.py", line 194, in lines
    return self.parser.statements
           ^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\python.py", line 185, in parser
    self._parser = PythonParser(
                   ^^^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\parser.py", line 59, in __init__
    self.text = get_python_source(self.filename)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "ROOT_DIR\win_b64\code\python\lib\PythonIntegration.tst_packages\Python311\site-packages\coverage\python.py", line 64, in get_python_source
    raise NoSource(f"No source for code: '{filename}'.")
coverage.exceptions.NoSource: No source for code: 'E:\adele\BSU3PythonCoverage\win_b64\code\python\lib\metrics.py'.

We can see on the log that tests are correctly ran (the unittest stdout is on top of the log), but coverage fails when doing the report, mostly because the .coverage file is empty (when opening it with an SQL reader, I saw that the database structure is here, but there are no lines)

Feel free to ask for more info if needed

nedbat commented 1 month ago

You can make this work by breaking it into two steps: coverage run with only the .pyc files, but then coverage html somewhere with the source files. If you have source files on the machine already, then you might need a [paths] section in your coverage configuration: https://coverage.readthedocs.io/en/7.6.1/config.html#paths

BTW: I'm not sure you need to write your own .py file to run coverage through the API. You should be able to do this with commands on the command line.

bensarthou commented 1 month ago

Hi @nedbat, thanks for the fast answer

Unfortunately, as the goal is to have the coverage being run automatically in different servers, with different configurations, I need to do this via a script, be it Python or maybe a bash indeed, if the only available option is through the command line I can try going the bash way

But I suppose that any parameter that can be passed through the coverage configuration, could probably be passed through the API, so if you have any idea how to do that, it would really help me

Thanks !

nedbat commented 1 month ago

You can try using Coverage.combine() with a paths argument to indicate how the source files correspond to the .pyc files.

bensarthou commented 1 month ago

I've tried this solution, but to no avail unfortunately, it fails with the same error If I understand correctly the Coverage documentation, .combine() is helpful to combine .coverage files from different locations. But here the issue is that the .coverage file is empty, so there is nothing to combine

I've tried different solutions since yesterday, for example loading the tests directly by instance in the TestLoader, instead of using discover, but no results for now

My understanding is that the issue is happening during the "run" phase, where coverage is unable to locate the source file related to the .pyc. I suppose the source option of the Coverage method is the standard way to go, but using the .pyc seems to hinder this functionnality

nedbat / coveragepy

Using Coverage API to cover tests from .pyc file, and with separate folders for source, test and .pyc #1861

Detailed description of the issue: