plasma-umass / slipcover

Near Zero-Overhead Python Code Coverage
Apache License 2.0
494 stars 19 forks source link

Functions that are expected to be empty are not empty #23

Closed di closed 4 months ago

di commented 2 years ago

I use a library, automat that has some runtime assumptions about whether certain functions have function bodies or not.

When I run my tests with pytest and coverage, it works as expected:

$ python -m coverage run -m pytest
================================== test session starts ==================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/di/example
collected 1 item

test_something.py .                                                               [100%]

=================================== 1 passed in 0.02s ===================================

When I run it with slipcover, it causes these assumptions to fail:

 python -m slipcover -m pytest
================================== test session starts ==================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/di/example
collected 0 items / 1 error

======================================== ERRORS =========================================
__________________________ ERROR collecting test_something.py ___________________________
test_something.py:1: in <module>
    from foo import Something
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
../.pyenv/versions/3.10.4/lib/python3.10/site-packages/slipcover/__main__.py:43: in exec_module
    exec(code, module.__dict__)
foo.py:4: in <module>
    class Something:
foo.py:12: in Something
    def some_input(self):
../.pyenv/versions/3.10.4/lib/python3.10/site-packages/automat/_methodical.py:376: in decorator
    return MethodicalInput(automaton=self._automaton,
<attrs generated init automat._methodical.MethodicalInput>:11: in __init__
    __attr_validator_method(self, __attr_method, self.method)
../.pyenv/versions/3.10.4/lib/python3.10/site-packages/automat/_methodical.py:166: in assertNoCode
    raise ValueError("function body must be empty")
E   ValueError: function body must be empty
================================ short test summary info ================================
ERROR test_something.py - ValueError: function body must be empty
!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!
=================================== 1 error in 0.17s ====================================

File                 #lines    #l.miss    Cover%  Missing
-----------------  --------  ---------  --------  ---------
foo.py                    8          1        88  15
test_something.py         3          2        33  3-4

A minimally reproducing example is below

requirements.txt:

automat
coverage
pytest
slipcover

foo.py:

import automat

class Something:
    _machine = automat.MethodicalMachine()

    @_machine.state(initial=True)
    def some_state(self):
        """Nothing"""

    @_machine.input()
    def some_input(self):
        """Nothing"""

    some_state.upon(some_input, enter=some_state, outputs=[])

test_something.py:

from foo import Something

def test_something():
    Something().some_input()
jaltmayerpizzorno commented 2 years ago

Hi, thank you for reporting it, and sorry I didn't get to it any sooner. I am not sure this can be fixed on Slipcover's side... Slipcover needs to instrument the code to measure coverage, but automat requires it not to be instrumented. This is the code in automat/_methodical.py:

def _empty():
    pass

def _docstring():
    """docstring"""

def assertNoCode(inst, attribute, f):
    # The function body must be empty, i.e. "pass" or "return None", which
    # both yield the same bytecode: LOAD_CONST (None), RETURN_VALUE. We also
    # accept functions with only a docstring, which yields slightly different
    # bytecode, because the "None" is put in a different constant slot.

    # Unfortunately, this does not catch function bodies that return a
    # constant value, e.g. "return 1", because their code is identical to a
    # "return None". They differ in the contents of their constant table, but
    # checking that would require us to parse the bytecode, find the index
    # being returned, then making sure the table has a None at that index.

    if f.__code__.co_code not in (_empty.__code__.co_code,
                                  _docstring.__code__.co_code):
        raise ValueError("function body must be empty")
jaltmayerpizzorno commented 4 months ago

I don't see how this can be fixed on SlipCover's side...