testmon doesn't track sources in venv directory

[emavgl]

Hi everyone. I am testing testmon for the project in my company and after having some issues, I am trying to create a simple use case to help identifying the problem and maybe also with the debugging.

I started creating a simple code:

# tests/test_gold_app.py

class MyCode:
    def add(self, a, b):
        return a + b

class TestCode:
    def test_mock(self):
        assert MyCode().add(1, 3) == 4

I then run testmon once

python -m pytest tests/test_gold_app.py --testmon

and tests are collected and executed fine.

Then, I do a small change in the MyCode class.

class MyCode:
    def add(self, a, b):
        return a + b + 1 # this was changed

class TestCode:
    def test_mock(self):
        assert MyCode().add(1, 3) == 4 # this should fail, but the test does not run again

and this time no tests is running.

========================================== test session starts ===========================================
platform darwin -- Python 3.9.6, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
testmon: changed files: 0, skipping collection of 1 files, environment: default
rootdir: /Users/emv/repos/test-testmon
plugins: pspec-0.0.4, xdist-2.3.0, testmon-1.4.5, html-3.1.1, mock-3.6.1, html-reporter-0.2.9, metadata-2.0.4, forked-1.6.0
collected 1 item / 1 deselected  

What I expect, is that since the code of the method add has changed. Also the test test_mock has to run again. Am I right?

Same thing, if I change the test only, maybe changing the assertion, like this:

class MyCode:
    def add(self, a, b):
        return a + b

class TestGoldApp:
    def test_mock(self):
        assert MyCode().add(1, 3) == 5

test_mock should run again, but it does not.

Am I missing something? Thank you!

[pokopt]

Hi @emavgl, thanks for your effort to provide example that didn't work for you! You are right this is not expected behavior of testmon - it definitely should recognize changes you described. And it does in my environment. We will investigate deeper and maybe will come back for some additional info.

[emavgl]

Hi @pokopt, thank you for checking my issue. In case you need something, I would be happy to help you with the investigation

[pokopt]

Hi @emavgl. We are able to reproduce this behavior if 'tests' directory is placed inside your virtual environment directory. Is it your case as well? Is your virtual environment created in /Users/emv/repos/test-testmon ? If so, there is workaround to create virtual environment in separate directory (e.g. /Users/emv/repos/test-testmon/venv) and testmon should run just fine. However we will try to provide real fix for this situation.

[emavgl]

Great catch! It is indeed my case. I tried with the workaround and it now works as expected. I think it is a bug that needs to be addressed, even if in my case, it is not that problematic. I will continue with other investigations then :)