tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
800 stars 54 forks source link

Add support for pytest5 #143

Closed AceFire6 closed 5 years ago

AceFire6 commented 5 years ago

Closes #142

AceFire6 commented 5 years ago

It seems the warning being shown changes the output the test expects.

In the file test.test_testmon.TestmonDeselect#test_tlf::261 the actual output is:

'===================== 1 failed, 1 warnings in 0.03 seconds ====================='

and what is expected is:

'===================== 1 failed in 0.03 seconds ====================='
blueyed commented 5 years ago
  1. needs to test pytest5 explicitly on CI

  2. the warning

    =============================== warnings summary ===============================
    /home/travis/build/tarpas/pytest-testmon/.tox/py37-pytestmaster/lib/python3.7/site-packages/_pytest/config/__init__.py:758
    /home/travis/build/tarpas/pytest-testmon/.tox/py37-pytestmaster/lib/python3.7/site-packages/_pytest/config/__init__.py:758: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: test
    self._mark_plugins_for_rewrite(hook)

Since this is new it would be good to know why it happens now, e.g. by git-bisecting pytest.

tarpas commented 5 years ago

Thanks guys. It's more important to release this than to find out cause of the warning, so it's fine to relax the assertion and create ticket about the warning. I'll have time tomorrow.

AceFire6 commented 5 years ago

@blueyed I did a bisect. The process I followed was:

  1. Have both pytest and pytest-testmon on my local filesystem
  2. Set up a tox config to install the local pytest
    recreate: true
    ...
    localpytest: git+file:///Users/jethromuller/flickswitch/pytest/
  3. Create a shell script to run the test in tox during the bisect run

    #!/usr/bin/env bash
    
    cd ~/flickswitch/pytest-testmon/
    tox -e localpytest -- test/test_testmon.py::TestmonDeselect::test_tlf
  4. In the pytest directory I did:
    $ git bisect start HEAD 4.5.0
    $ git bisect run ./bisect_test.sh

The result being that the first commit to pass the test was: pytest-dev/pytest@3d01dd3ad

I looked at the pytest files changes in this diff https://github.com/pytest-dev/pytest/compare/3d01dd3ad...master. It seems the removal of this is what might be causing the warning: https://github.com/pytest-dev/pytest/pull/5468/files#diff-1284161c2c98c8aedd66545ff79a8af6L803

I'm not sure if my assessment is correct. But that's what I've found

tarpas commented 5 years ago

@AceFire6 @blueyed thanks a lot guys. Jethro I think you're quite correct with the bisect. I'm not familiar with pytester and assertion rewrites that much, so at first glance I don't know what's the problem. I'm creating a new ticket for this warning (which is emitted from the test inside test), not from our primary test suite.