search

pylint-dev / pylint-pytest

A Pylint plugin to suppress pytest-related false positives.
https://pypi.org/project/pylint-pytest/
MIT License
14 stars 3 forks source link

PyTest FixtureRequest #74

Open dylan-at-na opened 2 months ago

dylan-at-na commented 2 months ago

Bug description

# Given some PyTest class with a setup_teardown
class TestFile:
    @pytest.fixture(scope="class")
    def setup_teardown_environment(self, request: pytest.FixtureRequest):
        # The following line assigns a value to `temp1` which can be accessed throughout
        # the test class via `self`
        request.cls.temp1 = 54321
        print(self.temp1) # This executes via PyTest successfully
        # PyLint check throws this error: Instance of 'TestFile' has no 'temp1' member 
        # PylintE1101:no-member

Command used

pylint ${file}

Pylint output

Instance of 'TestFile' has no 'temp1' member PylintE1101:no-member

Expected behavior

PyLint should not identify this as a problem at all, as request.cls and self are intrinsically linked when operating within the same test class.

Pylint version

pylint 3.2.3
astroid 3.2.2
Python 3.11.4 (main, May  8 2024, 12:11:54) [GCC 9.4.0]

OS / Environment

Ubuntu 20

jacobtylerwalls commented 1 month ago

I think this is better raised as a feature request at pylint-dev/pylint-pytest

stdedos commented 1 month ago

Hello @dylan-at-na!

Are you sure this is working like you are demonstrating?

We do actually have a test like that in the repo: https://github.com/pylint-dev/pylint-pytest/blob/8310a88916e4b0b7462fc525c56d60b9af4dc126/tests/input/no-member/not_using_cls.py

I have tried to replicate your setup

$ git --no-pager diff 
diff --git a/tests/input/no-member/not_using_cls.py b/tests/input/no-member/not_using_cls.py
index 95438af..16caf33 100644
--- a/tests/input/no-member/not_using_cls.py
+++ b/tests/input/no-member/not_using_cls.py
@@ -7,6 +7,14 @@ class TestClass:
     def setup_class(request):
         clls = request.cls
         clls.defined_in_setup_class = 123
+        print(clls.defined_in_setup_class)

     def test_foo(self):
         assert self.defined_in_setup_class
+
+    @pytest.fixture(scope="class")
+    def setup_teardown_environment(self, request: pytest.FixtureRequest):
+        # The following line assigns a value to `temp1`
+        # which can be accessed throughout the test class via `self`.
+        request.cls.temp1 = 54321
+        print(self.temp1)
$ pytest tests/input/no-member/not_using_cls.py -s
======================================= test session starts =======================================
platform linux -- Python 3.11.9, pytest-7.4.3, pluggy-1.3.0 -- ~/pylint-pytest/.venv/bin/python3.11
cachedir: .pytest_cache
rootdir: ~/pylint-pytest
configfile: pyproject.toml
plugins: cov-5.0.0
collected 1 item                                                                                                                                                                                          

tests/input/no-member/not_using_cls.py::TestClass::test_foo 123
PASSED

======================================== 1 passed in 0.01s ========================================

but, as you can see, I am unable to make your fixture work.

The test is printing out 123 (clls.defined_in_setup_class), but not 54321.

You MUST mark the fixture as autouse=True, according to the documentation: https://docs.pytest.org/en/latest/how-to/unittest.html#using-autouse-fixtures-and-accessing-other-fixtures

jacobtylerwalls commented 1 month ago

(Let's find out if the reporter is using pylint-pytest or not.)

dylan-at-na commented 1 month ago

@stdedos Please see the below example which demonstrates a functional example (autouse is ignored in the first example, but is used further down)...

import pytest

class TestExample:
    @pytest.fixture(scope="class")
    def setup_teardown_username(self, request: pytest.FixtureRequest):
        request.cls.username = 'dylan'
        yield True
        del request.cls.username

    @pytest.fixture(scope="function")
    def setup_teardown_password(self, request: pytest.FixtureRequest, setup_teardown_username):
        request.cls.password = '*****'
        yield setup_teardown_username, True
        del request.cls.password

    def test_username_and_password(self, setup_teardown_password):
        assert self.username == 'dylan'
        assert self.password == '*****'

And the log output from running the above...

source .../.venv/bin/activate.csh
% (pycldqa-py3.11) %  /usr/bin/env .../.venv/bin/python /u/dylanl/.vscode-server/extensions/ms-python.debugpy-2024.7.11591012-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 55139 -- -m pytest /u/dylanl/repos/local/scratch/test_scratch.py 
================================= test session starts ==================================
platform linux -- Python 3.11.4, pytest-8.2.2, pluggy-1.5.0
rootdir: /u/dylanl/repos/local
configfile: pyproject.toml
plugins: xdist-3.6.1
collected 1 item                                                                       

../local/scratch/test_scratch.py .                                               [100%]

================================== 1 passed in 0.04s ===================================

The following screenshot demonstrates the error being thrown... image

@jacobtylerwalls The following is the result of running pylint against the above code with pylint-pytest plugin enabled. Relevant lines are marked with ***

(pycldqa-py3.11) %  cd ...; /usr/bin/env .../.venv/bin/python /u/dylanl/.vscode-server/extensions/ms-python.debugpy-2024.7.11591012-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 36313 -- -m pylint --load-plugins=pylint_pytest /u/dylanl/repos/local/scratch/test_scratch.py 
************* Module test_scratch
/u/dylanl/repos/local/scratch/test_scratch.py:9:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:15:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:1:0: C0114: Missing module docstring (missing-module-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:3:0: C0115: Missing class docstring (missing-class-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:11:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:16:4: C0116: Missing function or method docstring (missing-function-docstring)
***/u/dylanl/repos/local/scratch/test_scratch.py:17:15: E1101: Instance of 'TestExample' has no 'username' member (no-member)
***/u/dylanl/repos/local/scratch/test_scratch.py:18:15: E1101: Instance of 'TestExample' has no 'password' member (no-member)

-----------------------------------
Your code has been rated at 0.00/10

If I now add autouse=True to the fixtures and make the necessary changes we end up with the following code...

import pytest

class TestExample:
    @pytest.fixture(scope="class", autouse=True)
    def setup_teardown_username(self, request: pytest.FixtureRequest):
        request.cls.username = 'dylan'
        yield True
        del request.cls.username

    @pytest.fixture(scope="function", autouse=True)
    def setup_teardown_password(self, request: pytest.FixtureRequest):
        request.cls.password = '*****'
        yield True
        del request.cls.password

    def test_username_and_password(self):
        assert self.username == 'dylan'
        assert self.password == '*****'

Which again, passes.

================================= test session starts ==================================
platform linux -- Python 3.11.4, pytest-8.2.2, pluggy-1.5.0
rootdir: /u/dylanl/repos/local
configfile: pyproject.toml
plugins: xdist-3.6.1
collected 1 item                                                                       

../local/scratch/test_scratch.py .                                               [100%]

================================== 1 passed in 0.06s ===================================

However, running PyLint on this produces the same errors (relevant lines marked with ***)...

pylint --load-plugins=pylint_pytest /u/dylanl/repos/local/scratch/test_scratch.py 
************* Module test_scratch
/u/dylanl/repos/local/scratch/test_scratch.py:9:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:15:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:1:0: C0114: Missing module docstring (missing-module-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:3:0: C0115: Missing class docstring (missing-class-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:11:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:16:4: C0116: Missing function or method docstring (missing-function-docstring)
***/u/dylanl/repos/local/scratch/test_scratch.py:17:15: E1101: Instance of 'TestExample' has no 'username' member (no-member)
***/u/dylanl/repos/local/scratch/test_scratch.py:18:15: E1101: Instance of 'TestExample' has no 'password' member (no-member)

------------------------------------------------------------------
Your code has been rated at 0.00/10 (previous run: 3.75/10, -3.75)

Interestingly enough if I use your code sample located at https://github.com/pylint-dev/pylint-pytest/blob/8310a88916e4b0b7462fc525c56d60b9af4dc126/tests/input/no-member/not_using_cls.py

I don't get the error

************* Module test_scratch
/u/dylanl/repos/local/scratch/test_scratch.py:12:0: C0304: Final newline missing (missing-final-newline)
/u/dylanl/repos/local/scratch/test_scratch.py:1:0: C0114: Missing module docstring (missing-module-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:4:0: C0115: Missing class docstring (missing-class-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:7:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:11:4: C0116: Missing function or method docstring (missing-function-docstring)

------------------------------------------------------------------
Your code has been rated at 2.86/10 (previous run: 0.00/10, +2.86)

This prompted me to attempt PyLint with @staticmethod applied to my fixtures but this did not work.

In a last ditch effort I assigned request.cls to clls as you do above...

import pytest

class TestExample:
    @pytest.fixture(scope="class", autouse=True)
    def setup_teardown_username(self, request: pytest.FixtureRequest):
        clls = request.cls
        clls.username = 'dylan'
        yield True
        del request.cls.username

    @pytest.fixture(scope="function", autouse=True)
    def setup_teardown_password(self, request: pytest.FixtureRequest):
        clls = request.cls
        clls.password = '*****'
        yield True
        del request.cls.password

    def test_username_and_password(self):
        assert self.username == 'dylan'
        assert self.password == '*****'

Again, this passes.

And it removes the error line for the class scoped fixture (setup_teardown_username), but not for the function scoped fixture (setup_teardown_password). Relevant line marked with ***.

************* Module test_scratch
/u/dylanl/repos/local/scratch/test_scratch.py:10:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:17:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:1:0: C0114: Missing module docstring (missing-module-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:3:0: C0115: Missing class docstring (missing-class-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:12:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:18:4: C0116: Missing function or method docstring (missing-function-docstring)
***/u/dylanl/repos/local/scratch/test_scratch.py:20:15: E1101: Instance of 'TestExample' has no 'password' member (no-member)

------------------------------------------------------------------
Your code has been rated at 2.00/10 (previous run: 2.86/10, -0.86)

(Incorrect assumption, see follow up comment for clarification) ~So it would seem that clls is a somewhat special variable as far as PyLint is concerned? Though based on this search there doesn't seem to be much reference to it https://www.google.com/search?q=pylint+%22clls%22&client=firefox-b-d&sca_esv=41efd453413bdc83&sca_upv=1&ei=1kKOZrm0GdiB9u8P0YCu0A0&ved=0ahUKEwi59_Dag5yHAxXYgP0HHVGAC9oQ4dUDCA8&uact=5&oq=pylint+%22clls%22&gs_lp=Egxnd3Mtd2l6LXNlcnAiDXB5bGludCAiY2xscyIyCBAhGKABGMMESNw1UIEVWP4zcAF4AZABAJgBjwGgAegDqgEDMC40uAEDyAEA-AEBmAIFoAKBBMICChAAGLADGNYEGEfCAggQABiABBiiBMICCBAAGKIEGIkFwgIFECEYoAGYAwCIBgGQBgiSBwMxLjSgB6cG&sclient=gws-wiz-serp~

dylan-at-na commented 1 month ago

Quick follow up - it isn't clls that resolves it in the class scoped fixture. It is simply the act of assigning request.cls to a variable. Any variable name will do.

import pytest

class TestExample:
    @pytest.fixture(scope="class", autouse=True)
    def setup_teardown_username(self, request: pytest.FixtureRequest):
        mocha = request.cls
        mocha.username = 'dylan'
        yield True
        del request.cls.username

    @pytest.fixture(scope="function", autouse=True)
    def setup_teardown_password(self, request: pytest.FixtureRequest):
        clls = request.cls
        clls.password = '*****'
        yield True
        del request.cls.password

    def test_username_and_password(self):
        assert self.username == 'dylan'
        assert self.password == '*****'

Passes.

PyLint:

************* Module test_scratch
/u/dylanl/repos/local/scratch/test_scratch.py:10:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:17:0: C0303: Trailing whitespace (trailing-whitespace)
/u/dylanl/repos/local/scratch/test_scratch.py:1:0: C0114: Missing module docstring (missing-module-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:3:0: C0115: Missing class docstring (missing-class-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:5:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:12:4: C0116: Missing function or method docstring (missing-function-docstring)
/u/dylanl/repos/local/scratch/test_scratch.py:18:4: C0116: Missing function or method docstring (missing-function-docstring)
***/u/dylanl/repos/local/scratch/test_scratch.py:20:15: E1101: Instance of 'TestExample' has no 'password' member (no-member)

------------------------------------------------------------------
Your code has been rated at 2.00/10 (previous run: 2.00/10, +0.00)

Again this does not resolve the issue in the functionscoped fixture.

stdedos commented 1 month ago

So it would seem that clls is a somewhat special variable as far as PyLint is concerned? ...

FWIW: Your search result gives exactly one result for me

image

clls name is not "special" per-se; any name would do: https://github.com/pylint-dev/pylint-pytest/blob/8310a88916e4b0b7462fc525c56d60b9af4dc126/pylint_pytest/checkers/class_attr_loader.py#L42 ... but for now, it MUST be like that.

I am not sure why the original author chose to write this test (clls.defined_in_setup_class = 123) and skipped the most obvious request.cls.X = Y (coming in a commit soon). It is an issue with "contrived/MWE" examples that you cannot see "where" are they coming from.

Quick follow up - it isn't clls that resolves it in the class scoped fixture. It is simply the act of assigning request.cls to a variable. Any variable name will do.

Exactly 🙃

@pytest.fixture(scope="function")

I will try this locally too. However, as of now, for a request.cls to be considered eligible, it needs to satisfy https://github.com/pylint-dev/pylint-pytest/blob/8310a88916e4b0b7462fc525c56d60b9af4dc126/pylint_pytest/utils.py#L83-L107

I will try all scopes and see what their result is. And then code/test appropriately.

... it seems that this is getting quick traction, so I'll push a branch within the day.

One thing that I haven't been able to figure out, unfortunately, is that the pylint/pylint-pytest is sensitive to ordering.

tl;dr:

class TestClass:
    @staticmethod
    @pytest.fixture(scope="class", autouse=True)
    def setup_class(request):
        clls = request.cls
        clls.defined_in_setup_class = 123

    def test_foo(self):
        assert self.defined_in_setup_class

works,

class TestClass:
    def test_foo(self):
        assert self.defined_in_setup_class

    @staticmethod
    @pytest.fixture(scope="class", autouse=True)
    def setup_class(request):
        clls = request.cls
        clls.defined_in_setup_class = 123

fails.

astroid visits top-to-bottom. In order to fix this "correctly" we MUST first visit eligible @pytest.fixture(autouse=True) fixtures. Perhaps after a possible __init__? Maybe even strictly in the correct call order (session, package, module, class, function)

dylan-at-na commented 1 month ago

For anybody else who comes across this issue - the easiest temporary workaround with minimal changes to your code is to introduce a fixture which is class scoped, and autouse and initializes all relevant variables to None within your request.cls. See below for an example relevant to the above code snippet...

import pytest

class TestExample:
    @pytest.fixture(scope='class', autouse=True)
    def setup_teardown_request_cls(self, request: pytest.FixtureRequest):
        _cls = request.cls
        _cls.username = None
        _cls.password = None

    @pytest.fixture(scope="class")
    def setup_teardown_username(self, request: pytest.FixtureRequest):
        request.cls.username = 'dylan'
        yield True
        del request.cls.username

    @pytest.fixture(scope="function")
    def setup_teardown_password(self, request: pytest.FixtureRequest, setup_teardown_username: bool):
        request.cls.password = '*****'
        yield True
        del request.cls.password

    def test_username_and_password(self, setup_teardown_password: bool):
        assert self.username == 'dylan'
        assert self.password == '*****'

This has the added benefit of isolating the initialization, thus being easy to remove when and if @stdedos changes are in place. You can of course initialize your variables to any default value you like but for demonstrations sake I have used None.