Missing pytest.CollectReport when invoking pylint with plugin enabled

[mcstoufer]

Describe the bug PyLint generation causes stack trace when the pylint-pytest plugin is enabled.

To Reproduce Package versions

$ pylint --version pylint 3.0.2 astroid 3.0.1 Python 3.11.7 (main, Dec 4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]

$ pytest --version pytest 6.2.5

pylint --output-format=json:pylint.json,parseable tests/test_add_coverage_report_to_pr.py > pylint_report.out

Folder structure

$ tree -L 2
tests/test_add_coverage_report_to_pr.py

File content

import sys
import os

import pytest
import mock
from unittest.mock import mock_open

from add_code_coverage_report_to_pr import AddCoverageReportToPR, AssembleReport
from lib.paths import Paths

@pytest.fixture(scope="module", autouse=True)
def env_reset():
    sys.argv = []

@pytest.fixture(name="env_setup")
def environment():
    os.environ['JENKINS_USER'] = "test"
    os.environ['JENKINS_AUTH'] = "test"
    os.environ['GITHUB_REPO'] = XXXXX/pr-12345'
    os.environ['REPOSOLNS_MVN_REPO'] = 'https://foo.com/artifactory/foo-mvn'
    yield 'environment'
    del os.environ['JENKINS_USER']
    del os.environ['JENKINS_AUTH']
    del os.environ['GITHUB_REPO']
    del os.environ['REPOSOLNS_MVN_REPO']
    sys.argv = []

@mock.patch('lib.wmartifactstorage.WMArtifactStorage.wmartifact_storage_factory')
def test_validate_environment(mock_fact, env_setup):
    acr = AddCoverageReportToPR()
    res = acr.validate_environment()
    assert not res
    mock_fact.assert_called()
# Additional tests removed for brevity's sake.

pylint output with the plugin

Traceback (most recent call last):
  File "/opt/homebrew/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
             ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/opt/homebrew/lib/python3.11/site-packages/pylint/lint/run.py", line 162, in __init__
    args = _config_initialization(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pylint/config/config_initialization.py", line 61, in _config_initialization
    linter.load_plugin_modules(utils._splitstrip(config_data["load-plugins"]))
  File "/opt/homebrew/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 381, in load_plugin_modules
    module.register(self)
  File "/opt/homebrew/lib/python3.11/site-packages/pylint_pytest/__init__.py", line 22, in register
    checker = importlib.import_module(module, package=os.path.basename(dirname))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/homebrew/lib/python3.11/site-packages/pylint_pytest/checkers/fixture.py", line 28, in <module>
    class FixtureCollector:
  File "/opt/homebrew/lib/python3.11/site-packages/pylint_pytest/checkers/fixture.py", line 31, in FixtureCollector
    errors: Set[pytest.CollectReport] = set()
                ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pytest' has no attribute 'CollectReport'. Did you mean: 'Collector'?

(Optional) pytest output from fixture collection

$ pytest --fixtures --collect-only <path/to/test/module.py>
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.11.7, pytest-6.2.5, py-1.11.0, pluggy-1.2.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /Users/m0s0l1z/Projects/mre-tools, configfile: pytest.ini
plugins: random-order-1.1.0, bigquery-mock-0.1.2, mock-3.11.1, requests-mock-1.11.0
collected 25 items

<Module tests/test_add_coverage_report_to_pr.py>
  <Function test_validate_environment>
  <Function test_extract_parser_info_nexus>
  <Function test_extract_parser_info_jfrog>
  <Function test_get_coverage_rules>
  <Function test_get_coverage_rules_no_path>
  <Function test_round_to_float>
  <Function test_get_previous_report_json>
  <Function test_get_previous_report_json_value_err>
  <Function test_get_previous_report_json_no_path>
  <Function test_get_current_report_json_no_path_gen_fail>
  <Function test_get_current_report_json>
  <Function test_get_current_report_no_gen>
  <Function test_get_coverage_by_counter_curr>
  <Function test_get_coverage_by_counter_none>
  <Function test_get_coverage_by_counter_less>
  <Function test_get_coverage_by_counter_greater>
  <Function test_get_coverage_close>
  <Function test_get_coverage_close_no_latest>
  <Function test_main>
  <Function test_generate_report_if_none_fail>
  <Function test_generate_report_if_none>
  <Function test_generate_report_if_none_correct>
  <Function test_assemble_report_no_current_report>
  <Function test_assemble_report_empty_current_report>
  <Function test_assemble_report_with_current_report>

--------------------------------------------------------------- fixtures defined from pytest_bigquery_mock.plugin ----------------------------------------------------------------
bq_client_mock
    /opt/homebrew/lib/python3.11/site-packages/pytest_bigquery_mock/plugin.py:42: no docstring available

-------------------------------------------------------------- fixtures defined from test_add_coverage_report_to_pr --------------------------------------------------------------
env_reset [module scope]
    tests/test_add_coverage_report_to_pr.py:13: no docstring available

env_setup
    tests/test_add_coverage_report_to_pr.py:18: no docstring available

Status code: 0

========================================================================== 25 tests collected in 0.21s ===========================================================================

Expected behavior The linting report is generated in both JSON and test output.

Additional context This works just fine w/o the pylint-pytest plugin enabled in the .pylintrc file.

[mcstoufer]

Verified it isn't there:

more /opt/homebrew/lib/python3.11/site-packages/pytest/init.py

...
from _pytest.nodes import Collector
...

[stdedos]

Hello @mcstoufer!

Apologies for the very long time it has taken me to respond to your issue. For some reason, I was not informed it was opened 😰 Edit: Apparently, I was not subscribed to the repo I am maintaining? 🤣

I will do some digging and try to make a plan to mitigate your issue tonight.

However, for the sake of completeness, may I ask: Why are you using "such an outdated" pytest version? 😕 I can see from your log it says pytest-6.2.5.

I don't know if I'll need it, but do you mind pasting a pip freeze | grep pytest output? Scratch that, you did it already 😅

[stdedos]

  File "/opt/homebrew/lib/python3.11/site-packages/pylint_pytest/checkers/fixture.py", line 31, in FixtureCollector
    errors: Set[pytest.CollectReport] = set()
                ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pytest' has no attribute 'CollectReport'. Did you mean: 'Collector'?

Indeed, it seems that this is a pytest-7.0.0rc1 thing:

https://github.com/pytest-dev/pytest/blob/6.2.x/src/_pytest/reports.py#L342-L344 https://github.com/bluetech/pytest/commit/85c372c9d7069dfc3a7deb5c95c60acdc7b4fef6 https://docs.pytest.org/en/8.0.x/changelog.html#id261

I'll do a

if TYPE_CHECKING:
    try:
        from pytest import CollectReport
    except ImportError:
        from _pytest.reports import CollectReport

here https://github.com/pylint-dev/pylint-pytest/blob/b6a8f22b7bfa0298f6004ab3051201a8466d0efb/pylint_pytest/checkers/fixture.py#L30 - including testing tomorrow and expanding tox to include pytest v6.

However, do note that the last bugfix of v6.2.5 was 2021-08-29 (https://docs.pytest.org/en/6.2.x/changelog.html#pytest-6-2-5-2021-08-29). I'd suggest you'd updated either way 🙏

[pzelnip]

Confirmed for me upgrading to pytest 8 (from 6.2.5) fixed the issue.

[stdedos]

Confirmed for me upgrading to pytest 8 (from 6.2.5) fixed the issue.

Thank you very much for responding!

Would you kindly test https://github.com/pylint-dev/pylint-pytest/tree/fix/45/missing-pytest.CollectReportin-pytest-6?

I would want to push that fix regardless (I do have a pipeline, but I'd like a human validation regardless 🙏)

[stdedos]

Closing, as the fix is verified (update to pytest v8), and no-one is interested to verify the v6 fix