Tests not discovered when using --black or --pylama in setup.cfg

[hoonhout]

Environment data

• VS Code version: 1.37.0
• Extension version (available under the Extensions sidebar): 2019.8.30787
• OS and version: Linux 665fce967506 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64 GNU/Linux
• Python version (& distribution if applicable, e.g. Anaconda): Conda Python 3.7.3
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Relevant/affected Python packages and their versions: black==19.3b0, pylama==7.7.1
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): no

Expected behaviour

pytest tests are discovered by extension and displayed in the Test panel; CodeLens options are added to the test functions.

Actual behaviour

pytest tests are discovered by extension, but not displayed in the Test panel; no CodeLens options are added to the test functions.

Test discovery exits with code 3 if and only if the options --black and/or --pylama are added to the pytest comment through the setup.cfg file ([tool:pytest] -> addopts)

Steps to reproduce:

1. Create a test_api.py file with a test_api function.
2. Enable pytest in the workspace:

.vscode/settings.json:

{
    "python.pythonPath": "/opt/conda/bin/python",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

3. Add options --black and/or --pylama to setup.cfg -> [tool:pytest] -> addopts.
4. Ctrl-Shift-P -> Python: Discover tests

Logs

Truncated output for Python Test Log in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python Test Log)

python /root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /opt/conda/bin/python
cachedir: .pytest_cache
rootdir: /pyramid-app-pi-backend, inifile: setup.cfg, testpaths: tests
plugins: pylama-7.7.1, black-0.3.7, cov-2.7.1
collecting ... collected 31 items
<PylamaItem tests/conftest.py>
<PylamaItem tests/helpers.py>
<PylamaItem tests/test_api.py>
<Module tests/test_api.py>
  <Function test_purge>
  ...
  <Function test_integrity>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 498, in perform_collect
INTERNALERROR>     session=self, config=self.config, items=items
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 80, in pytest_collection_modifyitems
INTERNALERROR>     test, parents = parse_item(item, self.NORMCASE, self.PATHSEP)
INTERNALERROR>   File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/pytest/_pytest_item.py", line 156, in parse_item
INTERNALERROR>     if not fspath.endswith(relfile[1:]):
INTERNALERROR> TypeError: 'NoneType' object is not subscriptable
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 206, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 249, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 259, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 501, in perform_collect
INTERNALERROR>     hook.pytest_collection_finish(session=self)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/opt/conda/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 94, in pytest_collection_finish
INTERNALERROR>     test, parents = parse_item(item, self.NORMCASE, self.PATHSEP)
INTERNALERROR>   File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/pytest/_pytest_item.py", line 156, in parse_item
INTERNALERROR>     if not fspath.endswith(relfile[1:]):
INTERNALERROR> TypeError: 'NoneType' object is not subscriptable

========================= no tests ran in 0.55 seconds =========================

Traceback (most recent call last):
  File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/run_adapter.py", line 18, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/__main__.py", line 90, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 36, in discover
    raise Exception('pytest discovery failed (exit code {})'.format(ec))
Exception: pytest discovery failed (exit code 3)

Related

Might be related to https://github.com/microsoft/vscode-python/issues/5458#issuecomment-520019947

[kimadeline]

Hi @hoonhout 👋 thank you for reporting this issue!

I tried reproducing your issue, however I don't get the same logs as what you pasted.

• Content of my test_api.py file:

def test_api():
    print("test api")
    pass

• Content of my setup.cfg file:

[tool:pytest]
addopts = --black

• Bash output:

> /anaconda3/bin/python ~/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear
equivalent command: /anaconda3/bin/python -m pytest --black --collect-only -s --cache-clear
ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: unrecognized arguments: --black
  inifile: /Users/kimiguel/Documents/Sandbox/conda-pytest/setup.cfg
  rootdir: /Users/kimiguel/Documents/Sandbox/conda-pytest

Traceback (most recent call last):
  File "/Users/kimiguel/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/run_adapter.py", line 18, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/Users/kimiguel/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/__main__.py", line 90, in main
    parents, result = run(toolargs, **subargs)
  File "/Users/kimiguel/.vscode/extensions/ms-python.python-2019.8.30787/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 36, in discover
    raise Exception('pytest discovery failed (exit code {})'.format(ec))
Exception: pytest discovery failed (exit code 4)

Could you provide me with a sample test file or repo? Additionally, I have a couple of questions:

• Do you use the base conda environment or a separate one?
• Do the tests get discovered correctly if you remove --black and/or --pylama from your setup.cfg file?
• Do you get any errors if you try discovering the tests in the terminal directly instead of through VS Code (path/to/conda/python -m pytest)

Thanks!

[hoonhout]

Hi @kimadeline,

Thanks for looking into the issue.

I just realised that I forgot to mention that I'm running inside a Docker container using the Visual Studio Code Remote - Containers extension. I did not test if the problem exists outside Docker, but I put a minimal (not) working example in this repo: https://github.com/hoonhout/issue-6994

If you open this repo in VSCode and reload the window in a DevContainer, tests will not be discovered unless you remove the options --black and --pylama from setup.cfg.

Bas

[kimadeline]

Thank you for providing more information and a sample repo, I could reproduce your issue using it 👍

📝 A few notes:

• the problem exists even when developing locally (can be reproduced outside of a Docker container)
• this issue wasn't introduced in 2019.8.29288, I could not discover tests when using --black or --pylama with version 2019.6.24221 of the Python extension either.

[UncleGoogle]

I've got the same using --flakes

[pytest]
testpaths = tests
addopts = --flakes

• and also using https://pypi.org/project/pytest-pythonpath/

[pytest]
python_paths = src
testpaths = tests

[eleanorjboyd]

Hello! I have reviewed this issue and given the architectural changes of the testing rewrite, this issue should no longer exist on the rewrite. Therefore I am going to close this issue but please comment or open a new issue if you are still seeing a problem when you try this yourself!

To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!