Argument filtering for pytest fails to handle custom arguments

[omartin2010]

Issue Type: Bug

I created a fixture in conftest.py that reads at input args to pytest (using pytest_addoption. See code below. When I try and add this option to my settings.json file so that my tests execute properly, it fails with the ERROR: file not found...

Here is my conftest.py file :

import pytest
import os

def pytest_addoption(parser):
    parser.addoption("--config_file", action="store")

@pytest.fixture(scope='session')
def config_file(request):
    config_file_value = request.config.option.config_file
    if config_file_value is None:
        print(f'config_file_value = {config_file_value}')
        pytest.skip()
    elif not os.path.exists(config_file_value):
        print(f'config_file_value = {config_file_value}')
        pytest.skip()
    return config_file_value

And then my settings.json relevant to this:

{
    "python.pythonPath": "/usr/bin/python3",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.cwd": "",
    "python.autoComplete.addBrackets": true,
    "python.testing.pytestArgs": [
        "--config_file app/config.json",
    ]
}

And finally the output of the console :

ython /root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /root/jetson-task-manager -s --cache-clear --config_file app/config.json
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.2, py-1.5.2, pluggy-0.13.1
rootdir: /root/jetson-task-manager
plugins: forked-0.2, xdist-1.22.1
collected 0 items

============================ no tests ran in 0.06s =============================
ERROR: file not found: --config_file app/config.json

Traceback (most recent call last):
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Running pytest version 5.4.2. Feels to me like this option is taken as if it were a file to test, versus an actual argument. Ideas? Am I missing something?

Extension version: 2020.4.76186 VS Code version: Code - Insiders 1.46.0-insider (d487078dc7fc1c276657cadb61b4f63833a8df55, 2020-05-07T16:19:54.327Z) OS version: Windows_NT x64 10.0.18363 Remote OS version: Linux arm64 4.9.140-tegra Remote OS version: Linux arm64 4.9.140-tegra

System Info

|Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (8 x 2112)| |GPU Status|2d_canvas: enabled
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
protected_video_decode: enabled
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
viz_display_compositor: enabled_on
viz_hit_test_surface_layer: disabled_off_ok
webgl: enabled
webgl2: enabled| |Load (avg)|undefined| |Memory (System)|15.93GB (7.08GB free)| |Process Argv|| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: jetson| |OS|Linux arm64 4.9.140-tegra| |CPUs|unknown (6 x 345)| |Memory (System)|7.68GB (0.66GB free)| |VM|0%| |Item|Value| |---|---| |Remote|Container taskmanager:latest (/taskmana…| |OS|Linux arm64 4.9.140-tegra| |CPUs|unknown (6 x 1420)| |Memory (System)|7.68GB (0.66GB free)| |VM|0%|

[karthiknadig]

@omartin2010 You have to split these into separate arguments. Right now it is treating --config_file app/config.json as a filename.

"python.testing.pytestArgs": [
        "--config_file", 
        "app/config.json",
    ]

[omartin2010]

if I do that, it doesn't work, I get this :

python /root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /root/jetson-task-manager -s --cache-clear --config_file app/config.json
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: unrecognized arguments: --config_file
  inifile: None
  rootdir: /root/jetson-task-manager

Traceback (most recent call last):
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Turns out I also have another argument that needs to be there which is the directory that contains the tests, so my args are :

    "python.testing.pytestArgs": [
        // "test",
        "--config_file",
        "app/config.json"
    ]

With or without the test argument (not having this line at all or putting it last), I get the same error..

[karthiknadig]

Looking at pytest's help i don't see a config-file but i see a -c. I don't think they have a expanded config file option.

> py -3.8 -m pytest --help | findstr config
                        configuration file raise errors.
  -c file               load configuration from `file` instead of trying to
                        locate one of the implicit configuration files.
test session debugging and configuration:
  -h, --help            show help message and configuration info
  --trace-config        trace considerations of conftest.py files.

Try this:

"python.testing.pytestArgs": [
        "-c",
        "app/config.json"
    ]

[omartin2010]

I think I may have been unclear. This is for my own, user defined parameter, one one for pytest itself, so that a fixture can retrieve it in this kind of a function. Here is an extract of my conftest.py file:

def pytest_addoption(parser):
    parser.addoption("--config_file", action="store")

so that it's passed to my pytest session. So I want this param to pass straight to my stuff. So far it works when I run this from the command line, just not when from vscode...

[karthiknadig]

Looks like the bug is in filterArguments for pytest: https://github.com/microsoft/vscode-python/blob/61b179b2092050709e3c373a6738abad8ce581c4/src/client/testing/pytest/services/discoveryService.ts#L46

We pass the config_file arguments there and we don't get back the expected arguments. In the case of :

        "python.testing.pytestArgs": [
            "--config_file",
            "app/config.json",
            "tests"
        ]

Expected:

python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file app/config.json tests

Actual:

python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file app/config.json
python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file tests

Not sure why it prints out the command twice with different arguments.

[Querela]

I have a similar issue, I also try to provide custom defined options to my tests and the discovery doesn't forward them correctly, so pytest raises an error and no tests will be found.

My setup is almost identical to the description above. In addition I allowed the config names to be specified via environment variables which is the only way to get it to work!

I had to default to "python.envFile": "${workspaceFolder}/.env", which worked in my case as I defaulted to environment variables if no explicitely provided. But not all tests will allow this...

# conftest.py
def pytest_addoption(parser):
    def environ_or_required(key: str) -> Dict[str, Any]:
        return (
            {"default": os.environ.get(key)}
            if key in os.environ
            else {"required": True}
        )

    parser.addoption(
        "--discord-token",
        type=str,
        help="discord bot token",
        **environ_or_required("DISCORD_TOKEN"),
    )
    parser.addoption(
        "--discord-channel",
        type=str,
        help="discord bot channel name or id",
        **environ_or_required("DISCORD_CHANNEL"),
    )

@pytest.fixture(autouse=True)
def env_setup(monkeypatch: pytest.MonkeyPatch, request: pytest.FixtureRequest):
    monkeypatch.setenv("DISCORD_TOKEN", request.config.getoption("--discord-token"))
    monkeypatch.setenv("DISCORD_CHANNEL", request.config.getoption("--discord-channel"))

// settings.json
{
    "python.pythonPath": "venv/bin/python",
    "restructuredtext.confPath": "",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--discord-token",
        "my-private-token",
        "--discord-channel",
        "Allgemein",
    ],
}

And one error message:

python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: the following arguments are required: --discord-token, --discord-channel

Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

I have to add that I have to restart VSCode else the changes in settings.json will not effect the current pytest discovery. With reloading, the pytest args are mangled ... I'm not sure how to provide them correctly ...

python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s --cache-clear --discord-token --discord-channel my-private-token
python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s --cache-clear --discord-token --discord-channel Allgemein
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: argument --discord-token: expected one argument

Configuring it like so also fails: (sometimes with a different message...)

    "python.testing.pytestArgs": [
        "--discord-token 'my-private-token'",
        "--discord-channel Allgemein",
    ],

python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s --cache-clear --discord-token 'my-private-token' --discord-channel Allgemein
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /mnt/c/work/project/python-transformer-discord-notifier, configfile: setup.cfg
plugins: cov-2.10.1
collected 0 items

========================= no tests collected in 0.07s ==========================
ERROR: file or directory not found: --discord-token 'my-private-token'

(Using terminal.integrated.env.linux for environment variables did not seem to work. Might be the wrong setting. Env file worked.)

Are others having the same issue?

[ralwing]

I'm also facing this issue. Passing arguments:

  "python.testing.pytestArgs": [
    "-s",
    "--result-logger", 
    "local"],

Triggers error:

Error: Process exited with code 4: ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --result-logger 
  test_simple_object_detection.py::test_simple_object_detection[roundabout_entrance]
  inifile: None
  rootdir: /home/gigi/workspace/test_harness/as_simulator_tests

[karthiknadig]

@ralwing that seems to be error coming from pytest itself. If you run that command manually in the terminal what happens?

[apassy]

Having a very similar problem: "python.testing.pytestArgs": [ "-rP", "--html=c:\Data\pytest-report.html", "--self-contained-html" ],

These are all ignored - not matter what I do with them, they do not effect running tests either with "run test" or "debug test"

window 10, vs code 1.58.2 pytest, pytest-html (among others)

[wkerzendorf]

same issue

[wkerzendorf]

@karthiknadig any movement on this issue. It cropped up in regards with another issue as well

[karthiknadig]

We are in the process of re-writing the testing adapter #17242 . It is actively being worked on in the current milestone.

[wkerzendorf]

@karthiknadig - as a fellow open-sourcer - happy to play guinea pig 😉 - thanks for your work

[binishkaspar]

Until a fix available, you can add these options on pytest.ini file,

[pytest]
addopts = --config_file app/config.json

https://docs.pytest.org/en/6.2.x/customize.html

[marq4]

So basically I can't dynamically choose between two different implementations like: python -m pytest --algorithm merge python -m pytest --algorithm quick To reuse the same tests?

[PaulWaldo]

Has there been any movement on this? I can't get any of the suggestions or workarounds to work (VS Code Version: 1.80.2)

[eleanorjboyd]

I created a proposal with a solution for this problem, would love your input on the above issue! https://github.com/microsoft/vscode-python/issues/21845

[elongstreet88]

I was able to use this format to get around it:

{
    "python.testing.pytestArgs": [
        "--mycustom-arg1=value1",
        "--mycustom-arg2=value2"
     ]
}