Pytest discovery in python code

[DonJayamanne]

(related to #3911)

• [ ] add Python script to with basic CLI to handle pytest discovery
• [ ] call pytest API to do discovery
• [ ] serialize (JSON) results into minimal data structure
• [ ] write to stdout
• [ ] plug into extension code (ts) (see #4035)

Proposed Minimal Output Format

[
    testRoot: string;
    folders: [
        {
            name: string; // folderName relative to `testRoot`.
            package?: [
                {
                    fileName: // test file name
                    testSuite?: [
                        {
                            name: string; // name of test suite
                            lineNumber: number; // line number suite definition exists on (1-based)
                            parentTestSuite?: string[] // if this is a suite within another suite, the immedate parent
                            functions: [
                                {
                                    name: string; // name of test function
                                    lineNumber: number; // line number function exists on (1-based)
                                }
                            ];
                        }
                    ]
                    functions?: [
                        {
                            // see definition of `testSuite:functions` above
                        }
                    ]
                    errors: string[]; // any errors found when discovering this specific package
                }
            ]
        }
    ] 
]

current pytest output:

# python -m pytest --collect-only -q

test_foo.py::test_a
test_foo.py::test_b
sub_a/sub_b/test_sub_b.py::TestFooSub2::test_value_below_one_hundred
sub_a/sub_b/test_sub_b.py::TestFooSub2::test_value_above_zero
sub_a/sub_b/sub_c/test_sub_c.py::test_something_odd
sub_a/sub_b/sub_c/test_sub_c.py::TestSomethingSomething::test_something_even
testsub/test_foo_sub1.py::TestFooSub1::test_foo_sub1_method1
testsub/test_foo_sub1.py::TestFooSub1::test_foo_sub1_method2
testsub/test_foo_sub1.py::TestFooSub1::TstFooSub1_A::test_foo_sub1_method2

serialized:

{
    testRoot: "/home/user/dev/workspace",
    folders: [
        {
            name: ".",
            testFiles: [
                {
                    fileName: "test_foo.py",
                    functions: [
                        {
                            name: "test_a",
                            id: "test_foo.py::test_a"
                        },
                        {
                            name: "test_b",
                            id: "test_foo.py::test_b"
                        }
                    ]
                }
            ]
        },
        {
            name: "testsub",
            testFiles: [
                {
                    fileName: "test_foo_sub1.py",

                }
            ]
        }
    ]
}

[DonJayamanne]

@ericsnowcurrently Please note, the arguments to pytest will be passed into the python code by node.js (i.e. the extension),

I.e. if using code as follows pytest.main(["-v", "tests", "--collect-only"], ....) please note, the args will be passed in sys.argv in some form, and the python code is not to construct these arguments.

Documenting details to avoid confusions.

[ericsnowcurrently]

I've opened https://github.com/pytest-dev/pytest/issues/4850 to find out the recommended way to do just discovery using the API.

[DonJayamanne]

@ericsnowcurrently They provide the API for this, here's the sample code https://github.com/Microsoft/vscode-python/issues/3911#issuecomment-452364856

I've tried this, with the command line args we have today. We do not need to parse the command line arguments, just use what the extension provides and it will work, after all, that's what pytest.main expects, just args.

Later we can look at optionally refactoring the parsing of arguments (probably in python or even better, ..) I.e. using existing args, will work (as its already parsed accordingly).

[ericsnowcurrently]

There was already example code: https://github.com/Microsoft/vscode-python/issues/3911#issuecomment-452364856

[WSLUser]

I'm still having an issue getting pytest discovered from my Miniconda installation.

[sleighsoft]

Any progress on this? This #3936 is still not working.