search

microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.29k stars 1.18k forks source link

Proposal: Enable separate configs per run/discovery/debug with testing #21845

Open eleanorjboyd opened 1 year ago

eleanorjboyd commented 1 year ago

The following outlines a proposal for how to handle testing args.

Submitted issues related to this problem:

Current design: Currently, to pass any command line args for test run/discovery via the extension there is a single setting per test type (one for unittest, one for pytest). The two settings are python.testing.pytestArgs: [] and python.testing.unittestArgs: [] which take a string array of command line args. The args are then processed by the extension to remove some not applicable to the current action (removing run args during discovery and discovery args during run) but this is buggy and hard to maintain. Additionally to debug a debug config can be created in the launch.json.

Issues with current design:

  1. With only a single place to put args for each test type, a user cannot distinguish between which command line args they want to include during test run, discovery, coverage, or debug.
  2. Since pytest does not allow some args to be included during with the arg --collect-only (used to distinguish run from discovery by the extension) these args are parsed out by the extension before sending the request. This means custom args are removed from test args and generally it creates a buggy experience that is hard to maintain with the breadth of args users would like to pass into their tests.
  3. There is no straightforward solution to providing environment variables while running and discovering tests.

Proposed new design: Create new custom launch configs that can be setup for test run and discovery. Also, increase compatibility with debug launch configs to make sure these work in tandem. The launch config would be set up something like this:

    {
      "name": "Python: pytest run",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    },
    {
      "name": "Python: pytest discover",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "VAR": "true"
      }
    }

The two particular fields "args" and "env" would provide functionality useful to users. The rest would be default values.

The new launch config types that would be added would be:

  1. Python: pytest run
  2. Python: pytest discover
  3. Python: unittest run
  4. Python: unittest discover The names of each one are up for discussion, please provide feedback below!

Next a way to set up these launch configs would be introduced to the extension. The initial steps would be:

image

add configuration -> python -> python config menu Two possible options for how to display the testing config options would be: A: to have another submenu called something like "testing" which, when opened, lists the 4 config options above B: have the 4 config options sit in the current python submenu.

(I think option A is better for clarification even though it requires one more menu clickthrough).

additionally the button "configure tests" found on the test explorer would be changed so it routed the user to a menu of the launch config options for testing (similar to option A).

image

Concerns with proposed design:

  1. The use of launch configs may be harder for new / beginner users.
  2. The discoverability of launch configs is harder.
  3. The transition from a setting to using a config could be complicated/confusing for current users.

Possible ways to address above concerns

  1. help beginner users a. Increase documentation and provide specific examples of launch configs. b. provide default values for all fields in the launch config (this is already planned so the launch config should be runnable upon creation without edits meaning no args or env vars need to be provided)
  2. discoverability a. There is already a larger discussion happening on how we can improve this, switching to configs for testing provides even more reason to tackle this problem b. switching the button "configure tests" to the new config flow is very discoverable since that is already the workflow
  3. transition a. We can have a deprecation notification regarding the setting and show it to all those still using the setting for test args b. It should be simple to take someone's test args in their settings and turn that into a launch config so we could offer a button that handles this for the user to switch them over (drawback is we would be setting discovery and run to the same args but the user could see this then go change it)

Notes on Debugging: Less investigation has been done into test debugging as we already offer a similar launch config. Further analysis should be done to see what is missing in this flow and how to improve it to align with test configs once a design is decided on.

Action Items: We are looking for feedback on this proposal so please comment below any thoughts. Thanks!

github-actions[bot] commented 1 year ago

Thanks for the feature request! We are going to give the community 60 days from when this issue was created to provide 7 👍 upvotes on the opening comment to gauge general interest in this idea. If there's enough upvotes then we will consider this feature request in our future planning. If there's unfortunately not enough upvotes then we will close this issue.

roblourens commented 1 year ago

The "name" of a launch config is for the name that the user picks, it shouldn't be used to change the behavior of the launch config. Also what does "discovery" involve? We shouldn't use launch configs for anything that isn't basically running/debugging a program.

roblourens commented 1 year ago

Another suggestion, I don't know how much complexity is behind the "python-pytest" debug adapter, but if the point is just to invoke the python DA with some args, another option would be to add a config snippet to help users figure out how to write their pytest launch configs. An example of this is the snippets that we have for js-debug to help users write launch configs that start an npm script.

eleanorjboyd commented 1 year ago

Yes, I forgot to update the "name" of the launch config to represent that as a user input and instead focus on using "type" to change the behavior.

discovery involves running pytest discovery. It is equivalent to running "python -m pytest --collect-only" from the command line. So it is running python code if that is the concern.

I will look into the js-debug snippet! Thanks

eleanorjboyd commented 1 year ago

adding https://github.com/microsoft/vscode-python/issues/18778 to the related issues as this discusses limitations with current setup.

roblourens commented 1 year ago

discovery involves running pytest discovery

Yes- and what is pytest discovery? 😄

and instead focus on using "type" to change the behavior.

I would say this also isn't what I expect- "type" should indicate which debug adapter is being used, and this should just be "pytest" or even "python" or something like that, and then there should be some other argument that changes the behavior.

roblourens commented 1 year ago

In that issue, it sounds like it's discussing limitations with launch configs, not settings? I don't understand what Karthik means when he says

We currently only read this from the launch.json. We could expand the scope of where we read it from, but that has some things that we have not decided yet.

eleanorjboyd commented 1 year ago

in https://github.com/microsoft/vscode-python/issues/18778? I tagged it since resolving this args issue would then remove the setting entirely so it would simplify this process since the launch config would be the only place to reference. I mis-attributed it to "limitations" it is more like this issue would provide clarity to the issue brought up in #18778.

eleanorjboyd commented 1 year ago

Yes- and what is pytest discovery? 😄

During discovery pytest goes through and discovers all files that fit the file format for test files and finds tests within those files (both test classes and just functions). It also should read markers to determine which tests to return. So it isn't executing any of the users test code.

I would say this also isn't what I expect- "type" should indicate which debug adapter is being used, and this should just be "pytest" or even "python" or something like that, and then there should be some other argument that changes the behavior.

gotcha yeah that makes sense

roblourens commented 1 year ago

During discovery pytest goes through and discovers all files that fit the file format for test files and finds tests within those files (both test classes and just functions)

This doesn't sound like a good match for a debug config then, if I'm not going to be setting breakpoints and debugging my code. It sounds more like a task perhaps? Or just something that the extension does, configured by some settings.

eleanorjboyd commented 1 year ago

@brettcannon, @karthiknadig, @luabud : thoughts on @roblourens comment?

luabud commented 1 year ago

The thing is that to configure the debugger to debug pytest tests, one needs to create a launch.json file. Given that, I think it makes sense to migrate how users can configure how to run pytest tests in launch.json too. But the whole issue is that users would like to pass different configurations for pytest discovery, which needs to be done prior run/debug. If the config for running and debugging pytest tests are in launch.json, isn't it a bit weird to have the configuration for discovery elsewhere?

luabud commented 1 year ago

The other thing to call out is .env files -- using launch.json allow users to have different pytest configs with different environment variables per run, which is harder to do with settings

brettcannon commented 1 year ago

I guess my question is is launch.json is just for debugging why isn't it named debug.json? 😉 I personally view launch.json as the place to configuring launching your code which is traditionally for the debugger. As well, I views tasks.json as a way to run something in the terminal that we don't have extension support for and to parse the output (which this isn't doing; it's directly tying into things).

The other thing is that while you might not use the debugger every time, you may want to use the debugger when running the tests, and so it isn't a disconnect, I think, to use launch.json this way. And since discovery and running are tied together, I think putting one somewhere else just because you typically don't debug with it seems like we're putting purity over practicality. I.e., what Luciana said. 😁

Do we know how other testing extensions handle configuration for debugging compared to running? And we can bring this up in the Pacific stand-up if necessary.

roblourens commented 1 year ago

I still don't really understand what discovery is or what the user flow is for this scenario. I haven't used a testing framework that has discovery as a separate step. What is the output of discovery?

So if I want to debug my tests, I'm going to run a "discovery" launch config, wait for that to finish, then run the "run tests" launch config?

roblourens commented 1 year ago

different configurations for pytest discovery, which needs to be done prior run/debug

Sounds like a prelaunch task?

brettcannon commented 1 year ago

I still don't really understand what discovery is or what the user flow is for this scenario. I haven't used a testing framework that has discovery as a separate step. What is the output of discovery?

What tests are known to pytest. We need to do this to get the list of tests to display in the test explorer. Otherwise we would have to execute the tests in order to know which tests exist. How does js-debug know what tests to list in the test explorer?

So if I want to debug my tests, I'm going to run a "discovery" launch config, wait for that to finish, then run the "run tests" launch config?

You won't directly, but yes because we need to know about the test in order for you to be able to run it, debugger or not. Now if you're asking about when you run your tests after we know the test exists, then the discovery step is skipped as we already have a way at that point to tell pytest "run this specific test you told us about previously".

Sounds like a prelaunch task?

If by "prelaunch" you mean "before running tests", then yes. But if that's a push for using tasks then I still think that disassociation is going to confuse people. To a Python developer, test discovery is just a flag/command to the test runner, not a separate task (it essentially a dry run of tests where you are saying, "what tests can you find?", and skipping the actual execution of the tests).

roblourens commented 1 year ago

How does js-debug know what tests to list in the test explorer?

There's no connection between js-debug and the test explorer. Our self-host test provider that we use for the vscode repo https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-selfhost-test-provider creates a filewatcher for our test files, and parses them when they are added or changed. It's automatic there, I don't know about other test provider extensions. I wouldn't expect to run a launch config to populate the test explorer, that feels backwards.

I didn't realize that this is about the test explorer. So is the expected user flow like this?

  1. Run the "discovery" launch config
  2. Tests show up in the test explorer
  3. Debug a test

And does 3 happen via the Test Explorer UI or the proposed Python: pytest run launch config?

brettcannon commented 1 year ago

parses them when they are added or changed.

pytest and unittest handle the test discovery, so we need to have those tools tell us what they find.

  1. Run the "discovery" launch config

We run the discovery launch config; there's no manual step here beyond any custom configuration from the user.

This issue is just about configuration; the user doesn't perform any extra action for testing because of this and this is only for special cases where users need to specify extra details.

roblourens commented 1 year ago

We run the discovery launch config

Launch configs are usually for users, they don't get run automatically in the background by an extension. When you do that, the debug toolbar will show up and the statusbar will turn orange and that will be spooky if it happens by itself.

eleanorjboyd commented 1 year ago

another idea @luabud suggested would be something like this:

    "inputs": [
        {
            "id": "fileArguments",
            "description": "The arguments to send to the debugger",
            "type": "promptString",
            "default": "",
        }
    ],

    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "console": "integratedTerminal",
            "request": "launch",
            "program": "${file}",
            "args": "${input:fileArguments}",
            "justMyCode": true
        }
    ]
roblourens commented 1 year ago

Had to type this out twice after it got lost in the github.com ether yesterday, here goes. Still thinking this through but here's some thoughts-

I hadn't realized that you can already define some config in launch.json for debugging tests in the test explorer. I see that in https://code.visualstudio.com/docs/python/testing#_debug-tests you can tag a launch config with "purpose": ["debug-test"] and it gets used for debugging from the test explorer. But it gets to me a bit that the docs suggest adding a config that isn't a "real" debug config since it can't be run by the debug UI on its own, and isn't hidden from the UI. Do you pull all the properties from the launch config in this case? I get why it would be annoying to duplicate all of that if this was going to be configured somewhere else. But isn't it also true that half of those properties are not going to do anything in the test explorer? name, request, program, and console seem like basically boilerplate that you need just to fit the pattern of a launch config, and I would also find that confusing as a user. This and having to hide the launch config from the UI shows that you are using a launch config for something other than its intended purpose. It's also confusing that you can have multiple launch configs tagged with purpose but there's no way to know which one is being used, which makes launch.json seem like it's the wrong shape for this config anyway.

You are also reading and writing the launch.json file on disk directly, right? I think extensions shouldn't be doing this with launch.json, or settings.json, or other files that are managed by vscode. I think that was a disconnect in our conversation, regarding what types of configuration extensions should store in launch.json- they shouldn't store anything in launch.json. You should interact with the features via extension API. Working with the files directly could lead to complications like not handling launch configs in workspace files or launch configs from settings properly (you can actually define launch configs in settings.json via the launch setting key) and vscode could change how it stores launch configs.

Using the purpose flag for test configuration would be a nice power-user feature if I could have one config that could run as a normal launch config, but also contribute debug config to the test explorer, but this seems impossible. If I add "module": "pytest" and run it in the test explorer I get errors about it being invalid, is that right?

Overall I really think that settings.json is the right place for extension configuration. I think it's confusing to have extensions storing things in launch.json that are not actually launch configs. Eleanor found an example of how the Java test extension stores test configs in settings, and that looks like a good pattern to follow, to me. I'm fine keeping what you already have with the purpose flag, since it already exists, but I think it would work better for the primary configuration source to be in settings.

connor4312 commented 1 year ago

Before reading Rob's comment and the docs, I actually arrived on something similar to the existing "purpose" -- and I'm curious why that isn't sufficient for this use case.

I think such a launch config could be a "real" config that runs/debugs all tests, and not hidden from users. You could short circuit the direct usage of that configuration in your DebugConfigurationProvider to start a test run so that test results are reported to the UI in the standard way. Perhaps you would only want a subset of options to be supported in that config, or use a different debug type with different options. Ideally configuration provided in that type would be enough for discovery and execution of tests, rather than having two different launch configs.

Also, you could support multiple test-purposed launch configs by adding them as different TestRunProfiles.

roblourens commented 1 year ago

If you expect this launch config to also configure test running/debugging when you do it via the test explorer UI, then the extension is still reading configuration from launch.json, and I don't really like that.

Ideally configuration provided in that type would be enough for discovery and execution of tests, rather than having two different launch configs.

I don't know whether having discovery args in that launch config makes sense, because discovery happens before running tests, and you can only have one active discovery config.

yawhide commented 11 months ago

Proposed new design: Create new custom launch configs that can be setup for test run and discovery. Also, increase compatibility with debug launch configs to make sure these work in tandem. The launch config would be set up something like this:

will envFile still be an option in these launch configs so that we dont need to hardcode env vars?

not sure if this belongs in this issue or a separate issue, but how would this support having overrides to envFiles? eg: I have a .env.dev and .env.test file committed in my repository. this is the 'default'. then developers can specify their own .env.dev.local and .env.test.local to override specific env vars from .env.dev/.env.test. .env.dev.local and .env.test.local are not checked in to the repository.

brettcannon commented 11 months ago

will envFile still be an option in these launch configs so that we dont need to hardcode env vars?

Very likely. I would assume we will effectively duplicate most of what's available in launch.json (hence why we are thinking of putting things there to begin with).

not sure if this belongs in this issue or a separate issue

Don't worry about it; it's fine sitting here while we discuss where to place the settings.

how would this support having overrides to envFiles?

There isn't a proposal for that right now as that hasn't been an explicit ask before (to my knowledge).

BrettMoan commented 11 months ago

I appreciate all of y'all looking into how we can make this more configurable.

I wouldn't find it odd to have testing configs inside the settings.json.

If I had to choose between simplicity and less configuration vs some complexity and more control, I would be in favor of the more complex and flexible configuration.

Specifically I think the best exposing multiple test configurations would be a huge lift for developers where the workspace is either a mono-repo, or simply a repo that includes multiple different python components that have their own standalone libraries (ie project that has a folder for their cron jobs, and a different folder for steaming)

Currently, because their is only one configuration object available anytime I want to run tests I have to do two things for the existing configs available:

It would be nice for theses options to be nested in a list, allowing me to have multiple options

And for part of the configuration to give the python interpreter.

Given that the interpreter selection is actually determines more than testing(such as linking and auto completion, etc) the testing configuration might not be the appropriate place to define the interpreter, and maybe that's a different setting entirely.

Right now selection of the python interpreter isn't something that's saved into settings.json, in fact I don't even know where that value gets saved and persisted and have no idea how to set the path other than via the bottom bar or the select python interpreter command.

Multiple python interpreters for different parts of the same project may have its own headache I imagine and maybe just isn't feasible, but I did want to call out how currently the selected python interpreter does interact with the test run/discovery despite not having anywhere where you can actually type the path for the executable.

eleanorjboyd commented 11 months ago

trying to write down a clear list of requirements for this new design and what we want its capabilities to be. I think if we can get to a final list of what we want the user to be able to do, it then might be easier to define how the user does so and how it looks.

Requirements for MVP (minimal viable product) of new test arg config:

Requirements for the Capabilities of the JSON testing config:

Requirements for Debugging

Other Requirements:

Python folks, what am I missing here or want is not a requirement in your option (both for MVP or for final design in general).

eleanorjboyd commented 11 months ago

Now in terms of UX and functionality, here is a potential design option. It is full of a lot of questions / different ideas on configurations. Precidence should be used to determine the best design but in grey areas I was not so sure how to use our current design (like python debug config) to fit a new need.

settings.json design

{
      "name": "run config: unique name", // user defined name, would have to be unique if we use it elsewhere to reference this config
      "args": ["-v"],
      "env": {
    "PYTHONPATH": "${workspaceFolder}",
        "DJANGO_TESTING": "true",
    "DJANGO_SETUP_SCRIPT_PATH": "/path/to/file",
      },
       "envFile": "path/to/file",

      // idea 1.1: use type & purpose
         "type": ["python-unittest", "python-pytest"],
    "purpose": ["discovery", "run"],

    // idea 1.2: use type & testKind
    "type": ["discovery", "run"],
    "testKind": ["unittest", "pytest"],

    "request": "launch", // is this needed for test configs?
},

"python.testing.pytestEnabled": false,
"python.testing.unittestEnabled": false,
// Question 1.3 
// should there be one value for default config?
"python.testing.defaultConfig": ["run config: unique name"],

// or should there be one per test library
"python.testing.pytestDefaultConfig": ["run config: unique name"],
"python.testing.unittestdefaultConfig": ["run config: unique name"],

// or one setting & multiple values there and we select 
// the one with the correct purpose/test kind in config
"python.testing.defaultConfig": ["run config: pytest", "run config: unittest"],

// or should default config be defined in the config like so
{
      "name": "user config option 2",   
      "args": ["-v"],
    "defaultConfig": true,
      .....
},

launch.json design

{
  "name": "Python: Debug Tests",
  "type": "python",
  "request": "launch",
  "program": "${file}",
  "console": "integratedTerminal",
  "justMyCode": false

    // Question 2.1: where to select which kind of run (options debug or run)?
    // Question 2:2: should the user be able to set just a run config from here so it shows up in the menu
    // Question 2.3: where to set which config to run?
    "purpose": ["debug-test"],
    "purpose": ["run config: unique name"], // see name is from above

    // Question 2.4: Should the user be able to define variables they can also define in
    // the settings.json? Can we restrict this option?
    "env": "", 
    "args": "",
    "envFile": "",
}
{
  "name": "Python: Run Pytest Tests",
  "type": "python",
  "request": "launch",
  "program": "${file}",
  "console": "integratedTerminal",
  "justMyCode": false

    // idea 2.5: use purpose & type values
    // question 2.6: should there be a purpose to just run test?
    // question 2.7: should purpose `debug-in-terminal` still exist?
    "purpose": ["debug-test"],
    "type": ["run config: unique name"]
}

Possible UI design: ( this would not be required for the MVP but could be the goal of what we want the final experience to be like)

run and debug panel allows for the same behavior with defined run & debug configurations shown in the dropdown

Screenshot 2023-10-13 at 11 17 41 AM

This is just an idea, but I think it would be a great addition if Test Explorer had a dropdown that allows the user to select which config they want to use. (this is just a quick mockup, I think the look is a bit smushed so we would have to give it more thought if we went for something like this). new_test_explorer

connor4312 commented 11 months ago

Some feedback reading through:

roblourens commented 11 months ago

"python.testing.pytestEnabled": false,

2.1: If it's in a launch.json, it should be debugging by default 2.2: I don't understand what you mean 2.3/2.4: I know you said

users should not have to repeat their args or environment configs in multiple places in VS Code

but I agree with Connor and if I was doing this I would probably leave out this part of referencing settings.json configs for simplicity. I wonder how complex the configurations typically are? But if this is really important, I think the pattern is fine. "purpose" doesn't quite mean the right thing to me. I would suggest "extends", implying that it will use that settings.json config as a base, but you can override properties from the settings.json config by setting them in the launch config as well.

2.6: There is a builtin command "Run without debugging", it starts your launch config with the "noDebug" property added dynamically, I would suggest building on that.

brettcannon commented 11 months ago
  • Are these settings an alternative to labelling the configs as pytest/unittest or are they doing something else?

Something else; they already exist as settings for the extension to control which testing framework you are using and thus which one you want us to integrate with.

eleanorjboyd commented 11 months ago

Some feedback reading through:

  • If your configs map to TestRunProfiles, you don't need to have settings for the default configs, as these are handled/persisted by the editor. That also solves some things like "where to set which config to run?"

Yes this looks perfect!

  • Can pytestEnabled/unittestEnabled be covered by the presence of the associated run config? / are there cases where I could have one but not the other?

I think this is possible that the configs could have an associated pytest/unittest with each and it runs the given one. So if you select "run profile A", somewhere in the config for "run profile A" you defined it runs with "pytest" so then anytime you attempt "run profile A" it selects "pytest". @brettcannon or @luabud is this too hard for users to do or to discover? I am unsure of how often people want to switch between "unittest" and "pytest" and therefore how important this consideration is.

  • I would try to minimize how much the user needs to cross-reference between settings.json/launch.json, especially since I don't think you can do much with intellisense as an extension there

Yeah I think this is a very good point. If a user presses the "debug" button on the test explorer with a given set of args / env vars what is the difference then them create a config in their launch.json with the same args/env vars and running it there? Wondering if we just allow the option to use launch.json to continue as it is and we focus on just expanding the test explorer.

  • Is it the most common case that a user will use the same configuration to both run and debug? If so, are there design decisions that can be made to simplify that path?

Yes I think so. I think it would be good since you could set a different default per run / debug if we maybe just offer every run config as an option on the debug config list. Like if the person creates a config "profile 1" they will see that in both the "run" and "debug" dropdowns? Or do we think people will find this cluttered / not enough control? I guess we could also always add an arg that is like "show for: debug, run" and the default is both but could be changed.

eleanorjboyd commented 11 months ago

The dropdown looks great! Sorry I missed this earlier!!! (also idk why it isn't letting me "reply" like above to your comment, the formatting was getting messed up)

For others here is how it could look with the dropdown:

Screenshot 2023-10-18 at 2 43 15 PM Screenshot 2023-10-18 at 1 59 49 PM
eleanorjboyd commented 11 months ago

replies to @roblourens comments:

2.1: If it's in a launch.json, it should be debugging by default 2.2: I don't understand what you mean 2.3/2.4: I know you said

Debug by default makes sense, that solves 2.3 as well. 2.4 was just about duplicating settings so I just more wanted to repeat that question there

but I agree with Connor and if I was doing this I would probably leave out this part of referencing settings.json configs for simplicity. I wonder how complex the configurations typically are? But if this is really important, I think the pattern is fine.

I think its common to have a few environment variables and args but maybe @luabud can speak more to this question.

"purpose" doesn't quite mean the right thing to me. I would suggest "extends", implying that it will use that settings.json config as a base, but you can override properties from the settings.json config by setting them in the launch config as well.

yes this sounds fine to use "extends" if we do want them to build off each other.

2.6: There is a builtin command "Run without debugging", it starts your launch config with the "noDebug" property added dynamically, I would suggest building on that.

this is great ya, no need then for 2.6

brettcannon commented 11 months ago

I think this is possible that the configs could have an associated pytest/unittest with each and it runs the given one. So if you select "run profile A", somewhere in the config for "run profile A" you defined it runs with "pytest" so then anytime you attempt "run profile A" it selects "pytest". @brettcannon or @luabud is this too hard for users to do or to discover?

I would expect users to never swap between them once they have chosen their test runner.

For Rob and Connor's edification: pytest can run pytest and unittest testing code (i.e., be a test runner for anyone), but unittest can only run unittest testing code, hence why users even have to specify anything (and this sparks an idea I'm going to go talk to Eleanor about 😁).

eleanorjboyd commented 11 months ago

Following this round of comments, here is my new proposal on the design.

settings.json design

{
      "name": "run config: unique name", // user defined name
      "args": ["-v"],
      "env": {
    "PYTHONPATH": "${workspaceFolder}",
        "DJANGO_TESTING": "true",
    "DJANGO_SETUP_SCRIPT_PATH": "/path/to/file",
      },
       "envFile": "path/to/file",
       "testRunProfileKind": ["discovery", "run", "debug"] // all run kind will also display under debug, but not vice versa 
},
"python.testing.framework": ["unittest","pytest"] // this is NOT required, the default will be used if not included

therefore the simplest config if you do all the most basic setup steps with pytest for example would be:

{
      "name": "pytest discovery config", 
      "args": [], // this might even be optional too- open to suggestions on if this is a required part of the json.
       "testRunProfileKind": ["discovery"]
},

default values

launch.json design Launch.json configuration for debugging testing will stay exactly as it is now with the current functionality and design. Users will NOT be able to reference their settings.json test configurations from their config.json file. They will need to duplicate / rewrite any configurations they want to use as a debug config there. This will cause duplication of work but I believe will be the simplest for the MVP and I predict users will tend to stick to a single way of use that will make sense to them. @brettcannon, @luabud, @karthiknadig let me know your thoughts on this as I most unsure about this decision and not sure it impact down the road.

Possible UI design: Will use TestRunProfiles and their related UX. I will update this issue shortly with the flow of how users will set up testing once we switch. Additionally @connor4312 would it be possible to get a dropdown the same as on the "run" and "debug" icons on the reload icon to allow us to configure different profiles of kind "discovery" that could appear here? With the same option to set a default? image

brettcannon commented 11 months ago

If we can't unify in a single file then I am afraid duplication is just unavoidable. We can operate under the assumption that if you're debugging a test it's a single test and thus going to be a bit specific rather than generic to your whole test suite.

One question about "python.testing.framework" is whether that should be global or should we make a per-config setting (thinking of the monorepo case)?

cameronbrill commented 6 months ago

hi @eleanorjboyd! I love this proposal, it would fix an issue my org has with our test suite (we use pytest-xdist to parallelize test runs, but this breaks when we try debugging tests in VS Code). Is there a timeline for when this will be supported?

brettcannon commented 6 months ago

Is there a timeline for when this will be supported?

Eleanor is busy w/ non-test stuff this month, but the plan is to pick it back up in April.

alenzo-arch commented 5 months ago

+1 for this. The collection thing has been really frustrating. Not just that it doesn't work with some args, but also that if you have coverage enabled like so, it overwrites the coverage report. Not critical but it is annoying if you have it setup to discover on save.

The debugging portion of this is a deal breaker. It's working again now, but my issue #22820 is an example of how the current solution is brittle. 

"python.testing.pytestArgs": [
        "--cov",
        "--cov-report=html",
        "-vvv",