[Testing] continuous run integration issues

[connectdotz]

(cc @connor4312)

In vscode-jest, we have started integrating the autoRun feature with VSCode's Continuous Testing API and have encountered a few issues that I hope you can help us resolve.

We offer two autoRun modes:

• auto run (watch)
• auto run (onSave) - This mode allows users to customize triggers, targeting either only test files or both test and source files.

To support these, we added two new run profiles using vscode.TestRunProfileKind.Run with supportsContinuousRun set to true. The onSave profile also includes a configureHandler to let users choose file types. This setup, however, has resulted in some unexpected behaviors:

1. All three profiles appear in the dropdown menu of the top "Run" button, which is confusing as these additional profiles should be exclusive to the autoRun category and should not be visible under the regular run options.

   

2. Unlike the "Run" dropdown, the autoRun dropdown does not offer a menu for configuration or selecting a default profile, which seems inconsistent. Shouldn't the dropdown menus be consistent across all test action buttons?

3. In a multi-root project with multiple TestControllers and autoRun profiles, there is currently no method to view the status of an individual autoRun profile or to stop one specifically. The autoRun status button attempts to stop all autoRuns collectively and indicates a 'running' status if any of the profiles are active.

4. Lacking the ability to enable/disable continuous button on TestItem level. Not all test items can initiate auto-run.

Additionally, I have a question: Is there a way to programmatically trigger the autoRun with a specific profile? I tried using 'testing.startContinuousRun' and passing a profile to it, but it still pops up the profile chooser instead of executing the autoRun.

Any insights or suggestions you could provide would be greatly appreciated. Please let me know if there are adjustments or experiments you'd recommend to better integrate these features. Thank you!

[connor4312]

Autorun is a modality of a profile, it's not its own type of profile. I would recommend letting the user configure their mode/trigger under the "configure test profile" action.

I tried using 'testing.startContinuousRun' and passing a profile to it, but it still pops up the profile chooser instead of executing the autoRun.

That's present if there are multiple available continuous runs. I could make a change to allow passing a profile and/or TestItem to this command

[connectdotz]

@connor4312, thanks for the reply.

Autorun is a modality of a profile, it's not its own type of profile. I would recommend letting the user configure their mode/trigger under the "configure test profile" action.

To ensure I fully understand your suggestion, are you advising against creating separate autoRun profiles and recommending we enable the supportsContinuousRun option within the regular run profile instead, correct?

Additionally, I am curious why "coverage" isn't treated as just another "modality" like supportsContinuousRun, rather than as a separate profile. The coverage button behaves differently compared to the run/debug buttons; it lacks a dropdown menu for selecting a coverage profile, setting a default, or configuring options. Is this by design? How are users expected to choose which coverage profile to run or adjust its configuration? This question isn't directly related to the current ticket, but your insights would help us better understand the conceptual design of the system.

That's present if there are multiple available continuous runs. I could make a change to allow passing a profile and/or TestItem to this command.

That would be great. 👍

[connectdotz]

@connor4312, notice a new issue with autoRun, adding to the original post:

3. In a multi-root project with multiple TestControllers and autoRun profiles, there is currently no method to view the status of an individual autoRun profile or to stop one specifically. The autoRun status button attempts to stop all autoRuns collectively and indicates a 'running' status if any of the profiles are active.

thoughts?

[connectdotz]

@connor4312, I'm encountering another issue related to the display of the continuous run button at the test item level, which might be connected to #178973:

In our configuration, initiating auto-run for each test item isn't practical. Ideally, I'd like to disable this for all items except possibly the root item. Initially, I assumed these settings were driven by runTag, similar to other TestItem buttons, so I attempted to create a new auto-run profile with a different runTag. Unfortunately, this method was ineffective as the buttons were automatically applied to all items once any auto-run profile was created. This underscores the need for more granular control.

Moreover, I'm uncertain if using a profile tag to manage the TestItem's continuous buttons is advisable, particularly since you've indicated that auto-run is not a separate profile but rather a modality of an existing profile. Under this assumption, we would have a single run profile marked continuous=true that is applicable to all test items via matching tags. Therefore, it seems we might need a different mechanism to specify continuous run eligibility for each TestItem...?

I've added this issue to the original post above (4). As the issue list grows, I'm starting to worry if I have misunderstood the continuous run design. Your insights would be highly valuable and greatly appreciated.

[connectdotz]

@connor4312, is there any traction on passing a profile and/or TestItem to the 'testing.startContinuousRun' command? If not, are you open for a PR?