Support run TestNG suites toward XML file

jdneo commented 5 years ago

The experience could be adding a command action into the file explorer context menu, like this:

Mohannad91 commented 5 years ago

Thanks, It would be a great feature that will help in debugging for automation testing frameworks written in Java. Are we getting this soon or not ?

jdneo commented 5 years ago

Hi @Mohannad91,

Do you mean in the context menu, you want another item called Debug Test?

Mohannad91 commented 5 years ago

Hi @jdneo .. yeah exactly what i meant.. it's a popular and important feature for automation engineers using java and TestNG to build their framework.. it'll make life way easier

dg-ceiland commented 5 years ago

@jdneo: While the "Run Test" context menu command would be handy, users can get equivalent functionality by right clicking a Maven project and choosing the test goal. To expand on the comments above, one thing that the VSCode extension can't do right now is run or debug at the class and method level using info specified in testng.xml.

Example: With TestNG, I can use the @Parameters decorator to specify that I'd like TestMethod1() to use a parameter called "username". Then, I can specify a value for "username" in a testng.xml file specified in the Maven project's pom.xml. I can also specify in the testng.xml file that I'd like to only run the TestMethod1() method next build. From there, I could run mvn test from the Terminal, only TestMethod1() would fire, TestNG would notice that I have a parameter called "username" specified for it in testng.xml, and that parameter would be plugged into my test method.

However, using the CodeLens "Run Test" or "Debug Test" action on TestMethod1() will result in a Skip by TestNG - by clicking on those, I'm running the class, not the whole project, so CodeLens doesn't know what "username" is. It does know that it's a @Parameter that's not specified in any file it knows about, so TestNG returns an error. Now, imagine this project has dozens or hundreds of test methods; running the whole project just to check one method is cumbersome, and editing my testng.xml each time I want to just run or debug one method is tedious.

A nice feature would be to automagically give Run Test and Debug Test visibility to the stuff specified in testng.xml. @Parameters would be one such example that's fairly straightforward. Getting more stuff out of testng.xml at runtime might be nontrivial and something you might consider splitting into subtasks.

jdneo commented 5 years ago

Hi @dg-ceiland,

Sorry for my late reply and thank you for the detailed explanation. Just as you have said, the extension currently has limited support on TestNG. And your suggestion is very inspiring.

I think I can first support running/debugging towards the testng.xml and then make the codelens(Run Test/Debug Test) more intelligent.

jdneo commented 5 years ago

Hi @all,

Now have one problem with this issue. The VS Code when API can only check the file name but has no ability to check the content. This limitation may introduce some problems here:

Add Run/Debug Test menu item for all *.xml file: This implementation can make sure that we won't miss any TestNG XML files in the project. But it will be confusing when the user right clicks on other XML files, like pom.xml.
Add Run/Debug Test menu item for all the XML files whose file name contains testng. By doing this, we limited the entry points but we may also miss some of the configuration files.

I prefer the first one, but I'm worrying about the confusion that may be introduced here. Do you guys have any thoughts on this?

dg-ceiland commented 5 years ago

@jdneo Just to confirm we're talking about the same thing (I don't believe we are), I've included a screenshot of the Run / Debug Test UI clickables that I'm thinking of.

Perhaps my suggestion is outside of the scope of this issue. In any case, it's tricky to say which of your options is better - option 1 would probably be my choice as well.

jdneo commented 5 years ago

@dg-ceiland I think you are talking about bringing the context defined in testng.xml to the Run Test & Debug Test actions. Am I right?

dg-ceiland commented 5 years ago

@jdneo Yes. Is this within scope, or even doable? If so, is this something that should be put into a different issue?

jdneo commented 5 years ago

@dg-ceiland Yes, I think it would be better to file this into another new issue.

Would you mind to file a new issue for that?

oliversundell commented 5 years ago

@jdneo Any updates on this? Would really want the ability to run testng tests from xmls, just by right clicking and "Run Test"/"Debug Test". Often have many tests in the xmls that run parallel. As it is done in Eclipse today.

jdneo commented 5 years ago

@oliversundell,

There is one concern that so far I cannot find a way to figure out testng specific xml. If I simply add the test/debug menu item to all xml files, this would be a little bit annoying.

oliversundell commented 5 years ago

@jdneo Yeah, understand. Is there any way to customize the context menu my self to achieve this (I can accept that I get the options on all xmls)?

jdneo commented 5 years ago

@oliversundell Your comment inspired me! So actually there is some ways to filter the unexpected run/debug menu items, for example:

When only the extension detects there is TestNG in class path, we show them
Or we can provide an extension setting to let the users specify whether they want to have them show in the context menu for the .xml files.

oliversundell commented 5 years ago

@jdneo Great! Think both options sounds good. I'm just eager to loose Eclipse and move my development of testng to vs :)

KrishnaSakinala commented 4 years ago

@jdneo - Is there any update on this thread. And can't you implement the same logic how eclipse and intellij idea is doing to run the testng.xml file?

CarlPer commented 3 years ago

Is there any progress on this issue, this feature would help me a lot

USTQE commented 3 years ago

It would be great to have the feature for test engineers to run BDD cucumber tests and a suite of test script execution as well. Please let us know when we can expect this changes :)

jfmenard commented 3 years ago

Any progress on this? Only missing part for me to move away from other IDEs.

joshzhuang1 commented 3 years ago

Same here. waiting for this to be implemented before i can dump Eclipse

GerobohamAlvarez commented 2 years ago

Any news about this issue ? or any work around to solve it ?

jfmenard commented 2 years ago

Really surprised this is not prioritized more. This is a pretty basic feature & a must-have IMO.

nickzhums commented 2 years ago

Hello folks, we are taking a look at this issue and evaluating the technical feasibility. Will keep you posted

jfmenard commented 2 years ago

Any update about this issue?

KipRM commented 2 years ago

I don't know how it may be stated that VSCode can support testNG when it cannot kick off by the test xml file.

abhaybharti commented 2 years ago

Any plan to add this feature. This is only piece which is making me use IntelliJ

wenijinew commented 2 years ago

"One of the reasons for commercial products can survive is that some popular features cannot be always implemented easily/quickly in FOSS products." -- Somebody

Workarounds to Run TestNG suite file in Vscode:

PROJECT/.vscode/launch.json (Run and Debug View to Start)

{
"version": "0.2.0",
"configurations": [
    {
        "type": "java",
        "name": "Run TestNG Suite",
        "request": "launch",
        "mainClass": "org.testng.TestNG",
        "args": "-d \"testng_output_dir\" -testnames \"test_names\" suite_file_path.xml",
        "vmArgs": "-Dname=value"
    }      
]
}

.vscode-server/data/Machine/settings.json (Maven Terminal Favorite Command - Maven: Execute Commands... -> Favorite...)

{
    "maven.terminal.favorites": [
        { 
            "alias": "Run TestNG Suite",
            "command": "clean test -Dname=value -Dsurefire.suiteXmlFiles=src/main/resources/suites/path/to/suite_file.xml -f /maven/project/path/pom.xml"
        }    
    ]
}

KipRM commented 2 years ago

"One of the reasons for commercial products can survive is that some popular features cannot be always implemented easily/quickly in FOSS products." -- Somebody

Workarounds to Run TestNG suite file in Vscode:

PROJECT/.vscode/launch.json (Run and Debug View to Start)
{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "Run TestNG Suite",
            "request": "launch",
            "mainClass": "org.testng.TestNG",
            "args": "-d \"testng_output_dir\" -testnames \"test_names\" suite_file_path.xml",
            "vmArgs": "-Dname=value"
        }      
    ]
}
.vscode-server/data/Machine/settings.json (Maven Terminal Favorite Command - Maven: Execute Commands... -> Favorite...)
{
    "maven.terminal.favorites": [
        { 
            "alias": "Run TestNG Suite",
            "command": "clean test -Dname=value -Dsurefire.suiteXmlFiles=src/main/resources/suites/path/to/suite_file.xml -f /maven/project/path/pom.xml"
        }    
    ]
}

DEV's IDE Use case. The developer, or automation developer may, and likely has any number of testNG xml files available for a test suite. In intellij you can right click these in the left pane files list and run them. Once they are ran they are placed up in a drop down up at the top of the IDE. It is for this reason, anyone using xml based testNG solutions would never use vscode due to a lack of these basic ergonomics.

No one's quote is going to change this.
It challenges the very value of the vsCode TestNG plugin.

wenijinew commented 2 years ago

Yes, right click is very good feature. However, in real world, not all the test suites can run simply by simple right click. For our cases, we have to configure tons of dynamical parameters(-Dname=value) or in properties file.

After configuring the TestNG suites, you can also have them in drop-down list: Run and Debug view or Maven Favorites (You can set shortcut for it - I used "Alt + F"). For my feeling, it's not perfect but really good enough to use.

benzman81 commented 2 years ago

For me, the integration into the UI is not only needed for starting tests, but also to integrate it in the UI with debugging and marked successful and failed tests in the test tree and the code with errors. That is the main benefit that vscode is currently missing on suite xmls compared to eclipse, for example.

karlkras commented 1 year ago

Wow, as great as vscode is, and this sits here like a huge sore thumb.

microsoft / vscode-java-test

Support run TestNG suites toward XML file #596