search

ballerina-platform / ballerina-lang

The Ballerina Programming Language
https://ballerina.io/
Apache License 2.0
3.58k stars 737 forks source link

[Bug]: Issue with running new tests in ballerina-cli #42339

Open RadCod3 opened 6 months ago

RadCod3 commented 6 months ago

Description

As part of the first task of https://github.com/ballerina-platform/ballerina-lang/issues/42124, I added tests for my additions to ballerina-cli https://github.com/ballerina-platform/ballerina-lang/pull/42236. However, those tests do not seem to run on the github actions and when I tried running ./gradlew :ballerina-cli:test locally that too doesn't run them. Whats even more interesting is that they do indeed run when I comment out certain tests in the testng.xml file in ballerina-cli.

  1. Adding my tests after the existing ones like below, 
    
<suite name="commands-test-suite">
<test name="command-tests" preserve-order="true">
    <classes>
        <class name="io.ballerina.cli.cmd.AddCommandTest" />
        <class name="io.ballerina.cli.cmd.BaseCommandTest" />
        <class name="io.ballerina.cli.cmd.BuildCommandTest" />
                                  :
                                  :
        <class name="io.ballerina.cli.cmd.ToolCommandTest"/>
        <class name="io.ballerina.cli.diagnostics.AnnotateDiagnosticsTest" />
        <class name="io.ballerina.cli.diagnostics.DiagnosticAnnotationTest" />
    </classes>
</test>
</suite>
Does not run them
![image](https://github.com/ballerina-platform/ballerina-lang/assets/104646586/f5ba0491-d6e5-4fcb-8bf8-d048dd7dbf75)
https://scans.gradle.com/s/ph23glwnw5v4k/tests/overview?collapse-all&toggled=WyI6YmFsbGVyaW5hLWNsaTp0ZXN0LTAiXQ
Commenting out `TestCommandTest` in the testng.xml file runs my tests but they fail and so do some other tests.
![image](https://github.com/ballerina-platform/ballerina-lang/assets/104646586/7c9b5f4a-105f-4d41-80c5-8a5440724701)
https://scans.gradle.com/s/jghd5wnt3ykc2/tests/overview?collapse-all&toggled=WyI6YmFsbGVyaW5hLWNsaTp0ZXN0LTAiXQ
2. Adding my tests in a serparate test tag doesn't run them either,
: : 

Here too when you comment out `TestCommandTest`, it runs the tests I need but they fail.

Sometimes when I repeat `./gradlew clean :ballerina-cli:test --scan` it doesn't run any of the tests as well.

In both of the above scenarios however, running the tests I wrote by specifying them using `./gradlew clean :ballerina-cli:test --scan --tests "io.ballerina.cli.diagnostics.AnnotateDiagnosticsTest"` runs them successfully.
https://scans.gradle.com/s/fhcamynpswjou/tests/overview

### Steps to Reproduce

_No response_

### Affected Version(s)

_No response_

### OS, DB, other environment details and versions

_No response_

### Related area

-> Other Area

### Related issue(s) (optional)

_No response_

### Suggested label(s) (optional)

_No response_

### Suggested assignee(s) (optional)

_No response_
RadCod3 commented 6 months ago

Found a workaround for this by including the newly added tests before the old tests like so,

<suite name="commands-test-suite">
    <test name="diagnostics-tests" preserve-order="true">
        <packages>
            <package name="io.ballerina.cli.diagnostics"/>
        </packages>
    </test>
    <test name="command-tests" preserve-order="true">
        <classes>
            <class name="io.ballerina.cli.cmd.AddCommandTest" />
                                      :
                                      :
            <class name="io.ballerina.cli.cmd.RunBuildToolsTaskTest"/>
        </classes>
    </test>
</suite>

Which runs the tests I added AnnotateDiagnosticsTest and DiagnosticAnnotationTest image