Running the test script doesn't seem to actually run the tests

[jeffb-sfdc]

When I run the test script (test/specs/basic.e2e.ts), I'm told the test runs, but it doesn't appear that the tests are actually running.

Steps to repro:

1. clone wdio-vscode-service
2. cd into wdio-vscode-service
3. run npm install
4. run npm run build
5. run npm run test (or run npm run ci)

Result: I got the following:

[0-0] RUNNING in chrome - /test/specs/basic.e2e.ts
[0-0] PASSED in chrome - /test/specs/basic.e2e.ts

Spec Files:  1 passed, 1 total (100% completed) in 00:00:19

Expected: I was expecting each of the tests in basic.e2e.ts to be listed. I think basic.e2e.ts has something like 48 tests in it, so I was expecting all 48 of them to be listed, and listed as passing.

6. I then created a new test file, /test/specs/another.e2e.ts, and added the following

   describe('another test suite', () => {
   it('this should pass', () => {
       expect(1).toBe(1)
   })
   
   it('this won\'t pass', () => {
       expect(1).toBe(2)
   })
   })

7. ran npm run build
8. ran npm run test

Result: I got the following:

Execution of 2 workers started at 2022-09-12T18:14:45.778Z

Found existing install in /Users/{my-user-name}/src/github.com/webdriverio-community/wdio-vscode-service/.wdio-vscode-service/vscode-darwin-1.71.0. Skipping download
[0-0] RUNNING in chrome - /test/specs/another.e2e.ts
[0-1] RUNNING in chrome - /test/specs/basic.e2e.ts
[0-1] PASSED in chrome - /test/specs/basic.e2e.ts
[0-0] PASSED in chrome - /test/specs/another.e2e.ts

Spec Files:  2 passed, 2 total (100% completed) in 00:00:18

Expected: Since another.e2e.ts has a test that should pass, and a test that should fail, I was expecting the results to tell me that one test passed and another failed.

I'm new to wdio-vscode-service and evaluating it, so if I'm missing a step or making a wrong assumption, please let me know.

[christian-bromann]

Hey @jeffb-sfdc , thanks for raising the issue.

What you see in the terminal is a highly summarised output. Basically it lists the spec files that were running and passing/failing. If you set a reporter, e.g. spec reporter, in your wdio.conf.ts you will see the expected output. Not sure why that one test is not passing. It definitely should. Maybe enabling the reporter gives more insights?

[jeffb-sfdc]

Hi @christian-bromann , re. summarized output, I tried this with both logLevel set to trace, as well as with it set to warn, but in both cases when I ran npm run test the output stated that both test suites passed (and since there is a failing test in another.e2e.ts, I expected some form of failure to have been reported).

re. setting the reporter to spec: the wdio.conf.ts that was pulled down when I cloned the repo, has this in it:

.
.
.
    reporters: ['spec'],
.
.
.

...and appears to already have the reporter set to spec. Or is there something else that needs to be configured?

[jeffb-sfdc]

@christian-bromann I found the issue. In wdio.conf.ts, line 195 there is:

        grep: grep.join('|'),

When this is removed, and mochaOpts is changed to:

    mochaOpts: {
        ui: 'bdd',
        timeout: 60000,
        invert: true
    },

...all of the tests run. One side effect I'm seeing, is that two of the tests in basic.e2e.ts ("should be able to find the explorer tree view" and "should be able to iterate over the (default) tree items") are failing.

[jeffb-sfdc]

@christian-bromann I created the PR, https://github.com/webdriverio-community/wdio-vscode-service/pull/43 to address this issue as well as a few others.