Sample integration test could use more substance to it

[jeffb-sfdc]

While the sample integration test does a great job of demonstrating how to set up the downloading of VS Code and setting up running the tests within VS Code, the integration test (https://github.com/microsoft/vscode-test/blob/main/sample/test/suite/extension.test.ts) which tests the sample extension (https://github.com/microsoft/vscode-test/blob/main/sample/src/extension.ts) doesn't have much substance to it - it validates that array.indexOf() returns -1, which doesn't really have much to do with validating the extension, and doesn't demonstrate how to drive the UI, and check & validate the state/contents of the UI after the test finished.

It would be very helpful if there were examples demonstrating how to implement integration tests (end-to-end/integration tests, not unit tests), and show how to drive the UI, and test/validate the UI for expected results.

Now, one might say, "it's easy, just..." 1) Add vscode.commands.executeCommand('helloworld.helloWorld') to the test 2) and then write a spy to verify that vscode.window.showInformationMessage() is called, and that it's called with "Hello World!"

This has some shortcomings:

1. Directly calling executeCommand() and passing mocked data (the "rest" parameter) which we mock ourselves is not adequate. We recently found a bug where VS Code passes bad data to commands. (see https://github.com/microsoft/vscode/issues/152993). Microsoft says it's not a bug, but it is for us - when we're passed an incorrect file path, a library we depend on fails, so we wrote code to get around this and we need to write tests which automate this end to end. For this reason, we want to issue an instruction to VS Code itself, and tell VS Code to "run the "ABC" command", and not directly call executeCommand()

2. We need to be able to drive the UI, and the driving needs to be performed at a high level - as if the user was typing themselves, and not programmatically by calling APIs or functions directly. Suppose our extension presents the user with a text prompt when the extension's command is invoked, and the user is prompted to enter in a string. We need to be able to write a test which waits for the text input prompt to appear, and then enter/input some text. Now suppose that when the command is invoked, that the user is prompted for input several times (to input several different values). We don't want to stub functions - we have a goal to make our integration tests be end-to-end, and be as if the user was inputing the data themselves, so this means no mocks/stubs.

What we want to do might not be aligned with how you intended integration tests to work, but...

1. if that's the case, it's hard to tell, because the sample test doesn't actually do much and doesn't test the extension
2. it would be helpful if there were several "real world" examples, so we (all VS Code extension developers, not just my team) could see the "best practice" approach to take.

Thanks,

Jeff

[connor4312]

1. I'm not sure what you mean by this... calling executeCommand is how you tell VS Code to run a command. The issue you referenced is designed behavior of canonical URIs in VS Code and is not related to the command system.
2. Is out of scope for this runner at the moment. You may look at https://github.com/redhat-developer/vscode-extension-tester as an alternative, for example

A more complete example would probably better live in https://github.com/microsoft/vscode-extension-samples

[jeffb-sfdc]

@connor4312 re. 1: I don't want to derail the issue that I posted and get off topic, but if curious, this is why calling vscode.commands.executeCommand() directly is different from invoking the command via VS Code's UI:

In extension.ts, change the HelloWorld callback function to:

    let disposable = vscode.commands.registerCommand('helloworld.helloWorld', (firstArg: any, secondArg: any, thirdArg: any) => {
        debugger;

        // The code you place here will be executed every time your command is executed
        // Display a message box to the user
        vscode.window.showInformationMessage('Hello World from HelloWorld!');
    });

Next, in package.json, add the command to the explorer context:

.
.
.
  "contributes": {
    "commands": [
      {
        "command": "helloworld.helloWorld",
        "title": "Hello World"
      }
    ],
    "menus": {
      "explorer/context": [
        {
          "command": "helloworld.helloWorld"
        }
      ]
    }
  },
.
.
.

Next, in extension.test.ts, change Sample test to:

    test('Sample test', async () => {
        debugger;

        await vscode.commands.executeCommand(
            'helloworld.helloWorld',
            {
                foo: 'myFoo',
                bar: 'my-bar'
            },
            'foo-bar-baz'
        );

        assert.strictEqual(-1, [1, 2, 3].indexOf(5));
        assert.strictEqual(-1, [1, 2, 3].indexOf(0));
    });

After compiling...

1. debug using the "Run Extension" configuration
2. After VS Code opens, (if not already open) open a folder which has files in it
3. select a few files in the explorer view
4. right-click and select the "Hello world" command

The debugger should break into the callback for the helloWorld command, and if you audit the parameters passed in to the callback, firstArg is a URI of the first selected file, and secondArg is an array of URIs of all selected files

5. Now, quit the instance of VS Code you are debugging, and go back to the main VS Code window
6. Change the debug configuration to "Extension Tests", and start debugging
7. After VS Code opens and the test suite starts running, the debugger should break into Sample test
8. Step through this, and run until you hit the debugger in the helloWorld callback
9. Audit firstArg, secondArg, and thirdArg.

In this case, the arguments being passed to the command are being constructed in the test itself, not in VS Code, thus hiding any bugs that might come up.

re. 2: Thanks! I'll check out vscode-extension-tester

re. residing in https://github.com/microsoft/vscode-extension-samples: Maybe... it looks like https://github.com/microsoft/vscode-test/tree/main/sample/test/suite ...is pretty much exactly the same as: https://github.com/microsoft/vscode-extension-samples/tree/main/helloworld-test-sample/src/test/suite ...so it kinda seems like this repo is redundant then.

Well, regardless, I think adding some "real world"-like examples either here or to vscode-extension-samples (and the boilerplate code that gets generated when running yo generator-code) would be better. The repo already has suite and suite2 (https://github.com/microsoft/vscode-test/tree/main/sample/test), perhaps more suites cold be added there.

[connor4312]

In steps 1-4, the command is being triggered from the context menu in the explorer view. When called in this way, certain arguments are being passed to a command. However, commands can be called from anywhere, even multiple places in the VS Code UI (depending on its package contributions) or other extensions (if they use executeCommand.) If called from other places, they can receive different arguments. Commands are "public functions" and VS Code doesn't enforce a single schema for any given command's arguments.