search

microsoft / vscode-test-cli

Command-line runner for VS Code tests
MIT License
23 stars 8 forks source link

Testing in isolation #57

Open nhedger opened 3 months ago

nhedger commented 3 months ago

Feature request

It would be nice to be able to run all tests in isolation.

Ideally, every test would receive a fresh VS Code instance, free of any previous state.

Use case

When running tests that affect the state of VS Code, it can be cumbersome to ensure that some actions have a direct link to a desired state. For example, if you want to test whether opening files of a given type activates the extension, if you run these tests one after the other, the first test will trigger the activation, and subsequent tests will succeed because the previous test has activated the extension.

connor4312 commented 3 months ago

You can do that by creating a .vscode-test file that has a configuration that runs each individual test. However, this is going to be quite slow, especially for large suites, and is better replaced by tracking state changes made in tests and resetting them in lifecycle teardown points.

nhedger commented 3 months ago

Hey @connor4312,

I've been doing that as a workaround; setting it up is a real pain.

Tracking changes and resetting would definitely work in some cases, but as far as I know, we cannot programmatically deactivate an extension, so that wouldn't work in the case outlined in my first message.

As a side note, mocha's afterEach and beforeEach don't seem to be available in the test environment as they seem to be only available when executing mocha on the CLI, which makes this even harder.

Anyway, just thought I'd leave this here as food for thought!

connor4312 commented 3 months ago

afterEach/beforeEach are definitely available, but note that in the default tdd mocha UI they're called setup and teardown https://mochajs.org/#interfaces

nhedger commented 3 months ago

Ah indeed, thanks for pointing it out!