search

microsoft / vscode-test-cli

Command-line runner for VS Code tests
MIT License
19 stars 7 forks source link

vscode-test bails at first configuration with test failures even without --bail #44

Closed eliericha closed 3 months ago

eliericha commented 3 months ago

Hello,

I'm using vscode-test-cli version 0.0.9. My .vscode-test.mjs file returns an array of test configurations. I'm observing that vscode-test stops after the first configuration that exhibits test failures, even though I didn't use the --bail CLI argument.

After some investigation I'm finding that it's a combination of behaviors in test-cli and test-electron.

In test-electron, I see here that a promise is rejected when the underlying process has an exit code != 0.

In test-cli, I see here that we consult the bail after awaiting the above promise. Since the promise is rejected, we don't get a chance to continue onto the next test configurations.

Is my analysis correct? And if so, is there a workaround I could use to unblock my setup pending a fix upstream?

Also more generally what's the status of vscode-test ? It seems to be the recommended test infrastructure in the official extension development documentation but the README in this repo says that it's experimental. Should I continue to try to use it in production or wait until it's more stable?

Thanks!

connor4312 commented 3 months ago

vscode-test is pretty stable now, we're using it for all new extensions we write on the team and most existing one.

Your analysis is correct. We should probably throw some better-typed error in test-electron and handle that appropriately in the CLI.

eliericha commented 3 months ago

Ok thanks for confirming. On my end I thought of a workaround: I can iterate the configurations and launch vscode-test --label <> for each one to be sure that they all run.