Automated end-to-end testing

simondotm / beeb-vsc

Visual Studio Code Extension to support code development for BBC Micro

https://marketplace.visualstudio.com/items?itemName=simondotm.beeb-vsc

MIT License

12 stars 3 forks source link

Automated end-to-end testing #108

Open tommy9 opened 3 months ago

tommy9 commented 3 months ago

The language server sample https://github.com/microsoft/vscode-extension-samples/tree/main/lsp-sample shows how to do end-to-end testing, with examples for completions and diagnostics. It would be useful to have a full test suite of lsp functionality that can be run like this before new releases.

I've started work on this but will first have to fix #107 so that the text fixture can have a test BeebASM project that works on more than just my machine.

Possible items to test this way (assuming vscode exposes enough info for us to detect whether they work or not):

[ ] Completions (added cases for BeebASM function/command examples but should expand to symbols/labels too
[ ] Diagnostics (added 3 samples but can pick many more from the BeebASM test suite)
[ ] Document links
[ ] Hovers
[x] Rename (label and symbol) (added test for renaming one of each type)
[ ] References
[ ] Signature help

tommy9 commented 3 months ago

Have merged the initial version, although will need updating after fixing #107 but this was easier to get in as a base then start a new branch.

simondotm commented 3 months ago

If e2e testing can be run just within the node environment, ie. uses just whatever is in packages.json then I'd be happy to add it as a CI step. 👍

tommy9 commented 3 months ago

That sounds really great.

It does appear to be able to run from node. I could get it working with two steps: 1) npm run compile-for-tests 2) node ./out/client/test/runTest.js

I had to change the code in runTest.ts to pass in the testing workspace folder path so that beeb-vsc can access settings.json. I'll push that little change.

I don't know if the CI environment will need any adjustment to the timing. We've got an await sleep(2000) command in helper.ts to give it enough time to finish vscode activation tasks.

simondotm commented 3 months ago

it's usually pretty reliable, if it works locally, it should work on the CI containers too.

simondotm commented 3 months ago

ohhh I just ran it locally, and see it uses a UI, so I wonder if its using selenium or something. this may not work in github actions as they are console only.

simondotm commented 3 months ago

yeah unfortunately it's not gonna fly 😞 : [2030:0419/150654.626199:ERROR:ozone_platform_x11.cc(240)] Missing X server or $DISPLAY

https://github.com/simondotm/beeb-vsc/pull/118

simondotm commented 3 months ago

cool tech though! 👍

tommy9 commented 3 months ago

yeah unfortunately it's not gonna fly 😞 : [2030:0419/150654.626199:ERROR:ozone_platform_x11.cc(240)] Missing X server or $DISPLAY

118

Ah, that's a shame. I do have some simpler unit tests for the parsing functionality src/server/test/tests.ts. I've only ever run those with the mocha test explorer extension however. It must be possible to run from the console however and these don't require any host. I'll dig into this as it would also be nice to have as part of the CI.