Automated end-to-end testing

[tommy9]

The language server sample https://github.com/microsoft/vscode-extension-samples/tree/main/lsp-sample shows how to do end-to-end testing, with examples for completions and diagnostics. It would be useful to have a full test suite of lsp functionality that can be run like this before new releases.

I've started work on this but will first have to fix #107 so that the text fixture can have a test BeebASM project that works on more than just my machine.

Possible items to test this way (assuming vscode exposes enough info for us to detect whether they work or not):

• [ ] Completions (added cases for BeebASM function/command examples but should expand to symbols/labels too
• [ ] Diagnostics (added 3 samples but can pick many more from the BeebASM test suite)
• [ ] Document links
• [ ] Hovers
• [x] Rename (label and symbol) (added test for renaming one of each type)
• [ ] References
• [ ] Signature help

[tommy9]

Have merged the initial version, although will need updating after fixing #107 but this was easier to get in as a base then start a new branch.

[simondotm]

If e2e testing can be run just within the node environment, ie. uses just whatever is in packages.json then I'd be happy to add it as a CI step. 👍

[tommy9]

That sounds really great.

It does appear to be able to run from node. I could get it working with two steps: 1) npm run compile-for-tests 2) node ./out/client/test/runTest.js

I had to change the code in runTest.ts to pass in the testing workspace folder path so that beeb-vsc can access settings.json. I'll push that little change.

I don't know if the CI environment will need any adjustment to the timing. We've got an await sleep(2000) command in helper.ts to give it enough time to finish vscode activation tasks.

[simondotm]

it's usually pretty reliable, if it works locally, it should work on the CI containers too.

[simondotm]

ohhh I just ran it locally, and see it uses a UI, so I wonder if its using selenium or something. this may not work in github actions as they are console only.

[simondotm]

yeah unfortunately it's not gonna fly 😞 : [2030:0419/150654.626199:ERROR:ozone_platform_x11.cc(240)] Missing X server or $DISPLAY

https://github.com/simondotm/beeb-vsc/pull/118

[simondotm]

cool tech though! 👍

[tommy9]

yeah unfortunately it's not gonna fly 😞 : [2030:0419/150654.626199:ERROR:ozone_platform_x11.cc(240)] Missing X server or $DISPLAY

118

Ah, that's a shame. I do have some simpler unit tests for the parsing functionality src/server/test/tests.ts. I've only ever run those with the mocha test explorer extension however. It must be possible to run from the console however and these don't require any host. I'll dig into this as it would also be nice to have as part of the CI.