After about a full workday trying to make things work, I've run against the following issues.
The docs state that "BrowserStack is a good option for testing on older browser versions" but neglect to include some very useful information for testing against older browser versions. Why no mention of dev-server-legacy there?
With the guide as-is, I kept getting the following error when testing against Chrome 57, Safari 12, and Firefox 90:
Browser tests did not start after xxx ms You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information.
I only discovered @web/dev-server-legacy via a deep code investigation into this repository and the test-runner-browserstack automated-testing configuration. I figured your tests had to be passing, after all, so I decided to see what the difference was. Adding the @web/dev-server-legacy plugin to my local configuration was enough to get the Chrome 57 test target operational.
For reference, this was the block of code that triggered the realization:
I'm curious why there's no mention of dev-server-legacy on the documentation page linked above; having info or a link about it on that page would have spared me a fair bit of headache.
There seem to be some "more recent" targets for testing that are not supported while some "older" targets are. (Notably, IE seems to have specific support.)
Sadly, adding the previously-noted plugin still did nothing for Safari 12-14 or Firefox 90. I still haven't figured out what's missing for those two - and there's no working use of them in your test config for me to glean new info from... but I can state that Safari 15 and Firefox 100 targets work fine. We'd prefer to test against Safari 12, of course.
In all cases where I received that error, I've made the following observations that may be useful:
The automated tests actually did run. By setting the debug flag, I saw messages from the dev-server indicating that files only requested during specific automated tests were requested by BrowserStack's remote devices. If the tests didn't actually run, these file requests wouldn't show up in the debug logs.
I have identified a key part of the code where behaviors diverge significantly when the error is reported and is not.
When the "Browser tests did not start" error is emitted, I can confirm that the serverStart method is called and that line 108 is reached. However, line 109 is not reached at any point - let alone any point after the wtr- guard. Setting a breakpoint at line 109 will never capture for these cases. When I get positive test results (when targeting more recent browsers), line 109 receives its expected breakpoint hits.
Assuming my inferences about the design are correct, this observation implies that the components that detect whether or not tests actually were able to run... simply never get activated. _onSessionStarted is what denotes started tests, and it's only called later on within the function. That call is what updates the test-progress state variable... thus making it appear that the tests never run. We don't get the "tests completed" message either, as it uses the same pattern.
Thus far, I haven't figured out how to trace what's happening on the browser side in order to determine why the WebSocket isn't sending messages through.
To be clear, I'm writing up this issue based on my experiences trying to utilize this doc page, also viewable at https://modern-web.dev/docs/test-runner/browser-launchers/browserstack/.
After about a full workday trying to make things work, I've run against the following issues.
dev-server-legacythere?With the guide as-is, I kept getting the following error when testing against Chrome 57, Safari 12, and Firefox 90:
I only discovered
@web/dev-server-legacyvia a deep code investigation into this repository and thetest-runner-browserstackautomated-testing configuration. I figured your tests had to be passing, after all, so I decided to see what the difference was. Adding the@web/dev-server-legacyplugin to my local configuration was enough to get the Chrome 57 test target operational.For reference, this was the block of code that triggered the realization:
https://github.com/modernweb-dev/web/blob/c4a1044bbfdf5960b655efdfb2083ed9e055ec9b/integration/test-runner/tests/basic/runBasicTest.ts#L14-L20
I'm curious why there's no mention of
dev-server-legacyon the documentation page linked above; having info or a link about it on that page would have spared me a fair bit of headache.Sadly, adding the previously-noted plugin still did nothing for Safari 12-14 or Firefox 90. I still haven't figured out what's missing for those two - and there's no working use of them in your test config for me to glean new info from... but I can state that Safari 15 and Firefox 100 targets work fine. We'd prefer to test against Safari 12, of course.
In all cases where I received that error, I've made the following observations that may be useful:
The automated tests actually did run. By setting the
debugflag, I saw messages from the dev-server indicating that files only requested during specific automated tests were requested by BrowserStack's remote devices. If the tests didn't actually run, these file requests wouldn't show up in the debug logs.I have identified a key part of the code where behaviors diverge significantly when the error is reported and is not.
https://github.com/modernweb-dev/web/blob/c4a1044bbfdf5960b655efdfb2083ed9e055ec9b/packages/test-runner-core/src/server/plugins/api/testRunnerApiPlugin.ts#L107-L109
When the "Browser tests did not start" error is emitted, I can confirm that the
serverStartmethod is called and that line 108 is reached. However, line 109 is not reached at any point - let alone any point after thewtr-guard. Setting a breakpoint at line 109 will never capture for these cases. When I get positive test results (when targeting more recent browsers), line 109 receives its expected breakpoint hits.Assuming my inferences about the design are correct, this observation implies that the components that detect whether or not tests actually were able to run... simply never get activated.
_onSessionStartedis what denotes started tests, and it's only called later on within the function. That call is what updates the test-progress state variable... thus making it appear that the tests never run. We don't get the "tests completed" message either, as it uses the same pattern.Thus far, I haven't figured out how to trace what's happening on the browser side in order to determine why the WebSocket isn't sending messages through.