[BUG] Tracing.StopAsync fails when using BrowserStack Automate as a remote server

[martincostello]

Context:

• Playwright Version: v1.18.0
• Operating System: Windows
• .NET version: .NET 6
• Browser: All
• Extra:

Code Snippet

Clone the Remote-Traces branch of the https://github.com/martincostello/dotnet-playwright-tests repository, and configure valid credentials for BrowserStack Automate (see here).

Describe the bug

With the release of v1.18.0 I've been doing some investigation into using Playwright with BrowserStack, and incorporated it into some sample code here.

Today I extended the sample further to capture traces, but found that when BrowserStack Automate is being used, calls to Tracing.StopAsync() fail with the below exception (https://github.com/martincostello/dotnet-playwright-tests/pull/40#issuecomment-1022215119):

Message: 

Microsoft.Playwright.PlaywrightException : save: expected boolean, got undefined

  Stack Trace:
 
Connection.SendMessageToServerAsync[T](String guid, String method, Object args)
BrowserContextChannel.StopChunkAsync(String mode)
Tracing.DoStopChunkAsync(String filePath)
Tracing.StopChunkAsync(TracingStopChunkOptions options)
Tracing.StopAsync(TracingStopOptions options)
BrowserFixture.WithPageAsync(Func`2 action, String testName) line 90
BrowserFixture.WithPageAsync(Func`2 action, String testName) line 97
BrowserFixture.WithPageAsync(Func`2 action, String testName) line 98
SearchTests.Search_For_DotNet_Core(String browserType, String browserChannel) line 38
--- End of stack trace from previous location ---

Build logs showing the error are available from https://github.com/martincostello/dotnet-playwright-tests/pull/41.

Cloning this repository locally and using it from source and doing debugging lead me to a mismatch between what the BrowserStack remote server expects vs. what Playwright sends to it for the tracingStopChunk command. With tracing enabled, the error from the BrowserStack server is the below:

pw:channel:event: {"guid":"jsonPipe@b811a7f63cd5e9b8db3335ba3440ddc3","method":"message","params":{"message":{"id":14,"error":{"error":{"message":"save: expected boolean, got undefined","stack":"Error: save: expected boolean, got undefined\n    at tBoolean (C:\\Windows\\playwright\\1.17.2\\node_modules\\playwright\\node_modules\\playwright-core\\lib\\protocol\\validatorPrimitives.js:40:9)\n    at Object.BrowserContextTracingStopChunkParams (C:\\Windows\\playwright\\1.17.2\\node_modules\\playwright\\node_modules\\playwright-core\\lib\\protocol\\validatorPrimitives.js:99:21)\n    at DispatcherConnection._validateParams (C:\\Windows\\playwright\\1.17.2\\node_modules\\playwright\\node_modules\\playwright-core\\lib\\dispatchers\\dispatcher.js:224:26)\n    at DispatcherConnection.dispatch (C:\\Windows\\playwright\\1.17.2\\node_modules\\playwright\\node_modules\\playwright-core\\lib\\dispatchers\\dispatcher.js:267:26)\n    at WebSocket.\u003Canonymous\u003E (C:\\Windows\\playwright\\1.17.2\\node_modules\\playwright\\node_modules\\playwright-core\\lib\\remote\\playwrightServer.js:121:20)\n    at WebSocket.emit (events.js:196:13)\n    at Receiver.receiverOnMessage (C:\\Windows\\playwright\\1.17.2\\node_modules\\ws\\lib\\websocket.js:1008:20)\n    at Receiver.emit (events.js:196:13)\n    at Receiver.dataMessage (C:\\Windows\\playwright\\1.17.2\\node_modules\\ws\\lib\\receiver.js:517:14)\n    at Receiver.getData (C:\\Windows\\playwright\\1.17.2\\node_modules\\ws\\lib\\receiver.js:435:17)","name":"Error"}}}}}
pw:channel:response: {"id":14,"error":{"error":{"message":"save: expected boolean, got undefined","name":"Error"}}}

I'm unsure of whether this is a bug, an incompatibility between the v1.17.2 and v1.18.0 protocol versions, or some customisation made on BrowserStack's end that breaks it, but I thought I'd ask here first in case you have some insight into where/when this different parameters are/were used.

If I add a save and skipCompress boolean arguments to the command as shown in this commit, then it works as expected.

[mxschmitt]

The playwright client version needs to be the same as the playwright server version. In this case the Browserstack version is too old, thats why tracing.StopAsync is not working anymore. If I remember correctly you can pass a version parameter on their side so they give you a corresponding server which matches your client version.

(Since the protocol is an implementation detail we don't consider it as part of the SemVer versioning.)

[martincostello]

Yeah we pass that through, but they don't seem to have 1.18.0.

OK thanks, I'll contact BrowserStack support and ask them when 1.18.0 will be available.