[BUG] Canvas is not visible in the trace

[aramvr]

System info

• Playwright Version: [v1.35.1]
• Operating System: [macOS 13.4]
• Browser: [Chromium]
• Other info:

The content of HTML5 canvas cannot be seen in the trace, which makes debugging the tests more difficult. I am unsure if this is expected behavior or not. I would appreciate any information that can be shared on the topic.

trace.zip

[pavelfeldman]

Playwright continuously produces DOM snapshots and those snapshots don't contain the canvas pixels. The only way to capture canvas content as an image is to perform a GPU readback (canvas is composited on GPU), and that is expensive.

Having said that, if there is a canvas on the screen, it is likely that the information on the canvas is critical to the trace and we can allow users opt into canvas rendering.

[aramvr]

Playwright continuously produces DOM snapshots and those snapshots don't contain the canvas pixels. The only way to capture canvas content as an image is to perform a GPU readback (canvas is composited on GPU), and that is expensive.

Having said that, if there is a canvas on the screen, it is likely that the information on the canvas is critical to the trace and we can allow users opt into canvas rendering.

To make sure I understand this right, are you saying it's possible to capture canvas on the trace but it's expensive action or it's not possible at all?

[pavelfeldman]

It is possible and we will consider implementing it!

[fergusmeiklejohn]

@pavelfeldman please do consider it! We're trying to test an app that draws on the canvas and you can imagine how near impossible it is to debug failing tests if we can't see the canvas in the UI and Trace 🙏🏻

[jrolfs]

Yes, we were hoping to be able to use a new Chromatic/Storybook feature to capture the state of an application that draws to a canvas and were disappointed to realize that the traces don't include the contents of the canvas (the archiver seems to rely on traces when capturing visual diffs).

This would be incredibly helpful for many of our applications. Thank you for considering it and thank you for Playwright :)

[DrTeit]

I would also very much like this feature :) Spent a good few days trying to figure this out, with waits and fired events and whatnot (since my canvas fades in and it shows up in the timeline, but not in the screenshot).

[hamzawain7]

+1. This would be a beneficial feature to debug my canvas-based software, too.

[DrTeit]

@pavelfeldman are you able to say if this feature will receive priority / be implemented at all in the near future? If this gets made, do you know if webgl on canvas will be supported out of the box? Thanks for the work you do on Playwright, it's by far the best GUI test tool I've used :)

[dfcan]

This feature would be very very helpful for us as well. Any idea on if it will be considered?
In the meantime, are there any manual workarounds we can do?

[dfcan]

I would also very much like this feature :) Spent a good few days trying to figure this out, with waits and fired events and whatnot (since my canvas fades in and it shows up in the timeline, but not in the screenshot).

@DrTeit were you able to successfully create a work around with waits / fired events? Do you have any snippets of what that looks like?

[DrTeit]

@dfcan sorry no workaround for the preview, howver clicking the canvas «blindly» using coordinates works. You won’t be doing any asserts, but it can be helpfull to click anyways and manually inspect a trace later on, but no automatic visual regression test I’m afraid

[AJMcKane]

We have a heavily map based application that we have been testing with Cypress and it was able to send Canvas content in the trace, is there any update on support for <canvas> or potential workarounds?

[Skn0tt]

Hi folks! We've had a go at storing the canvas contents inside the trace in https://github.com/microsoft/playwright/pull/32009, but it fell short at anything using WebGL. canvas is rendered via the GPU, and that makes it pretty hard to get the trace contents reliably. See the PR for more details on why that approach failed.

Here's a different idea for a solution though: As @DrTeit points out in https://github.com/microsoft/playwright/issues/23964#issuecomment-1744837690, the timeline shows the canvas content perfectly. That's because the timeline is a raw screencapture of the browser, and not a structured DOM representation like traces.

Now, we're thinking of adding a toggle to see the screencapture where you're normally seeing the trace. It wouldn't allow you to inspect the DOM or get a locator, but it'd show the canvas contents. Would that help you in your workflows?

[dfcan]

Hi folks! We've had a go at storing the canvas contents inside the trace in #32009, but it fell short at anything using WebGL. canvas is rendered via the GPU, and that makes it pretty hard to get the trace contents reliably. See the PR for more details on why that approach failed.

Here's a different idea for a solution though: As @DrTeit points out in #23964 (comment), the timeline shows the canvas content perfectly. That's because the timeline is a raw screencapture of the browser, and not a structured DOM representation like traces.

Now, we're thinking of adding a toggle to see the screencapture where you're normally seeing the trace. It wouldn't allow you to inspect the DOM or get a locator, but it'd show the canvas contents. Would that help you in your workflows?

Yes this would be great for us!

[WestonThayer]

+1, a screencapture toggle would make this much more usable in our case, since a <canvas> takes up most of the viewport. We've actually been using hovering on the timeline to see the screencapture as a workaround, but it's a tiny thumbnail and not easily discoverable.

[ruifigueira]

@Skn0tt, just a crazy idea: what about rendering the image as a background but leave elements with opacity: 0? It would allow us to keep inspecting the DOM.

I did a quick test and it's not perfect but it seems to work (run with trace viewer to then inspect the last snapshot):

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.setContent(`
    <html>
    <head>
      <style>
        label, input, button { margin: 8px; }
        #login { border: 2px dashed dodgerblue; background: lightskyblue; width: 320px; }
      </style>
    </html>
    <body>
      <form id="login">
        <label for="user">Username:</label><input type="text" id="name">
        <br>
        <label for="pass">Password:</label><input type="password" id="pass">
        <br>
        <button type="submit">Login</button>
      <form>
    </body>
    <html>`);
  const bodyScreenshot = await page.screenshot({ scale: 'device' });
  await page.evaluate(imageSrc => {
    const { width, height } = document.documentElement.getBoundingClientRect();
    const template = document.createElement('template');
    template.innerHTML = `<style>body * { opacity: 0 }</script>`;
    document.head.append(template.content.firstChild!);
    document.documentElement.style.backgroundImage = `url(${imageSrc})`;
    document.documentElement.style.backgroundSize = `${width}px ${height}px`;
  }, `data:image/png;base64,${bodyScreenshot.toString('base64')}`);
});

[Skn0tt]

That's a nice idea! I think we'd also need to block scrolling for that to work. I'll play around with the idea later today.

[Skn0tt]

We tried it out in https://github.com/microsoft/playwright/pull/32515, but decided not to go that route for now - read the reasoning in the PR.

[ruifigueira]

Yes, in the tests I did it was not pixel perfect, and it was just a very simple html, so all those discrepancies will add up.