[BUG] page.mouse() does not seem to draw on canvas

[sirolf2009]

We have a page with a Form IO signature field, which has a canvas where you can draw a digital signature. We are trying to write a test that our form only submits when a signature is drawn, but we are unable to draw the signature

System info

• Playwright Version: [v1.37.0]
• Operating System: [macOS 13.5.2]
• Browser: [Chromium, Edge]
• Other info:

Source code

ElementHandle canvas = page.querySelector("//div[@ref='padBody']//canvas[@ref='canvas']");
var boundingBox = canvas.boundingBox();

System.out.println(boundingBox.x+" "+ boundingBox.y+" "+ boundingBox.width+" "+ boundingBox.height);
//outputs 430.0 904.46875 860.0 914.8671875

page.mouse().move(boundingBox.x+50, boundingBox.y+50);
page.mouse().down();
page.mouse().move(boundingBox.x+100, boundingBox.y+100);
page.mouse().move(boundingBox.x+200, boundingBox.y+100);
page.mouse().up();

Expected

Two lines to be drawn on our canvas

Actual

The test moves along the code, but does not draw anything on the canvas. Inspecting the trace in https://trace.playwright.dev/ shows that the steps were executed, but does not show any other information on what is being executed/where the mouse is drawing

[yury-s]

This is likely the same as https://github.com/microsoft/playwright/issues/23964, canvas is not captured in the dom snapshots.

[aslushnikov]

@sirolf2009 if you watch the test execution in headed mode live, does it actually draw lines on the canvas?

If not, any chance you can provide us with a repro?

[sirolf2009]

No, no lines are drawn in headed mode. You can try this script, it builds a form using an online Form IO builder with the signature field.

package org;

import com.microsoft.playwright.*;

import java.util.List;

public class App {
    public static void main(String[] args) {
        try (Playwright playwright = Playwright.create()) {
            Browser browser = playwright.chromium().launch(new BrowserType.LaunchOptions().setHeadless(false));
            var browserContext = browser.newContext(new Browser.NewContextOptions().setViewportSize(1728, 915));
            Page page = browserContext.newPage();

            page.navigate("https://formio.github.io/formio.js/app/builder");

            page.click("//*[@id=\"heading-advanced\"]/h5/button");
            page.dragAndDrop(
                    "//*[@id=\"group-container-advanced\"]/span[11]",
                    "//*[@id=\"builder\"]/div/div[2]/div/div");
            page.click("//*[contains(text(), 'Save')]");

            List<ElementHandle> canvases = page.querySelectorAll("//div[@ref='padBody']//canvas[@ref='canvas']");
            ElementHandle canvas = canvases.get(canvases.size()-1);
            var boundingBox = canvas.boundingBox();

            System.out.println(boundingBox.x+" "+ boundingBox.y+" "+ boundingBox.width+" "+ boundingBox.height);

            page.mouse().move(boundingBox.x+50, boundingBox.y+50);
            page.mouse().down();
            page.mouse().move(boundingBox.x+100, boundingBox.y+100);
            page.mouse().move(boundingBox.x+200, boundingBox.y+100);
            page.mouse().up();

            System.out.println("complete");
        }
    }
}

[aslushnikov]

@sirolf2009 turns out that's how the website works. Instead of dispatching 2 events, you have to dispatch all the intermediate events as well.

The following worked well for me in javascript:

import { chromium, firefox } from '@playwright/test';

const browser = await chromium.launch({ headless: false });
const browserContext = await browser.newContext({ viewport: { width: 1728, height: 915 } });
const page = await browserContext.newPage();

await page.goto('https://formio.github.io/formio.js/app/builder');

await page.click('//*[@id="heading-advanced"]/h5/button');
await page.dragAndDrop(
  '//*[@id="group-container-advanced"]/span[11]',
  '//*[@id="builder"]/div/div[2]/div/div'
);
await page.click('//*[contains(text(), "Save")]');

const canvases = await page.$$('//div[@ref="padBody"]//canvas[@ref="canvas"]');
await page.$$eval('//div[@ref="padBody"]//canvas[@ref="canvas"]', canvases => console.log(canvases));
const canvas = canvases[canvases.length - 1];
await canvas.scrollIntoViewIfNeeded();
const boundingBox = await canvas.boundingBox();

console.log(`${boundingBox.x} ${boundingBox.y} ${boundingBox.width} ${boundingBox.height}`);

await page.mouse.move(boundingBox.x + 50, boundingBox.y + 50);
await page.mouse.down();
for (let i = 0; i < 100; ++i) {
  await page.mouse.move(boundingBox.x + i, boundingBox.y + i);
}
await page.mouse.up();

console.log('complete');

// await browser.close();

Pay attention to the for-loop that actually dispatches events one-by-one:

for (let i = 0; i < 100; ++i)
  await page.mouse.move(boundingBox.x + i, boundingBox.y + i);

So looks like not a playwright issue; closing as works as intended!

[sirolf2009]

Oh damn that's really smart, I hadn't thought of that. Thanks man, you're great!

[nico75005]

Adding steps works as well:

    await page.mouse.move(lineStartingPoint.x, lineStartingPoint.y);
    await page.mouse.down();
    await page.mouse.move(lineEndingPoint.x, lineEndingPoint.y, { steps: 1.1 });
    await page.mouse.up();

Having an explicit steps of 1 or not explicit steps will lead to that issue. Anything equal or above to 1.1 will work.