search

w3c / webdriver-bidi

Bidirectional WebDriver protocol for browser automation
https://w3c.github.io/webdriver-bidi/
363 stars 40 forks source link

Support retrieval of content quads for elements with CSS transitions #787

Open whimboo opened 1 week ago

whimboo commented 1 week ago

It would be beneficial to support the retrieval of content quads for elements with CSS transitions applied. Currently, this functionality lacks a uniform W3C specification, leading to inconsistent implementations across browsers, including differences in API naming. Both Chrome and Firefox, however, support content quad retrieval.

The content quad of an element should accurately reflect its current position and size, even during CSS transitions, and should be used instead of getBoundingClientRect() to get the actual position and size for performing precise actions on such elements.

Integrating this API directly into WebDriver BiDi would enable clients to use a dedicated command that works consistently across all browsers, eliminating the need for browser-specific implementations via script.evaluate or script.callFunction.

For Chrome, the DOM.getContentQuads() command is available in the Chrome DevTools Protocol, while Firefox implements this as Element.getBoxQuads().

css-meeting-bot commented 6 days ago

The Browser Testing and Tools Working Group just discussed Support retrieval of content quads for elements with CSS transitions.

The full IRC log of that discussion <simonstewart> Topic: Support retrieval of content quads for elements with CSS transitions
<simonstewart> github: https://github.com/w3c/webdriver-bidi/issues/787
<simonstewart> jgraham: some people want to retrieve content quads for elements. I don't know if that's different from calling getBoundingClientRect for an element, though it might have a nicer return value
<simonstewart> jgraham: isn't getBoundingClientRect affected by CSS Transform already?
<simonstewart> <quiet reading in the room>
<simonstewart> gsnedders: this does something with the document scale factor.
<simonstewart> gsnedders: It's not clear what the difference is.
<simonstewart> <people start reading browser source code to figure out what's happening>
<gsnedders> getClientRects, not getBoundingClientRect, because the Playwright API is returning all the rects.
<simonstewart> jgraham: No-one is clear what the difference is between getting content quads and getClientRects, so some clarification would be helpful.
OrKoN commented 6 days ago

I believe the difference is that quad returns 4 points whereas the rect returns only 2. So rect is always rectangular, while the quad can reflect the actual shape of the element after the transform. I think it can be still important for hit-testing.

whimboo commented 6 days ago

Yes, this is actually for https://developer.mozilla.org/en-US/docs/Web/API/DOMQuad and is important (as I already mentioned) when elements are transformed via CSS.

Here an example: https://drafts.fxtf.org/geometry/#example-9bbe24bd

It looks like that we maybe should discuss it again today when we are all present?

css-meeting-bot commented 5 days ago

The Browser Testing and Tools Working Group just discussed Support retrieval of content quads for elements with CSS transitions, and agreed to the following:

The full IRC log of that discussion <AutomatedTester> topic: Support retrieval of content quads for elements with CSS transitions
<AutomatedTester> github: https://github.com/w3c/webdriver-bidi/issues/787
<simonstewart> https://test.csswg.org/harness/test/geometry-1_dev/
<whimboo> https://github.com/web-platform-tests/wpt/tree/fd25aeb946a8df94314cf6fb4fe63b46eee2cd8c/css/geometry
<gsnedders> https://wpt.fyi/results/css/geometry?run_id=5201124441456640&run_id=5179680340836352&run_id=5184785144348672 would be a better link for tests, rather than the old CSS WG Test Harness (which is the same tests)
<gsnedders> s/link for tests/link for test results/
<gsnedders> https://drafts.csswg.org/cssom-view/#dom-geometryutils-getboxquads seems like it should give a DOMQuad with what we want… if it were specified. Or implemented.
<gsnedders> c.f. https://wpt.fyi/results/css/cssom-view/idlharness.html?run_id=5201124441456640&run_id=5179680340836352&run_id=5184785144348672 for evidence of lack of implementation of getBoxQuads
<jgraham> jgraham: So the understanding I have is that getBoundingCLientRect returns a DOMRect, which is the bounds of a rectangle containing the element. For transformed elements that's different to the actual bounds of the element if those are non-rectangular. DOMQuads in https://drafts.fxtf.org/geometry/#DOMQuad represents the latter. This is what CDP
<jgraham> exposes via its API. There is a draft API for getting these values from content in CSSOM-view https://drafts.csswg.org/cssom-view/#dom-geometryutils-getboxquads however this is basically all TODO.
<jgraham> This is actually implemented in Firefox.
<jgraham> So the problem we have here is that in terms of a standard we don't have anything to point at to explain the behaviour of the requested API. We could implement it in Gecko and Chromium, but it would be nonstandard or implementation-defined. If we actually wrote a standard here we might as well finish CSSOM-view and then expose it to content
<jgraham> (assuming there's no compat issue), and then clients could use that via script. Having a BiDi only API here seems rather odd.
<davehunt> jgraham: we need someone from layout or from the CSS WG to help us out here. We should talk to Emilio
<AutomatedTester> ACTION: jgraham to speak to Emilio and get advice on what we can do here
gsnedders commented 5 days ago

while Firefox implements this as Element.getBoxQuads()

Note this is effectively currently unspecified: https://drafts.csswg.org/cssom-view/#dom-geometryutils-getboxquads (though supported in Firefox behind a flag).

See https://github.com/w3c/csswg-drafts/issues/10968.

jgraham commented 5 days ago

https://github.com/w3c/csswg-drafts/issues/10537 is an issue for defining getBoxQuads, which is a content API implementing this feature. I think this issue is blocked on having some usable definition of what getBoxQuads does, and if that definition is also tied to a DOM API I think it would make sense for clients to use that content API via script rather than having a special API in WebDriver that does the same thing.

gsnedders commented 5 days ago

See also https://github.com/w3c/webdriver-bidi/issues/769 / https://github.com/microsoft/playwright/issues/32577:

Add support for getting content quads to ensure accurate clicking inside transformed elements. This is particularly relevant for tests involving transformed elements, as seen in tests/page/page-click.spec.ts.