Need a solution for allowing custom elements to be a CanvasImageSource

[andyearnshaw]

Custom elements with shadow trees that contain a video, img or canvas element cannot be passed to CanvasRenderingContext2D.prototype.drawImage. Instead, the element needs to be exposed, breaking encapsulation.

<x-video name="vid1" autoplay>
  #shadow-root
    <video ...></video> 
    <!-- other stuff -->
</x-video>

var vid = document.querySelector('x-video');
canvasContext.drawImage(vid, 0, 0, vid.width, vid.height);

Initially, I thought this could be an extension to ShadowRoot, but @annevk suggested at https://github.com/w3c/webcomponents/issues/388 to file the issue here so that it can be tackled from the drawImage side.

[tabatkins]

This sounds similar to the work needed to make custom elements interoperate with form validation/submission. In other words, we need to to expose some API demagicking the "extract an imagesource from an element" algo in the spec.

Do we want to start relying on JS Symbols for this kind of thing? Check if the first arg exposes a particular well-known symbol, attempt to call its value, and verify that it returns something drawable?

[andyearnshaw]

I think a callable well-known symbol would be a simple, but effective approach. It would provide consistency with how JS demagicked and exposed some internal behaviour of APIs, as well as allowing additional rendering or the entire drawing to be done on demand for some things.

We'd probably need to have the WebGL spec updated for texImage2D also.

[annevk]

Paging @junov. I like the symbol approach. I wonder if we could go as far as explain <img> and <canvas>, and <video> through that too and let their behavior be overridden.

[domenic]

A couple of issues with the symbol approach:

• Last time we tried this, for custom elements, there was pushback that it didn't solve any real problems and was too foreign to developers. My arguments that meta-level things should be stratified into symbols were met with "yeah, that kind of makes sense, but it's still not worth it." What's changed?
• How does this interface with normal usage? That is, if we say that ctx.drawImage(x, ...) performs x[CanvasRenderingContext2D.getDrawableImageBitmap](), does that mean I can change the behavior when passed a HTMLVideoElement by overwriting HTMLVideoElement.prototype[CanvasRenderingContext2D.getDrawableImageBitmap]?

[annevk]

I'm not sure, it seems a little different since custom element hooks are only present until you invoke defineElement(). And as you indicate in your second point, this hook could be used to override the default behavior of elements. We didn't figure out a way to do that for the custom element hooks.

[tabatkins]

How does this interface with normal usage? That is, if we say that ctx.drawImage(x, ...) performs x[CanvasRenderingContext2D.getDrawableImageBitmap](), does that mean I can change the behavior when passed a HTMLVideoElement by overwriting HTMLVideoElement.prototype[CanvasRenderingContext2D.getDrawableImageBitmap]?

That's what Anne was referring to, yes. I'm fine with allowing it, or disallowing it (making the properties readonly or ignored on builtins). Either works for me.

[junov]

This is an awesome idea. It's the kind of feature that makes the platform more hackable for framework devs to do all sorts of magic.

I am missing a bit of context here... From the name getDrawableImageBitmap I infer this is supposed to return an ImageBitmap? Not sure how convenient ImageBitmaps are for this because createImageBitmap is async, which makes it challenging to produce ImageBitmaps in the context of synchronous function call, depending on what you are trying to do. Perhaps this should be a function that returns any type of CanvasImageSource?

[domenic]

Yeah, I agree that any CanvasImageSource would make more sense.

Here are a few possible approaches I see:

• Maximally hookable:
  • drawImage() is changed to take an object argument
  • drawImage() calls its argument's arg.getImageSource(), and then checks that it got back a CanvasImageSource and if so proceeds as it does today. (If not, it throws a TypeError.)
  • We define {HTMLImageElement,SVGImageElement,HTMLVideoElement,HTMLCanvasElement,CanvasRenderingContext2D,ImageBitmap}.prototype.getImageSource() to do return this.
  • So, if someone does something like var img = new Image(); img.getImageSource = () => document.querySelector("canvas").getContext("2d"), then calling ctx.drawImage(img) will draw the canvas context, instead of drawing the img.
• In between:
  • We add an overload of drawImage() that takes an object
  • The current overload proceeds as normal
  • The object overload calls its argument arg.getImageSource(), then hands it off to the logic of the normal drawImage().
  • Nothing gets added to the existing image source classes
  • So, if someone does something like the above example, then ctx.drawImage(img) will draw img, since the non-object overload wins.
• Custom element hooks based
  • We add an overload of drawImage() that takes a HTMLElement
  • We add getImageSource() to the list of custom element "lifecycle hooks" (which we might need to rename)
  • The HTMLElement overload looks up the getImageSource lifecycle hook for the element provided, and tries to use it
  • This means you cannot customize on an individual per-object or even per-element basis; instead, it's the same get-image-source behavior for all elements of a given custom element type, with that behavior set at definition time.

My thoughts:

• Maximally hookable: the attempt at a generic design seems to be made awkward by the privileged position of the six existing types. I don't like it that much.
• In between: I like this the best, probabably? It acknowledges that some types are special, but still provides an extension point.
• Custom elements hook: it's OK, although it's a bit jarring how different getImageSource is from the existing lifecycle callbacks.

NOTE: none of these really address the OP's desire for encapsulation, since in all cases you could just do customEl.getImageSource() and get back the inner canvas element. So that's a bit of a bummer. I guess you could do it with the custom elements hook version, if after defining your element you did delete MyCustomEl.prototype.getImageSource.

[annevk]

Another way to offer encapsulation would be to tie extensibility to ShadowRoot and put the extension hooks there. The existing HTML elements would have the hook already but it's not accessible since their ShadowRoot is not accessible.

[domenic]

Oooh, that's very interesting. So any element with a shadow root could be drawn. You'd do something like

el.attachShadow({
  getImageSource() {
    return this.querySelector("video");
  }
});

That seems to fit with the OP's idea and with the platform the most.

[andyearnshaw]

Yeah I like that idea, it keeps the encapsulation intact and looks really simple.

[junov]

I like the idea of tying extensibility to ShadowRoot as well. It seems like a very well contained API surface that targets specifically the problem we aim to solve.

Something that crossed my mind: whichever solution we retain should be implementable in a way that does not regress the performance of drawImage on existing CanvasImageSource types. Not an issue with the approach of extending through ShadowRoot.

[marcoscaceres]

This was discussed at TPAC 2018, but will be deferred for now.

[annevk]

The way to go about this should be via ElementInternals added in #4324, similar to the plan for form controls.

cc @tkent-google

[andyearnshaw]

I've been recently fixing an issue on a feature of a company project which led me back to this in a roundabout way. We're using html-to-image to take a snapshot of select elements on the page, the library itself uses SVG <foreignObject> elements to do this.

A simplified version of the steps is as follows:

1. Clone the node (including descendants) into the <foreignObject> tag
2. Copy all the computed styles (and pseudo-elements) over to each element
3. Download images and fonts, convert them to data uris and embed them
4. Serialise to SVG

At that point the image can be painted onto a canvas if desired. So this made me wonder why most elements can't simply be a CanvasImageSource, or have some way to be rendered onto a canvas. Naturally, there are some things (like iframes) that you probably don't want to be able to paint onto a canvas, which I understand, but a browser could render a snapshot of most elements and blank out any that have cross-origin sources. And it could do it much, much, much faster than the above approach. This would also (partially) solve the issue for Custom Elements, with maybe an ElementInternals option or CSS selector for filtering out certain elements during the snapshot.

I'm perhaps missing some important reason why this can't be done, but is it possible that this could be a better solution to the problem? I feel like it could open up new avenues for simpler image creation tools (think easier object manipulation than clearing and redrawing canvases).

[junov]

The main reason this is not a thing is security and privacy concerns. In theory the problem is solvable, but it is really complicated to do it right without adding vulnerabilities that could allow web sites (malicious or not) to access private data without the user's consent. SVG's foreignObject tag adds an insulation barrier because the content inside it is rendered without having access to the entire browser's context (cookies, browser history, etc.) and it cannot load cross-origin content. For example, you will notice that visited links do not get displayed in a different color when inside a foreignObject tag since it does not have access to the user's browsing history. This is a good thing, otherwise websites could sniff your browsing history. There are plenty other subtle private things that could be leaked, for example, whether the user has a vision impairment (using larger font size, or an extension that enhances color contrast). Also such a feature could be a significant source of entropy for fingerprinting.

Even if we did find a way to directly support custom elements as canvas image sources without compromising the user's privacy and security, It would probably not be much better in terms of functionality than going through a tag. I think the most realistic solution is to make a JS library that wraps the foreignObject trick in a way that makes it convenient to use, and share that with the world.

On Sun, Mar 7, 2021 at 7:05 PM Andy Earnshaw notifications@github.com wrote:

I've been recently fixing an issue https://github.com/bubkoo/html-to-image/issues/96 on a feature of a company project which led me back to this in a roundabout way. We're using html-to-image https://github.com/bubkoo/html-to-image to take a snapshot of select elements on the page, the library itself uses SVG

elements to do this. A simplified version of the steps is as follows: 1. Clone the node (including descendants) into the tag 2. Copy all the computed styles (and pseudo-elements) over to each element 3. Download images and fonts, convert them to data uris and embed them 4. Serialise to SVG At that point the image can be painted onto a canvas if desired. So this made me wonder why most elements can't simply be a CanvasImageSource, or have some way to be rendered onto a canvas. Naturally, there are some things (like iframes) that you probably don't want to be able to paint onto a canvas, which I understand, but a browser could render a snapshot of most elements and blank out any that have cross-origin sources. And it could do it much, much, much faster than the above approach. This would also (partially) solve the issue for Custom Elements, with maybe an ElementInternals option or CSS selector for filtering out certain elements during the snapshot. I'm perhaps missing some important reason why this can't be done, but is it possible that this could be a better solution to the problem? I feel like it could open up new avenues for simpler image creation tools (think easier object manipulation than clearing and redrawing canvases). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or unsubscribe .

[andyearnshaw]

You make some good points, but I'm not convinced by all of them. Some of the concerns you raised are moot because you're already running scripts on the page, so things like font size and colour contrast extensions are easily accessible to the page owner and third-party scripts anyway. I mentioned potentially blanking out elements with cross-origin content, which I think would be the major hurdle, but the point about :visited links is fair, although I'm sure that would be solvable too (like how the styling is already inaccessible). And if these problems are already solved for SVG's <foreignObject>, is it difficult to reuse those solutions in a hypothetical feature that allows painting HTML onto a canvas?

Even if we did find a way to directly support custom elements as canvas image sources without compromising the user's privacy and security, It would probably not be much better in terms of functionality than going through a tag. I think the most realistic solution is to make a JS library that wraps the foreignObject trick in a way that makes it convenient to use, and share that with the world.

This is what the library I mentioned already does, but I disagree that the browser couldn't do it any better. The library has to re-download and embed all images and fonts for elements, those are network requests that may result in an error or a different result for what is on the page already, not to mention the extra time it takes.

[junov]

The need to re-download and embed images is a good argument IMHO.

There would be no strict requirement to blank-out cross-origin content because canvas 2d contexts support tainting. Basically, when cross-origin content is drawn to a 2d context, the origin-clean flag gets set to false, which blocks the canvas contents from being read. The problem is with spec'ing and implementing the tainting mechanism for drawing arbitrary DOM-content to a canvas, which is non-trivial/

FWIW, A similar feature was attempted a few years-ago in WebGL (render HTML to a WebGL texture). The enforcement of cross-origin content security was very limiting. The feature prototype was limited to same-origin iframes. That made the feature not as useful as was hoped and it ended up being abandoned. At the time, the use case people were trying to solve was to embed YouTube videos into 3D scenes, but the videos would have been cross-origin, which meant the content was blocked, and we saw no way around that for this particular use case. Unlike with 2D contexts, WebGL completely disallows cross-origin content. The reason is that it was ascertained that it was not possible to protect cross-origin content from timing attacks in WebGL. Basically, you can write a WebGL shader that takes a long or short time to execute based on the color of a pixel, and that allows cross-origin content to be leaked via code that measures shader execution time. Anyways... 2d contexts do not have this vulnerability, I just wanted to give some background. For 2D contexts, this feature was not attempted yet because we never had significant enough demand for it.

At this point in time, the WebGL spec is not evolving much anymore, but the idea of injecting DOM content into a rendering context is not dead. It is now being pursued by the WebGPU spec authors. There is talk of adding some type of secure rendering surface that could host DOM content and be used safely by rendering contexts. Since the WebGPU folks are further along in their pursuit of such a feature, I think it makes sense to wait and see what happens in WebGPU-land. If there is sufficient demand to justify the engineering effort of spec'ing and implementing the feature for WebGPU, then we could just ride that wave and get the feature for almost free in 2D and WebGL contexts, assuming that whatever solution they come up with will be easy to use as a CanvasImageSource.

On Mon, Mar 8, 2021 at 11:14 AM Andy Earnshaw notifications@github.com wrote:

You make some good points, but I'm not convinced by all of them. Some of the concerns you raised are moot because you're already running scripts on the page, so things like font size and colour contrast extensions are easily accessible to the page owner and third-party scripts anyway. I mentioned potentially blanking out elements with cross-origin content, which I think would be the major hurdle, but the point about :visited links is fair, although I'm sure that would be solvable too (like how the styling is already inaccessible). And if these problems are already solved for SVG's , is it difficult to reuse those solutions in a hypothetical feature that allows painting HTML onto a canvas?

Even if we did find a way to directly support custom elements as canvas image sources without compromising the user's privacy and security, It would probably not be much better in terms of functionality than going through a tag. I think the most realistic solution is to make a JS library that wraps the foreignObject trick in a way that makes it convenient to use, and share that with the world.

This is what the library I mentioned already does, but I disagree that the browser couldn't do it any better. The library has to re-download and embed all images and fonts for elements, those are network requests that may result in an error or a different result for what is on the page already, not to mention the extra time it takes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/whatwg/html/issues/706#issuecomment-792867148, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAT4V6F24XDWSWVC3O5FWZDTCTZWZANCNFSM4B3U2JVQ .

[andyearnshaw]

That background is interesting, and so is the revival of the idea, do you have any links to those discussions? Thanks for sharing this!

The reason I mentioned blanking out cross-origin elements was we're creating images from user content and exporting them. That content might contain images that don't have CORS configured, but the user may not necessarily have control over that. Blanking out or replacing that content would allow those users to make use of the feature without having to remove all those images. You could work around that by iterating over all the elements beforehand and replacing them if they would taint the canvas, I suppose, but it's a little awkward to do this and it may cause some visual flickering (while content that is restored afterwards reloads). Some kind of callback would be useful in that case.

[junov]

Discussions about the WebGPU feature were on a Google internal mailing list AFAIK. Perhaps there has also been chatter about this on the community group mailing list (I'm not sure), which you can find here: https://www.w3.org/community/gpu/ If you don't see any relevant public threads there, I suggest you start one.

I am pretty sure that with blanked-out cross-origin content there would still be a risk of leaking cross-origin information via layout. There would have to be a lot of weird and complex constraints on CSS to prevent leaking the size of an image resource or the length of a paragraph of text, for example. This might seem like benign information to leak, but hackers can be quite creative about exploiting that to extract meaningful information. For example, detecting whether the user is signed-in to a third party website would be pretty easy I think.

On Tue, Mar 9, 2021 at 5:33 AM Andy Earnshaw notifications@github.com wrote:

That background is interesting, and so is the revival of the idea, do you have any links to those discussions? Thanks for sharing this!

The reason I mentioned blanking out cross-origin elements was we're creating images from user content and exporting them. That content might contain images that don't have CORS configured, but the user may not necessarily have control over that. Blanking out or replacing that content would allow those users to make use of the feature without having to remove all those images. You could work around that by iterating over all the elements beforehand and replacing them if they would taint the canvas, I suppose, but it's a little awkward to do this and it may cause some visual flickering (while content that is restored afterwards reloads). Some kind of callback would be useful in that case.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/whatwg/html/issues/706#issuecomment-793687232, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAT4V6CW327H3BUA7WGYCBLTCX2QVANCNFSM4B3U2JVQ .

[andyearnshaw]

Thanks for the link, I'll try to have a look there when I have a bit of free time.

With regards to leaking information from cross-origin sources, you'd already have access to layout information via the DOM elements you're painting from, so I don't think that's an issue.

[junov]

Oh, you're right. That state is already leaked. And it seems it is a common attack vector. I found this interesting paper on the topic: https://arxiv.org/pdf/1908.02204.pdf

On Tue, Mar 9, 2021 at 6:35 PM Andy Earnshaw notifications@github.com wrote:

Thanks for the link, I'll try to have a look there when I have a bit of free time.

With regards to leaking information from cross-origin sources, you'd already have access to layout information via the DOM elements you're painting from, so I don't think that's an issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/whatwg/html/issues/706#issuecomment-794281359, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAT4V6HSL34SIPCGPZBFHBLTCZS77ANCNFSM4B3U2JVQ .