HTMLCanvasElement.toDataURL() is broken with respect to CSS scaling

juj commented 4 years ago

https://html.spec.whatwg.org/multipage/canvas.html#serialising-bitmaps-to-a-file

The image file's pixel data must be the bitmap's pixel data scaled to one image pixel per coordinate space unit, and if the file format used supports encoding resolution metadata, the resolution must be given as 96dpi (one image pixel per CSS pixel).

Not sure how this wording should be interpreted, but to me it looks like the implementation of canvas.toDataURL() is broken in all current browsers with respect to CSS Scaling (window.devicePixelRatio handling).

The spec states in https://html.spec.whatwg.org/multipage/canvas.html#dom-canvas-todataurl for canvas.toDataURL() that Let file be a serialization of this canvas element's bitmap as a file, passing type and quality if given.. The function name .toDataURL() and the wording about serialization suggests that the conversion should be as lossless as possible, i.e. a "dump" of the contents of the canvas to the specified format.

However this is not the case in the implementation of any current browser, but when testing the following code in Safari, Firefox and Chrome on a MacBook with window.devicePixelRatio==2, they all upscale the resulting image file to 2x of its original size.

<html>
<body>
<canvas width=300 height=150>
<script>
var canvas = document.querySelector('canvas');
canvas.style.width = (canvas.width / window.devicePixelRatio) + 'px';
canvas.style.height = (canvas.height / window.devicePixelRatio) + 'px';
var image = new Image();
image.src = canvas.toDataURL();
image.onload = function() {
    document.body.innerHTML +=
`<div>window.devicePixelRatio: ${window.devicePixelRatio}<br>
canvas.width: ${canvas.width} hardware pixels<br>
canvas.height: ${canvas.height} hardware pixels<br>
canvas CSS width: ${canvas.getBoundingClientRect().width} CSS pixels<br>
canvas CSS height: ${canvas.getBoundingClientRect().height} CSS pixels<br>
canvas.toDataURL() -> image.width: ${image.width} CSS pixels (${image.width*window.devicePixelRatio} hardware pixels)<br>
canvas.toDataURL() -> image.height: ${image.height} CSS pixels (${image.height*window.devicePixelRatio} hardware pixels)<br>
canvas.toDataURL() -> image.naturalWidth: ${image.naturalWidth} CSS pixels (${image.naturalWidth*window.devicePixelRatio} hardware pixels)<br>
canvas.toDataURL() -> image.naturalHeight: ${image.naturalHeight} CSS pixels (${image.naturalHeight*window.devicePixelRatio} hardware pixels)<br>
</div`;
};
</script>
</body>
</html>

Testing on my Macbook prints

window.devicePixelRatio: 2
canvas.width: 300 hardware pixels
canvas.height: 150 hardware pixels
canvas CSS width: 150 CSS pixels
canvas CSS height: 75 CSS pixels
canvas.toDataURL() -> image.width: 300 CSS pixels (600 hardware pixels)
canvas.toDataURL() -> image.height: 150 CSS pixels (300 hardware pixels)
canvas.toDataURL() -> image.naturalWidth: 300 CSS pixels (600 hardware pixels)
canvas.toDataURL() -> image.naturalHeight: 150 CSS pixels (300 hardware pixels)

Not sure if this is accidentally intended by the spec, or a bug of the implementation in all browsers, but such resizing of the image certainly goes against the spirit of what serialization should be about?

There are 4K laptops out there that have window.devicePixelRatio==4, for those taking a "serialization" of a fullscreen 3840x2160 canvas would result in a 15360x8640 pixel image file getting generated. (e.g. in our use case for WebGL unit test harness)

Also, window.devicePixelRatio is affected by the user agent's UI zoom level. If the test page is visited with browser page zoomed in to +300% zoom level (refresh the page with that zoom level to reload), then the page outputs

window.devicePixelRatio: 6
canvas.width: 300 hardware pixels
canvas.height: 150 hardware pixels
canvas CSS width: 50 CSS pixels
canvas CSS height: 25 CSS pixels
canvas.toDataURL() -> image.width: 300 CSS pixels (1800 hardware pixels)
canvas.toDataURL() -> image.height: 150 CSS pixels (900 hardware pixels)
canvas.toDataURL() -> image.naturalWidth: 300 CSS pixels (1800 hardware pixels)
canvas.toDataURL() -> image.naturalHeight: 150 CSS pixels (900 hardware pixels)

It would be very counterintuitive if a function that intends to serialize the contents of a canvas to an image file, would perform an upscaling (or lossy downscaling if browser zoom is at < 100%) based on user's browser zoom level.

Either all browser implementations have this wrong, or if the spec originally intended it to work like this, I think the spec should be reworded so that a proper non-resizing serialization is done. What do you think?

annevk commented 4 years ago

cc @whatwg/canvas

fserb commented 4 years ago

This looks like correct behavior to me. toDataURL returns the size of the backing buffer (canvas.width, canvas.height). devicePixelRatio is mostly irrelevant to toDataURL. There's no upscale/downscale happening anywhere.

The paragraph you quote is indeed slightly weird, as it's referring the file format's explicit resolution (like SVG), while talking about the bitmap data. We should amend that.

juj commented 4 years ago

CC @kenrussell @jdashg

juj commented 4 years ago

toDataURL returns the size of the backing buffer (canvas.width, canvas.height). devicePixelRatio is mostly irrelevant to toDataURL. There's no upscale/downscale happening anywhere.

Hmm, are you sure? The backing buffer in this test is 300x150, but the resulting image is 600x300? That looks like a 2x resize?

fserb commented 4 years ago

What makes you think that image.width is in CSS Pixels? It's not.

juj commented 4 years ago

Oh? https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement writes

HTMLImageElement.width
    An integer value that reflects the width HTML attribute, indicating the rendered width of the image in CSS pixels.

I did not dare to call that being wrong, but perhaps that is the root confusion then?

fserb commented 4 years ago

Well. I think there's a bit of a confusion here. Let me try again.

If you load a random image with an image element that is 80x80, the size of the object will be "devicePixelRatio-independent". I.e., talking about "hardware pixels" is irrelevant - the width * dpr means nothing. That said, the HTMLImageElement that is used to present this is 80x80 in "CSS Pixels", which means a 80*dpr monitor pixels size. What happens in the end is just an upscaling of the original image while compositing it. Nowhere in memory a 80*dpr image is created, in theory.

The exact same thing happens with canvas. A 300x150 canvas will have a backing storage of 300x150, even if the logical size of the element is 300 CSS Pixels (and 300*dpr monitor pixels). There's an upscaling when compositing (or whatever, depending if there are other CSS transforms at play).

juj commented 4 years ago

Thanks for the clarification. I was indeed not aware that the native/"hardware size" of an Image is interpreted "hidden" like this - unlike it is with canvas.

Is there a JavaScript API to get the actual pixel size of an image? It is far from irrelevant when interacting with canvases, where drawing is done in hardware pixel units (e.g. in WebGL).

juj commented 4 years ago

Is there a JavaScript API to get the actual pixel size of an image?

Oh hmm, nevermind, the size will be image.naturalWidth/.naturalHeight pixels.

juj commented 4 years ago

Thanks for the fast turnaround, I suppose this can be closed - I was confused by the wording on MDN, where the vocabulary for Image was unfamiliar coming from working with WebGL canvases.

tabatkins commented 4 years ago

Note that the quoted text in the OP is meant to be addressing the topic of high-resolution backing canvases. If your canvas is using a 2x "high DPI" backing store, then you want toDataUrl() to produce an image that'll, assigned to an <img> with no additional sizing information, produces an image natively sized the same as the <canvas> natively sizes. That requires downsampling it to a 1x resolution, currently; if it just directly reflected the backing store, it would indeed double in size.

(Can additional headers be sent in a data url? If so, as the DPI-header proposal matures, we could handle such a case that way instead, keeping the exact image data and informing the <img> to render it as a 2x/etc image.)

So far, however, high-resolution backing canvases have been dropped as a feature; right now you achieve that by just making a double-size canvas and setting its CSS width and height to half its native width and height, giving you an effective 2x resolution. If you want a 1x data url out of that, you need to draw it into a half-sized canvas, so the drawing operation takes care of the downsampling for you.

kdashg commented 4 years ago

https://html.spec.whatwg.org/multipage/embedded-content.html#dom-img-width:

The IDL attributes width and height must return the rendered width and height of the image, in CSS pixels, if the image is being rendered, and is being rendered to a visual medium; or else the density-corrected intrinsic width and height of the image, in CSS pixels, if the image has intrinsic dimensions and is available but not being rendered to a visual medium; or else 0, if the image is not available or does not have intrinsic dimensions. [CSS]

On setting, they must act as if they reflected the respective content attributes of the same name.

The IDL attributes naturalWidth and naturalHeight must return the density-corrected intrinsic width and height of the image, in CSS pixels, if the image has intrinsic dimensions and is available, or else 0.

In short, naturalWidth/Height are the "source ~pixel size, if available". width/height are dual-purpose: If in the dom/css/'rendering', it's the css pixel counts, else it's the naturalWidth/Height "resource pixel counts".

WebGL has no clue about its CSS size, and so doesn't change behavior based on CSS style.width/height. (It only checks canvas width/height, which is always the backing-store size, which is drawingBufferWidth/Height in WebGL)

annevk commented 4 years ago

Reopening this as per https://github.com/whatwg/html/issues/5387#issuecomment-602550868 we should change the paragraph quoted in OP.

whatwg / html

HTMLCanvasElement.toDataURL() is broken with respect to CSS scaling #5387