Float16Array + HTML Canvas

[palemieux]

FYI. Below is input from the Color on the Web CG regarding Float16Array in the context of HTML Canvas. This input was prompted by https://github.com/w3c/ColorWeb-CG/issues/87

-- W3C Color on the Web CG chairs https://www.w3.org/community/colorweb/ public-colorweb@w3.org

====================

CG proposal re: Float16Array + HTML Canvas

The ColorWeb CG is seeking consensus with parties interested in discussing floating-point backing stores for canvas rendering contexts.

The CG spec proposal is: https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md

The primary outstanding issue at present is how to represent values which are read back from the canvas into typed arrays which can be manipulated from ECMAScript.

The current spec proposal supports readbacks into both Uint8ClampedArray and Float32Array via ImageDataColorType.

It has been suggested to also add support for readbacks into Float16Array (https://github.com/w3c/ColorWeb-CG/issues/87). This would make the specification dependent on the Float16Array type currently being defined by the ECMAScript committee (ISO/TC 39).

The CG supports the idea of adding support for Float16Array to ECMAScript. However, the CG feels it is most appropriate to decouple the ongoing improvements to ECMAScript from the work on floating-point canvases. Users of the web platform are expressing the need for floating-point canvas backing stores now; in Chromium, implementation is underway in crbug.com/1230619. Moreover, readback and CPU-side manipulation paths are not high-performance, so the additional memory bandwidth of using Float32Array for this task will not critically impact applications.

The ColorWeb CG therefore suggests:

1) Moving ahead with https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md in its current form, with unorm8 and float32 as options for readback.

2) In parallel, working on adding Float16Array to ECMAScript, and prototyping it in browsers.

3) Specifying a "float16" enum in ImageDataColorType once implementation experience has been achieved with Float16Array, . Applications can feature-detect its support in browsers by catching exceptions raised by getImageData or createImageData.

Background

For context, introducing a Float16Array into the Typed Array hierarchy was discussed several years ago among members of TC39, as well as the Khronos Group where Typed Arrays originated. At the time, the costs of supporting this type natively in ECMAScript engines outweighed the benefits for the following reasons:

1) For practical purposes, Float16Array could be sufficiently polyfilled in ECMAScript:

https://github.com/petamoriken/float16

Corner cases existed regarding denormalized values, NaNs, and infinities - but applications could generally achieve their desired results, with good performance on all ECMAScript engines, using existing primitives.

2) Direct CPU support for half-float numbers did not seem to exist. At the time, C libraries which worked with this data type universally emulated it, rather than using compiler intrinsics on certain platforms.

3) A significant amount of work would be required in every ECMAScript engine to make this typed array type perform well.

4) A significant amount of work would be required to specify the behavior of this numeric type in ECMAScript.

The landscape has changed in recent years. Per https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Hardware_support, ARM and x86 processors either support FP16 natively, or will in the near future. FP16 formats are increasingly heavily used in neural network evaluation and image processing, including on the web. The ECMAScript editors may be able to save spec work by referencing IEEE specifications (which, to be fair, already existed when Float16Array was originally considered).

[ccameron-chromium]

In the most strict sense:

• don't just throw an exception for float-backed canvases (this was suggested at the F2F)

In a wider sense:

• don't introduce changes that will cause existing code to break (e.g, changing the default return type)
• don't restrict its ability to perform format conversions (e.g, reading an 8-bit buffer as float or vice versa)

[mdrejhon]

Example usages are:

  // Returns an ImageData with a Uint8ClampedArray
  ctx.getImageData(1, 2, 3, 4);

  // Returns an ImageData with a Uint8ClampedArray
  ctx.getImageData(1, 2, 3, 4, {pixelFormat:'rgba8unorm'});

  // Returns an ImageData with a Float32Array
  ctx.getImageData(1, 2, 3, 4, {colorSpace:'display-p3', pixelFormat:'rgba32float'}); 

I think that's a great compromise without breaking getImageData;

Long term, as creator of TestUFO I need direct access to WCG/HDR data at its full precision, and eventually access to tonemapping data, so I can at least determine native target lumens (before displays' clipping decisions), to scientifically quantify display measurements. Some tests are just demos, teaching animations, while others are used for display measurements. However, just getting access to WCG/HDR in a canvas, is a big first step.

As of September 1st, 2024, beta.testufo.com now does HDR Canvas 2D in Chrome via experimental flag. Test whether WCG/HDR is working at beta.testufo.com/hdr. I've added HDR color picker support where applicable among the other selectable tests at top.

So I'm already butting against limitations of getImageData() because I would love to render a HDR pallete in a canvas (e.g. beta.testufo.com/palette and simply getImageData() from it, for an improved WCG+HDR colorpicker for certain display tests (like HDR pixel response tests), some tests like /flicker are used with an oscilloscope.

[BlobTheKat]

10624

[mdrejhon]

What are the use cases for Float32Array?

Here's My Industry Answer As Display Scientific Expert

While float16 is good enough, there are some use cases (10 year window) for float32 such as linearized HDR

Or doing large number of canvas operations (matching two images together repeatedly).

We necessarily have to go non-linear to avoid banding artifacts with float16, which is why PQ and Hybrid Log Gamma exists.

16-bit is not enough precision for linearized HDR over the entire dynamic range when doing adjacent shades in a cave versus adjacent shades in direct sun.

This was not important in 15" VGA CRT era

But important for gigantic curved 49" super-ultra wide (32:9) displays where some pixels are now outside your peripheral vision if you sit close enough.

Or different content in different head-facing directions in VR allowing your eyes to adapt to the new dynamic range subset.

Screens that exceed FOV now show banding at 16bit linearized in these extreme cases!

A screen optimized to one dynamic range shows different banding at a different dynamic range once your eyes adjust to the new brightness of the new direction you're facing on the giant-FOV screen.

The increase in FOV means now you have to account for human effect of exiting/entering cave vs outdoors, simply by a human facing different parts of the same giant screen. This amplifies banding.

Screens are getting bigger, you see? And now Apple Vision Pro and similar use cases of virtual screens. But it also applies to fiant-FOV screens like current Corsair Xeneon Flex 45" 240Hz OLED that I use to make some of the replies here.

2030s GPUs with float32 frame buffers may want native faster access to linearized HDR if math ops are more efficient in the linear space, especially for high-processing cases with fewer workarounds than using nonlinear color spaces. Linear color space means pixel values more linearly corresponds to photons/sec count, allowing easier maths.

Obviously, this isn't important to "I just wanna browse" use cases, but if we want native accsss to future GPU color spaces, permanent float16 assumption for 2030s = gun pointed to foot for architecture-self-into-corner.

Fine, (as long as we're not doing any image math/peocessing) we don't perceptually need more than float16 -- but only as long as we're not doing LINEAR float16 on a Vegas Sphere quality VR headset with little light back scatter (e.g. improved lens like post-pancake lens). Where you are in a cave entranceway and turn your headset back and forth between the dark and bright, giving enough time for eye adjustment to see different in cave shades of colors versus later shades of daylight. And that's without doing lots of image math without banding rounding errors. We need float32 for Holodeck displays where FOV amplifies quantization due to eye adjustment simply by facing different parts of FOV.

P.S. The Apple employees in the WebKit department needs to talk to the Vision Pro people, to self-educate on these sciences. By the way, I am cited in over 30 research papers, and I also teach display-sciences training classrooms [image]

Two scientific rasion d'être's exceeds more than enough.

We do not ever need to make float32 default, or even standardize today, but there should be a decade path to supporting it natively. Support float16 today, but keep paths to other GPU-native color channel data types in future.

Note: I work with display manufacturers. Its amazing how weak links shows in extreme tests.

Either way, float16 standardization should be expedited, as long as "does our float16 support path allow future support for GPU native color channel formats such as float32?" tests pass.

The @ccameron-chromium path passes that test.