Proceed with `Float16Array`

[petamoriken]

Float16Array is proposed in TC39 at Stage 1. Can we move in the direction of advancing that?

---

Choice of Float32Array for ImageData

There does not exist a Float16Array type, and so it is not an option. The only other floating-point type, Float64Array, is of unnecessarily high precision.

https://github.com/w3c/ColorWeb-CG/blob/14bffcb85327b011026def4f7ebf9b88e724afad/canvas_float.md#choice-of-float32array-for-imagedata

[petamoriken]

Another option is to use Uint16Array.

---

FYI: WebGL 2.0 uses Uint16Array for half float texture.

https://registry.khronos.org/webgl/specs/latest/2.0/#3.7.6

[palemieux]

See also https://lists.w3.org/Archives/Public/public-colorweb/2022Dec/0014.html

[palemieux]

Below is consensus input from the CG per https://lists.w3.org/Archives/Public/public-colorweb/2023Jan/0000.html

CG proposal re: Float16Array + HTML Canvas

The ColorWeb CG is seeking consensus with parties interested in discussing floating-point backing stores for canvas rendering contexts.

The CG spec proposal is: https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md

The primary outstanding issue at present is how to represent values which are read back from the canvas into typed arrays which can be manipulated from ECMAScript.

The current spec proposal supports readbacks into both Uint8ClampedArray and Float32Array via ImageDataColorType.

It has been suggested to also add support for readbacks into Float16Array (https://github.com/w3c/ColorWeb-CG/issues/87). This would make the specification dependent on the Float16Array type currently being defined by the ECMAScript committee (ISO/TC 39).

The CG supports the idea of adding support for Float16Array to ECMAScript. However, the CG feels it is most appropriate to decouple the ongoing improvements to ECMAScript from the work on floating-point canvases. Users of the web platform are expressing the need for floating-point canvas backing stores now; in Chromium, implementation is underway in crbug.com/1230619. Moreover, readback and CPU-side manipulation paths are not high-performance, so the additional memory bandwidth of using Float32Array for this task will not critically impact applications.

The ColorWeb CG therefore suggests:

1) Moving ahead with https://github.com/w3c/ColorWeb-CG/blob/main/canvas_float.md in its current form, with unorm8 and float32 as options for readback.

2) In parallel, working on adding Float16Array to ECMAScript, and prototyping it in browsers.

3) Specifying a "float16" enum in ImageDataColorType once implementation experience has been achieved with Float16Array, . Applications can feature-detect its support in browsers by catching exceptions raised by getImageData or createImageData.

Background

For context, introducing a Float16Array into the Typed Array hierarchy was discussed several years ago among members of TC39, as well as the Khronos Group where Typed Arrays originated. At the time, the costs of supporting this type natively in ECMAScript engines outweighed the benefits for the following reasons:

1) For practical purposes, Float16Array could be sufficiently polyfilled in ECMAScript:

https://github.com/petamoriken/float16

Corner cases existed regarding denormalized values, NaNs, and infinities - but applications could generally achieve their desired results, with good performance on all ECMAScript engines, using existing primitives.

2) Direct CPU support for half-float numbers did not seem to exist. At the time, C libraries which worked with this data type universally emulated it, rather than using compiler intrinsics on certain platforms.

3) A significant amount of work would be required in every ECMAScript engine to make this typed array type perform well.

4) A significant amount of work would be required to specify the behavior of this numeric type in ECMAScript.

The landscape has changed in recent years. Per https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Hardware_support, ARM and x86 processors either support FP16 natively, or will in the near future. FP16 formats are increasingly heavily used in neural network evaluation and image processing, including on the web. The ECMAScript editors may be able to save spec work by referencing IEEE specifications (which, to be fair, already existed when Float16Array was originally considered).

[chrisn]

TC39 discussed Float16Array in their March meeting - see notes.

[petamoriken]

Update: the proposal to add Float16Array to JavaScript today reached stage 3, meaning the design is finished, the committee is in favor, and engines can start implementing and shipping it.

https://github.com/KhronosGroup/WebGL/issues/3531#issuecomment-1550494818

[mdrejhon]

Will the HDR CANVAS version of .getImageData() and .putImageData() migrate to FP16?

As a stopgap, I'd even settle for 16-bit integers (and use software-based pixel processing) and/or putting the numbers in higher-precision floating point numbers.

I eventually need to be able to HDR-colorpicker for TestUFO, an existing HDR CANVAS color swatch rendered by gradients/etc. As TestUFO HDR 2.0 is about to be launched (see my comment at https://github.com/w3c/ColorWeb-CG/issues/111#issuecomment-1833094245)...