KHR_accessor_float64 Draft Proposal

[lexaknyazev]

Split from #2395.

As discussed in the Interactivity DTSG, it's important for the animation pointer data to be able to represent the same Object Model property values that could be set via behavior graphs.

[bghgary]

How does this extension interact with mesh compression extensions like Draco or Meshopt?

[donmccurdy]

For Draco, here's an (old) quote...

• https://github.com/google/draco/issues/360

... so it doesn't sound like f64 is supported. Draco's "edgebreaker" codec will change vertex count and order, so compressing only a subset of a mesh primitive's attributes must be avoided. The "sequential" codec does not change vertex count if deduplication is disabled using the "Expert Encoder API". So with some work, f64 attributes could be skipped during compression. Perhaps support for compressing f64 attributes could be added to the Draco library without changes to KHR_draco_mesh_compression, I'm not sure.

---

For Meshopt, I don't think there's support for f64 in the library today, but vertex count and order does not change during compression, so excluding specific vertex attributes from compression is easy. That's probably what I'd do in glTF Transform, just skip compression for f64 attributes. I assume that adding f64 support to EXT_meshopt_compression is technically possible, but not trivial:

• https://github.com/zeux/meshoptimizer/issues/2#issuecomment-806359065

---

Not to say that KHR_accessor_float64 should prohibit these compression methods. But there are technical obstacles.

[emackey]

In theory, if one is using f64 for precision, one should not switch to quantized, right? That's less precision than f32.

[donmccurdy]

I tend to think of the component type's precision only as an upper bound on quantization precision. It's common to quantize POSITION accessors to 12–14 bits, stored in 16-bit integer accessors. HTTP-level compression will then reduce the transferred size. The same approach should work with f64, with accessors quantized to 32–64 bits. See gltfpack's -vpf flag, for an example with f32.

[lexaknyazev]

accessors quantized to 32–64 bits

Note that there are no 32- or 64-bit normalized data types.

@bghgary, @emackey, @javagl OK to move the draft to Release Candidate?

[bghgary]

OK to move the draft to Release Candidate?

Should we add sample models? We usually want to include sample models if the intention is to get implementations so that people can test.

[javagl]

I agree that sample models are important (and I'd probably see whether I can quickly create one of these "minimal test models", like the Triangle storing everything with double or so). But I don't know whether they are required for this to go from 'draft' to 'release candidate'.

[bghgary]

If RC means we want implementations (which I believe is the intention), then whoever implements the spec will need something to test. If we don't provide the sample, the implementors will have to create something instead which will make it less likely for the implementation to happen.

[javagl]

Don already covered the topic of generating test models via https://github.com/donmccurdy/glTF-Transform/pull/1417

In the meantime, if it matters, here's the AnimatedTriangle sample asset (embedded), using double/float64 values for the POSITION data and the rotation animation key frames/quaternions:

{
  "extensionsUsed" : [ "KHR_accessor_float64" ],
  "extensionsRequired" : [ "KHR_accessor_float64" ],
  "accessors" : [ {
    "bufferView" : 0,
    "byteOffset" : 0,
    "componentType" : 5123,
    "count" : 3,
    "type" : "SCALAR",
    "max" : [ 2 ],
    "min" : [ 0 ]
  }, {
    "bufferView" : 1,
    "byteOffset" : 0,
    "componentType" : 5130,
    "count" : 3,
    "type" : "VEC3",
    "max" : [ 1.0, 1.0, 0.0 ],
    "min" : [ 0.0, 0.0, 0.0 ]
  }, {
    "bufferView" : 2,
    "byteOffset" : 0,
    "componentType" : 5130,
    "count" : 5,
    "type" : "SCALAR",
    "max" : [ 1.0 ],
    "min" : [ 0.0 ]
  }, {
    "bufferView" : 2,
    "byteOffset" : 40,
    "componentType" : 5130,
    "count" : 5,
    "type" : "VEC4",
    "max" : [ 0.0, 0.0, 1.0, 1.0 ],
    "min" : [ 0.0, 0.0, 0.0, -0.7070000171661377 ]
  } ],
  "animations" : [ {
    "channels" : [ {
      "sampler" : 0,
      "target" : {
        "node" : 0,
        "path" : "rotation"
      }
    } ],
    "samplers" : [ {
      "input" : 2,
      "interpolation" : "LINEAR",
      "output" : 3
    } ]
  } ],
  "asset" : {
    "generator" : "JglTF from https://github.com/javagl/JglTF",
    "version" : "2.0"
  },
  "buffers" : [ {
    "uri" : "data:application/gltf-buffer;base64,AAABAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADwPwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPA/AAAAAAAAAAAAAAAAAAAAAAAAAAAAANA/AAAAAAAA4D8AAAAAAADoPwAAAAAAAPA/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA8D8AAAAAAAAAAAAAAAAAAAAAAAAAgL6f5j8AAACAvp/mPwAAAAAAAAAAAAAAAAAAAAAAAAAAAADwPwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIC+n+Y/AAAAgL6f5r8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADwPw==",
    "byteLength" : 280
  } ],
  "bufferViews" : [ {
    "buffer" : 0,
    "byteOffset" : 0,
    "byteLength" : 6,
    "target" : 34963
  }, {
    "buffer" : 0,
    "byteOffset" : 8,
    "byteLength" : 72,
    "target" : 34962
  }, {
    "buffer" : 0,
    "byteOffset" : 80,
    "byteLength" : 200
  } ],
  "meshes" : [ {
    "primitives" : [ {
      "attributes" : {
        "POSITION" : 1
      },
      "indices" : 0,
      "mode" : 4
    } ]
  } ],
  "nodes" : [ {
    "mesh" : 0,
    "rotation" : [ 0.0, 0.0, 0.0, 1.0 ]
  } ],
  "scene" : 0,
  "scenes" : [ {
    "nodes" : [ 0 ]
  } ]
}

---

EDIT: The buffer/bufferView/accessor structure of this file:

---

_{Generator code: GeneratedAnimatedTriangleFloat64.zip}

[donmccurdy]

Here's MorphStressTest with f64 accessors:

MorphStressTest.zip

I just picked this arbitrarily, as a scene containing both mesh data and animation. Converting other samples would be easy enough, happy to add whichever are wanted. The gist of the script (after other changes to support f64 in gltf transform) is ...

import { NodeIO } from '@gltf-transform/core';
import { KHRONOS_EXTENSIONS, KHRAccessorFloat64 } from '@gltf-transform/extensions';
import { dequantize } from '@gltf-transform/functions';

const io = new NodeIO().registerExtensions(KHRONOS_EXTENSIONS)
const document = await io.read('sample.glb');

function f64(options = {}) {

  return (document) => {
    document.createExtension(KHRAccessorFloat64).setRequired(true);

    for (const accessor of document.getRoot().listAccessors()) {
      if (accessor.getComponentType() === 5126 /* FLOAT */) {
        accessor.setArray(new Float64Array(accessor.getArray()));
      }
    }
  }

}

await document.transform(dequantize(), f64());

await io.write('sample_f64.glb', document);

So it's a direct conversion from f32 to f64, the additional precision is not required to render the asset correctly. Further changes will be required in glTF Transform to fully support f64 data, so I wouldn't expect features like mesh simplification or compression to work with f64 yet. See:

• https://github.com/donmccurdy/glTF-Transform/pull/1417

[zeux]

I assume that adding f64 support to EXT_meshopt_compression is technically possible, but not trivial

If filters aren’t used then it should just work. For translation type vectors filters are typically omitted or the exponential is used; the spec only allows f32 outputs for exponential filter so that wouldn't work. It isn’t hard to make it work but I doubt it’s super useful because that implies a precision tradeoff and f64 implies the opposite.

Without filters but with attribute codecs on f64 you will see some meaningful compression iff the values have less entropy than the format allows: for example I would expect reasonable results if f64 encodes f32 values, and minimal compression (10%?) for random values in 0..1 interval.

[zeux]

The coupling between animation usage and mesh/instance usage in this extension is unfortunate.

WebGL, WebGL2, WebGPU don't support double precision buffers or use of doubles in shaders. On desktop you do get some level of support through native APIs although it's conditional on hardware support and fairly sparsely represented in the API (eg in D3D12 there's no way to load 64-bit floats from a buffer directly, so you have to synthesize it from 2 32-bit halves manually).

Overall I would expect that the 99.99% path for a renderer to support this extension for accessors that require GPU access is to decode f64->f32 on load. This is obviously possible, and for some loaders this will not be hard because they don't work with buffer views directly and assume a "accessor -> GPU buffer" conversion model anyhow, but that does make life rather more difficult for renderers that map bufferViews to GPU directly, as you need to analyze accessors that refer to that.

On the flip side, choosing not to support this extension means you can't support f64 property animation, which might be undesirable. This extension could also help for some cases where you need eg instance positions to be f64 and you're using mesh_gpu_instancing extension purely as a compression technique (rendering instances one by one on CPU) or have special careful shader plumbing to actually support this via a GPU path...

I wish that this extension would at least not apply to mesh attributes but maybe there's some reasons to favor that.

[lexaknyazev]

that does make life rather more difficult for renderers that map bufferViews to GPU directly, as you need to analyze accessors that refer to that.

The sparse accessors feature from the base glTF 2.0 spec already requires implementations to be able to read/write individual attribute values.