Support .gltf with available KTX2 textures

[jwnimmer-tri]

Is your feature request related to a problem? Please describe.

When loading large scenes into Meshcat, a common bottleneck appears to involve textures. As of today, we typically use *.png images for textures in our glTF models.

Meshcat fetches the pngs in compressed form (i.e., our http server just serves the png file, still compressed), which is as expected and fairly fast. (Our new CAS cache helps here.) However, after fetching the data the client side decodes the image into full bitmaps (e.g., 4096x4096x3 byte array) and then uploads that bitmap to the GPU. In most cases the decoding is done in webworker, but the upload needs to happen on the main thread.

Uploading that bitmap is actually really, really slow (on the order of tenths of a second). A glTF with PBR materials will typically have several texture images (color, normal, roughness...). When loading a scene with multiple glTFs and their multiple textures, it can take tens of seconds before the threejs objects are fully loaded and show up on the screen.

That's too slow. We'd like to load the textures much, much faster.

Describe the solution you'd like

The glTF format offers an extension named KHR_texture_basisu. This uses the KTX2 texture format, typically stored in files named *.ktx2 with content type image/ktx2.

This format is optimized for speedy transmission (compression) and speedy upload to GPUs (they can consume it directly). In experiments so far, it substantially mitigates texture loading delays.

We'd like Drake's Meshcat to offer support for this extension. When it is part of extensionsUsed, Meshcat should load the KTX2 image instead of the PNG image.

Note that we would advise keeping the *.png images intact alongside the *.ktx2 images. For now, only Meshcat would know about the different texture encoding. Other gltf consumers (VTK, Blender, etc.) don't yet support that texture format, so they would still load the *.png images.

Describe alternatives you've considered

Use less realistic PNG textures with reduced pixel size.

[jwnimmer-tri]

As of https://github.com/meshcat-dev/meshcat/pull/176, Meshcat will actually load the new texture images with no fuss. Very easy.

The remaining work inside of Drake is to:

• [x] Demonstrate / document how to add fallback textures
  • [x] Our geometry file format docs should explain that Meshcat supports glTF's basisu extension, but that nothing else does.
    • [x] https://github.com/RobotLocomotion/drake/pull/21193
  • [~] Provide a tool to automatically rewrite a glTF file with PNG textures into a file with PNG+KTX2 textures.
    • Done. We have this in TRI's Anzu repository, and will open-source it after it's been more thoroughly tested.
• [x] Test and fix RenderEngineVtk on the new file convention
  • [x] #21090
  • [x] #21183
• [x] Test and fix RenderEngineGltfClient on the new file convention
• [x] #21249.
• [x] https://github.com/RobotLocomotion/models/issues/44

---

What we want for RenderEngineVtk is that it doesn't spew errors about the unknown extension. A single warning about the unknown extensionsUsed would be fine. Errors about unknown mime types on unused images are not OK:

OK (maybe):

Warning: In vtkGLTFDocumentLoaderInternals.cxx, line 1328
vtkGLTFDocumentLoader (0x5555b879b200): glTF extension KHR_texture_basisu is used in this model, but not supported by this loader. The extension will be ignored.

Not OK:

ERROR: In vtkGLTFDocumentLoaderInternals.cxx, line 621
vtkGLTFDocumentLoader (0x5555b879b200): Invalid image.mimeType value. Must be either image/jpeg or image/png for image pyramid_em
ERROR: In vtkGLTFDocumentLoaderInternals.cxx, line 621
vtkGLTFDocumentLoader (0x5555b879b200): Invalid image.mimeType value. Must be either image/jpeg or image/png for image pyramid_normal
ERROR: In vtkGLTFDocumentLoaderInternals.cxx, line 621
vtkGLTFDocumentLoader (0x5555b879b200): Invalid image.mimeType value. Must be either image/jpeg or image/png for image pyramid_ao-pyramid_mr
ERROR: In vtkGLTFDocumentLoaderInternals.cxx, line 621
vtkGLTFDocumentLoader (0x5555b879b200): Invalid image.mimeType value. Must be either image/jpeg or image/png for image cube_template

---

For RenderEngineGltfClient, probably no work is necessary, but we should test and make sure.

[jwnimmer-tri]

Perhaps a good "acceptance test" would be to convert tri_homecart to offer KTX2 textures (after https://github.com/RobotLocomotion/drake/issues/20932#issuecomment-1942161202).

[jwnimmer-tri]

My WIP branch for render engine fixes will be here.

[jwnimmer-tri]

Test and fix RenderEngineGltfClient on the new file convention.

Done.

As hoped for, there were no changes necessary. I ran the bazel run //examples/hardware_sim:demo_cc demo using drake-blender as the camera RPC server, and the output images were bit-identical when switching to the extensionsUsed. In other words, blender quietly ignored the unknown extension. Here's the Drake patch I tested: 20935-test.patch.zip. The test was switching between the current version of drake_models and the https://github.com/RobotLocomotion/models/pull/37 pull request version.

[jwnimmer-tri]

As of #21313, I think this Epic is finished.