[Proposal - 3D Tiles] Use an external library for 3D Tiles support

This issue is a feature proposal. Feel free to upvote (with 👍 ), comment and provide your use-cases if you're interested by this feature.

Context

(Originally discussed in https://github.com/iTowns/itowns/discussions/2146)

3D Tiles support in itowns is good but could be improved. 3D Tiles format is quite complex and maintaining its support is costly. In addition, 3D Tiles 1.1 has been released recently.

Proposal

Analyze the pros and cons and the different possibilities for using an external library to support 3D Tiles in itowns and decide if we should use one or keep the 3D Tiles support in itowns.

We have identified three possible libraries:

loaders.gl: contains everything to parse, and handle 3D tiles for rendering (culling for instance)
three-loader-3dtiles: Uses loaders.gl to load 3D tiles and displays them with threejs
3DTilesRendererJS: parses and renders 3D Tiles, based on threejs

The (unordered) elements to bear in mind when analyzing external libraries are:

License
3D Tiles support (1.0, 1.1 and also previous versions official or unofficial in which there are still tilesets available) and GLTF support (1.0 and 2.0)
Performance
Community, activity, etc.
Possibility to contribute
Dependencies / bundle size

Potential Problems

Losing support of some 3D tiles features
Losing cache control
Losing style control
Losing gltf 1.0 support

Documentation

Given the wide variety of 3D Tiles datasets available and of the variety of possible representations (meshes with or without textures, point clouds, gltfs, instanced meshes, multiple possible bounding volumes, draco and ktx2 compression, etc., etc.), we need to test against multiple datasets. Here are some public datasets that I have gathered. I also have datasets that I cannot share but I can run tests on them and post the results:

3D Tiles 1.0

b3dm without textures:

https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.0/TilesetWithDiscreteLOD: A dragon with multiple LOD in the USA, EPSG:4978
https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.0/TilesetWithRequestVolume: Tileset with request volume, EPSG:4978
https://saturnus.geodan.nl/maquette_nl_compressed/data/amsterdam_test/tileset.json: Amsterdam buildings, EPSG:3857

b3dm with textures:

https://github.com/iTowns/iTowns2-sample-data/tree/master/3DTiles/lyon_1_3946_textured_draco: Lyon 1er borrough, EPSG:3946
https://github.com/iTowns/iTowns2-sample-data/tree/master/3DTiles/lyon_1_4978: Lyon 1er borrough, EPSG:4978, draco compression

pnts

https://github.com/iTowns/iTowns2-sample-data/tree/master/3DTiles/lidar-hd-gorges-saint-chely-tarn: Gorges Saint Chely du tarn (from IGN LIDAR HD), EPSG:4978
https://github.com/iTowns/iTowns2-sample-data/tree/master/pointclouds/pnts-sete-2021-0756_6256: Sete (from IGN LIDAR HD), EPSG:4978, with classification
Lyon agglomeration: can be created with py3dtiles from this dataset

i3dm:

https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.0/TilesetWithTreeBillboards: instanciated trees, EPSG:4978

3D tiles 1.1

https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.1: Cesium official examples of 3D Tile 1.1 datasets

Other possible sources:

TU Delft 3D datasets list: https://3d.bk.tudelft.nl/opendata/opencities/
Univ Singapour list: https://ual.sg/project/ogbd/

For benchmarking, I identified three common usecases that can be tested:

Fixed scene loading: launch the page with a fixed camera position centered on the dataset (optionnally at multiple zoom levels)
3D navigation: simulates user navigation in the 3D scene with predefined camera positions and camera animation with CameraUtils functions to animate camera (don't forget to pan, rotate, etc. like a real user would)
Strong zoom: Start at a fixed camera position centered on the dataset (wait or not until data is loaded) and zoom a lot and quickly on the dataset

Some indicators that should be analyzed are:

FPS
GPU memory
Loading and rendering time
Visual impression (subjective but does it seems smooth, etc.)

A summary of my analysis and conclusion about the 3 libraries and about what we should do in my opinion (possible to give more information on demand on specific points).

3DTilesRendererJS:

Analysis:

Project active, well architectured, good 3D tiles 1.0 support overall, well tested
Only one dev working on it though
3D Tiles 1.1 not supported yet
Naive integration to itowns. First results:
- possible to use our own gltf loaders (1.0 and 2.0 support + specific old 3D Tiles spec implem)
- b3dm tiles are well supported and rendering is efficient (quicker than itowns)
- pnts tiles rendering is slower than itowns
- i3dm tiles are partially supported (only with internal gltf files)
functionally good, althoutgh improvements would be needed (especially for point clouds rendering)
Seems open to contribution
Manages all aspects of 3D Tiles rendering (parsing, converting, rendering, caching)
introduces a constraint on the threejs version (has three as a peer-dependency)

My opinion: Could be a quick win to improve 3D Tiles support and to reduce maintainance on this format but with a lot of drawbacks (point clouds perfs, bus factor, threejs peer-dependency).

three-loaders-gl:

Analysis:

based on loaders.gl for parsing
only one dev working on it and does not seem active anymore
Many of the tested datasets could not be displayed (tests based on the following POC)
Hard to benchmark but no visibile perf gain

My opinion: too risky and not enough gain.

loaders.gl:

project active and well supported although the 3D Tiles loader is not the most active loader.
used in deck.gl
Not POCed nor tested yet but I've created the following table comparing what's supported and what isn't in itowns and loaders.gl against the 3D tiles spec (+ some other criteria): https://lite.framacalc.org/6m2jo7j7av-a4c8

My opinion: Most of the 1.0 and 1.1 spec is supported by loaders.gl, good community, architecture and tests, it's the most promising one. Waiting on @Desplandis and @mgermerie POC to test its effective 3D Tiles support and its perfs once integrated to itowns.

Note that in the POC we should also determine whether we should use loaders.gl GLTFLoader or Threejs one or either one or the other depending on the dataset (considering support, especially for gltf 1.0, extensions and perfs).

for the other 3D tiles possibilities: Other possible sources: you might consider the nexus projects, https://github.com/cnr-isti-vclab/nexus 3D web tile visualization of large models, see examples: https://3dhop.net/demo.php

Finally we've decided to go with 3DTilesRendererJS. I'll let @Desplandis and @mgermerie detail why we decided not to choose loaders.gl since they experimented it.

I've created the following document comparing itowns and 3DtilesRendererJS support of 3D tiles. We can also use it to follow the migration status.

I started the migration on the following branch

Here is a list of the work to do, you can put your name next to it if you start working on it so we don't step on each other's toes :) Don't hesitate to add stuff to it too.

V0:

[x] Create a new 3D tiles layer using 3dtilesrendererjs and deprecate C3DTilesLayer
[x] Draco and KTX2 support
[x] Cesium ion support
[x] Google 3D tiles support
[x] Gltf 1.0 support
[x] CESIUM_RTC GLTF 1.0 extension support
[x] itowns PointsMaterial support (ongoing) @jailln
[x] Share cache and queues between ThreeDTilesLayers @jailln
[ ] Missing C3DTilesLayer options (decide if we should keep them or not: e.g. sseThreshold, cleanupDelay) @Desplandis
[x] Point clouds @Desplandis
- [x] SSE issue ? (happened between 0.3.26 and 0.3.30 version) : fixed in https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/623
- [x] Slower: fixed in https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/623
- [x] Batch table is never filled : fixed by v0.3.33
- [x] ~PointCloudMaterial: propose an API change to 3dtilesrendererjs to pass our own material~ : copy parsedMaterial to our own material instead.
[x] Attached layers won't work for ThreeDTilesLayer because getObjectToUpdateForAttachedLayers is called from MainLoop.updateElements which needs to have elements to update to be executed (while there is not because the update is done by 3dtilesrendererjs):
- [x] What is the usecase of attaching layers to 3D Tiles ? To attach a 2D layer that will be used to color a 3D tiles for instance? -> The only usecase was for debugging which is handled by 3d-tiles-renderer-js
- [x] ~Assert it is working with the current C3DTilesLayer~
- [x] Either prevent the possibility to attach layers or make it work with ThreeDtilesLayer (but we need a working usecase to test it): implement it with a warning saying it is not supported @jailln
[ ] Picking: ensure picking is working @Desplandis: https://github.com/iTowns/itowns/pull/2364 waiting on https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/627
[ ] Batch table: provide an easy API to access batch table data (the same than before or a new one ?) @mgermerie : https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/627

V1:

[ ] Perfs analyses @jailln @Desplandis
[ ] Style: @mgermerie @Desplandis (consider this: https://github.com/iTowns/itowns/issues/2336 + review the current implementation)
- [ ] implement Style support from C3DTilesLayer
- [ ] C3DTilesStyle widget
[ ] Debug tools / debug mode: Use DebugTilesRenderer or change the implementation to have a wrapper instead of a new type of TilesRenderer? (also so it can support IonTilesRenderer and GoogleTilesRenderer)
[ ] 3dtiles_ion.html does not work (because of batch_table_hierarchy extension that is not supported ? Or because of the bug that we resolved for cesium ion?) @jailln

V2:

[ ] Analyze new options offered by 3d-tiles-renderer-js TilesRenderer, LRUCache and co and add them in ThreeDTilesLayer constructor config options
[ ] Add the possibility to register an extension to OGC3DTilesLayer that registers it to 3d-tiles-renderer
[ ] Use (or provide the ability to use) 3dtilesrendererjs FadeTilesRenderer
[ ] Tests: functional tests are de facto migrated, unit tests can be ignored (not much to test). The first thing to do would be to test 3d-tiles-renderer-js instead.
[ ] Provide an easy API to access batch table data (e.g. getDataForId(tileId, batchId))
[ ] Full support of 3D Tiles 1.1:
- [ ] Implicit tiling extension @AnthonyGlt (ongoing)

Someday?

[ ] Batch table hierarchy extension support (really needed? Only for 3D tiles 1.0)

@jailln Forgot to answer this one. Nice job, we will assist you with @mgermerie on this one. =)

On the list of TODOs:

Share cache and queues between ThreeDTilesLayers

Analyze new options offered by 3d-tiles-renderer-js TilesRenderer, LRUCache and co and add them in ThreeDTilesLayer constructor config options

Yes, It would be cool to be able to centralize control of the cache and queues in iTowns.

Missing C3DTilesLayer options (decide if we should keep them or not: e.g. sseThreshold, cleanupDelay, registeredExtensions)

Batch table: provide an easy API to access batch table data (the same than before or a new one ?)

Shall we call for contributions on those topics ? By opening an issue on iTowns and/or on known projects using itowns' 3DTiles (e.g. UD-Viz)?

Point clouds

Yeah, we should be able to pass our own material. I could investigate the performance issues on this one.

Use (or provide the ability to use) 3dtilesrendererjs FadeTilesRenderer

Whoa, nice functionality to have!

Attached layers [...]

I will ask @gchoqueux for inputs.

To this list, we should add all 3D Tiles 1.1 functionalities that we will contribute back to 3d-tiles-renderer-js. =)

iTowns / itowns