[Proposal - 3D Tiles] Use an external library for 3D Tiles support

[jailln]

This issue is a feature proposal. Feel free to upvote (with 👍 ), comment and provide your use-cases if you're interested by this feature.

Context

(Originally discussed in https://github.com/iTowns/itowns/discussions/2146)

3D Tiles support in itowns is good but could be improved. 3D Tiles format is quite complex and maintaining its support is costly. In addition, 3D Tiles 1.1 has been released recently.

Proposal

Analyze the pros and cons and the different possibilities for using an external library to support 3D Tiles in itowns and decide if we should use one or keep the 3D Tiles support in itowns.

We have identified three possible libraries:

• loaders.gl: contains everything to parse, and handle 3D tiles for rendering (culling for instance)
• three-loader-3dtiles: Uses loaders.gl to load 3D tiles and displays them with threejs
• 3DTilesRendererJS: parses and renders 3D Tiles, based on threejs

The (unordered) elements to bear in mind when analyzing external libraries are:

• License
• 3D Tiles support (1.0, 1.1 and also previous versions official or unofficial in which there are still tilesets available) and GLTF support (1.0 and 2.0)
• Performance
• Community, activity, etc.
• Possibility to contribute
• Dependencies / bundle size

Potential Problems

• Losing support of some 3D tiles features
• Losing cache control
• Losing style control
• Losing gltf 1.0 support

Documentation

Given the wide variety of 3D Tiles datasets available and of the variety of possible representations (meshes with or without textures, point clouds, gltfs, instanced meshes, multiple possible bounding volumes, draco and ktx2 compression, etc., etc.), we need to test against multiple datasets. Here are some public datasets that I have gathered. I also have datasets that I cannot share but I can run tests on them and post the results:

3D Tiles 1.0

b3dm without textures:

• https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.0/TilesetWithDiscreteLOD: A dragon with multiple LOD in the USA, EPSG:4978
• https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.0/TilesetWithRequestVolume: Tileset with request volume, EPSG:4978
• https://saturnus.geodan.nl/maquette_nl_compressed/data/amsterdam_test/tileset.json: Amsterdam buildings, EPSG:3857

b3dm with textures:

• https://github.com/iTowns/iTowns2-sample-data/tree/master/3DTiles/lyon_1_3946_textured_draco: Lyon 1er borrough, EPSG:3946
• https://github.com/iTowns/iTowns2-sample-data/tree/master/3DTiles/lyon_1_4978: Lyon 1er borrough, EPSG:4978, draco compression

pnts

• https://github.com/iTowns/iTowns2-sample-data/tree/master/3DTiles/lidar-hd-gorges-saint-chely-tarn: Gorges Saint Chely du tarn (from IGN LIDAR HD), EPSG:4978
• https://github.com/iTowns/iTowns2-sample-data/tree/master/pointclouds/pnts-sete-2021-0756_6256: Sete (from IGN LIDAR HD), EPSG:4978, with classification
• Lyon agglomeration: can be created with py3dtiles from this dataset

i3dm:

• https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.0/TilesetWithTreeBillboards: instanciated trees, EPSG:4978

3D tiles 1.1

• https://github.com/CesiumGS/3d-tiles-samples/tree/main/1.1: Cesium official examples of 3D Tile 1.1 datasets

Other possible sources:

• TU Delft 3D datasets list: https://3d.bk.tudelft.nl/opendata/opencities/
• Univ Singapour list: https://ual.sg/project/ogbd/

For benchmarking, I identified three common usecases that can be tested:

• Fixed scene loading: launch the page with a fixed camera position centered on the dataset (optionnally at multiple zoom levels)
• 3D navigation: simulates user navigation in the 3D scene with predefined camera positions and camera animation with CameraUtils functions to animate camera (don't forget to pan, rotate, etc. like a real user would)
• Strong zoom: Start at a fixed camera position centered on the dataset (wait or not until data is loaded) and zoom a lot and quickly on the dataset

Some indicators that should be analyzed are:

• FPS
• GPU memory
• Loading and rendering time
• Visual impression (subjective but does it seems smooth, etc.)

[jailln]

A summary of my analysis and conclusion about the 3 libraries and about what we should do in my opinion (possible to give more information on demand on specific points).

3DTilesRendererJS:

Analysis:

• Project active, well architectured, good 3D tiles 1.0 support overall, well tested
• Only one dev working on it though
• 3D Tiles 1.1 not supported yet
• Naive integration to itowns. First results:
  • possible to use our own gltf loaders (1.0 and 2.0 support + specific old 3D Tiles spec implem)
  • b3dm tiles are well supported and rendering is efficient (quicker than itowns)
  • pnts tiles rendering is slower than itowns
  • i3dm tiles are partially supported (only with internal gltf files)
• functionally good, althoutgh improvements would be needed (especially for point clouds rendering)
• Seems open to contribution
• Manages all aspects of 3D Tiles rendering (parsing, converting, rendering, caching)
• introduces a constraint on the threejs version (has three as a peer-dependency)

My opinion: Could be a quick win to improve 3D Tiles support and to reduce maintainance on this format but with some drawbacks (point clouds perfs, bus factor, threejs peer-dependency).

three-loaders-gl:

Analysis:

• based on loaders.gl for parsing
• only one dev working on it and does not seem active anymore
• Many of the tested datasets could not be displayed (tests based on the following POC)
• Hard to benchmark but no visibile perf gain

My opinion: too risky and not enough gain.

loaders.gl:

• project active and well supported although the 3D Tiles loader is not the most active loader.
• used in deck.gl
• Not POCed nor tested yet but I've created the following table comparing what's supported and what isn't in itowns and loaders.gl against the 3D tiles spec (+ some other criteria): https://lite.framacalc.org/6m2jo7j7av-a4c8

My opinion: Most of the 1.0 and 1.1 spec is supported by loaders.gl, good community, architecture and tests, it's the most promising one. Waiting on @Desplandis and @mgermerie POC to test its effective 3D Tiles support and its perfs once integrated to itowns.

Note that in the POC we should also determine whether we should use loaders.gl GLTFLoader or Threejs one or either one or the other depending on the dataset (considering support, especially for gltf 1.0, extensions and perfs).

[PerspectivesLab]

for the other 3D tiles possibilities: Other possible sources: you might consider the nexus projects, https://github.com/cnr-isti-vclab/nexus 3D web tile visualization of large models, see examples: https://3dhop.net/demo.php

[jailln]

Finally we've decided to go with 3DTilesRendererJS. I'll let @Desplandis and @mgermerie detail why we decided not to choose loaders.gl since they experimented it.

I've created the following document comparing itowns and 3DtilesRendererJS support of 3D tiles. We can also use it to follow the migration status.

I started the migration on the following branch

[jailln]

Here is a list of the work to do, you can put your name next to it if you start working on it so we don't step on each other's toes :) Don't hesitate to add stuff to it too.

V0:

• [x] Create a new 3D tiles layer using 3dtilesrendererjs and deprecate C3DTilesLayer
• [x] Draco and KTX2 support
• [x] Cesium ion support
• [x] Google 3D tiles support
• [x] Gltf 1.0 support
• [x] CESIUM_RTC GLTF 1.0 extension support
• [x] itowns PointsMaterial support (ongoing) @jailln
• [x] Share cache and queues between ThreeDTilesLayers @jailln
• [x] Missing C3DTilesLayer options (decide if we should keep them or not: e.g. sseThreshold, cleanupDelay) @Desplandis
• [x] Point clouds @Desplandis
  • [x] SSE issue ? (happened between 0.3.26 and 0.3.30 version) : fixed in https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/623
  • [x] Slower: fixed in https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/623
  • [x] Batch table is never filled : fixed by v0.3.33
  • [x] ~PointCloudMaterial: propose an API change to 3dtilesrendererjs to pass our own material~ : copy parsedMaterial to our own material instead.
• [x] Attached layers won't work for ThreeDTilesLayer because getObjectToUpdateForAttachedLayers is called from MainLoop.updateElements which needs to have elements to update to be executed (while there is not because the update is done by 3dtilesrendererjs):
  • [x] What is the usecase of attaching layers to 3D Tiles ? To attach a 2D layer that will be used to color a 3D tiles for instance? -> The only usecase was for debugging which is handled by 3d-tiles-renderer-js
  • [x] ~Assert it is working with the current C3DTilesLayer~
  • [x] Either prevent the possibility to attach layers or make it work with ThreeDtilesLayer (but we need a working usecase to test it): implement it with a warning saying it is not supported @jailln
• [x] Picking: ensure picking is working @Desplandis: https://github.com/iTowns/itowns/pull/2364 waiting on https://github.com/NASA-AMMOS/3DTilesRendererJS/pull/627
• [x] Bump to v0.3.36 (see changelog)
• [x] 3dtiles_ion.html does not work (because of batch_table_hierarchy extension that is not supported ? Or because of the bug that we resolved for cesium ion?) @jailln: fixed in 3d-tiles-renderer
• [x] Batch table hierarchy extension support

V1:

• [x] Assert why some of the functional tests don't work with the new OGC3DTilesLayer @jailln
• [x] Update 3dtiles_loader example: @jailln
  • [ ] fix "multiple instances of three imported waning"
  • [x] don't reload the page each time we change the url of the tileset
  • [x] Add a terrain (world)
  • [x] Remove the other 3D Tiles examples
  • [x] Add buttons to load pre-defined 3D tiles datasets
  • [ ] write functional tests Attempt here, unsuccessful for now, delayed.
• [ ] Rename getC3DTileFeatureFromIntersectsArray @Desplandis
• [ ] Remove C3DTilesLayer from the documentation @Desplandis
• [x] Perfs analyses @jailln @Desplandis
• [ ] Style: @mgermerie @Desplandis (consider this: https://github.com/iTowns/itowns/issues/2336 + review the current implementation)
  • [ ] implement Style support from C3DTilesLayer
  • [ ] C3DTilesStyle widget
  • [ ] Improve classification (move it to style)
• [x] Add a deprecation warning in C3DTilesLayer constructor: will be removed in 3.0 @jailln
• [x] Analyze new options offered by 3d-tiles-renderer-js TilesRenderer, LRUCache and co and add them in ThreeDTilesLayer constructor config options @jailln -> done through OGC3DTilesLayer.tilesRenderer
• [ ] Add the possibility to register an extension to OGC3DTilesLayer that registers it to 3d-tiles-renderer
• [ ] Use (or provide the ability to use) 3dtilesrendererjs FadeTilesRenderer @jailln => not exposed by 3DTilesRenderer yet and after some tests, not fully ready yet, we leave the choice to each user to find a way to use it.
• [x] Browse itowns 3D Tiles issues and assert if usage of the new layer fixes them @jailln
• [ ] Provide an easy API to access batch table data (e.g. getDataForId(tileId, batchId)) => use 3DTilesRenderer one instead

[Desplandis]

@jailln Forgot to answer this one. Nice job, we will assist you with @mgermerie on this one. =)

On the list of TODOs:

• Share cache and queues between ThreeDTilesLayers

• Analyze new options offered by 3d-tiles-renderer-js TilesRenderer, LRUCache and co and add them in ThreeDTilesLayer constructor config options

Yes, It would be cool to be able to centralize control of the cache and queues in iTowns.

• Missing C3DTilesLayer options (decide if we should keep them or not: e.g. sseThreshold, cleanupDelay, registeredExtensions)

• Batch table: provide an easy API to access batch table data (the same than before or a new one ?)

Shall we call for contributions on those topics ? By opening an issue on iTowns and/or on known projects using itowns' 3DTiles (e.g. UD-Viz)?

• Point clouds

Yeah, we should be able to pass our own material. I could investigate the performance issues on this one.

• Use (or provide the ability to use) 3dtilesrendererjs FadeTilesRenderer

Whoa, nice functionality to have!

• Attached layers [...]

I will ask @gchoqueux for inputs.

To this list, we should add all 3D Tiles 1.1 functionalities that we will contribute back to 3d-tiles-renderer-js. =)