search

qgis / QGIS-Enhancement-Proposals

QEP's (QGIS Enhancement Proposals) are used in the process of creating and discussing new enhancements for QGIS
118 stars 37 forks source link

Streaming large 3D dataset, support for 3Dtiles #225

Closed benoitdm-oslandia closed 1 year ago

benoitdm-oslandia commented 3 years ago

QGIS Enhancement: Streaming large 3D dataset, support for 3Dtiles

Date 2021/05/03

Author Benoit De Mezzo (benoit.de.mezzo@oslandia.com)

Contact benoit dot de dot mezzo at oslandia dot com

maintainer @benoitdm-oslandia

Version QGIS 3.18

Summary

We want to provide a software architecture in QGIS to integrate the loading of large 3D datasets via the 3D Tiles format.

This architecture should:

To fullfil this QEP we already have secured some funding to work on mainly with Métropole Européenne de Lille (@Jean-Roc).

Proposed Solution

As part of this mission we will develop a partial (but working) implementation of 3D Tiles which includes:

We also needs to:

We will try to provide a class architecture to be able to support other formats (I3S, etc.).

Example(s)

An example implementation without the management of featureTables and batchTables is available in this branch.

Implementations of the QgsChunkLoader, QgsChunkedEntity, QgsChunkLoaderFactory, QgsAbstract3DRenderer and QgsMapLayer classes have been created for the LOD in the directory src/3d/3dtiles.

As part of this POC, the qgis_3d_sandbox test has been modified in order to apply this workflow:

  1. load the tileset from a tileset.json file passed as a parameter (via the 3D Tiles classes we created)
  2. create a Qgs3dTilesLayer using the previously loaded tileset
  3. create a Qgs3dTilesLayer3DRenderer using the previously created layer. This class will use the Qgs3dTilesChunkLoaderFactory and the Qgs3dTilesChunkedEntity

For now, the loaded meshs are replaced by torus:

lod_bbox_torus

Affected Files

Issues

3D stack

The 3D stack to handle the chunk loading is missing documentation and samples to properly understood the usecases and the proper way to implement a new chunk loader.

As a result the proposed implementation may lack the proper software designs to match the requirement of the chunk loading API. For example, the background loading thread, the unloading process, loader factory usage, root node/children nodes creation, etc.

Gltf

The gltf support via Qt3D only allow to read specific format via url/uri.

It does NOT allow direct reading from the binary format included in theb3dm, it only supports the JSON text format and only via file name (in the GLTFImporter class, the setData function is not exported in the API).

To overcome this shortcoming, we use an extra lib (libtinygltf) to extract the gltf binary data and then dump the gltf components to files (JSON, bin, etc.). At final, we provide the url of the dumped JSON file to the Qt3DRender::QSceneLoader.

But this diverted process by using temporary files is not efficient but creates unusable geometries (no normal defined, could be fixed). To overcome it, we need to improve the gltf support.

We have 3 suggestions:

  1. In Qt, without the help of the libtinygltf, we could add the possibility to the Qt3DRender::QSceneLoader API to handle the data loading via QByteArray and add the gltf binary format support to GLTFImporter. May be there are some leads into the existing gltf support in QML?

  2. In QGIS, without the help of the libtinygltf, we would have to (re)create a basic gltf to support the binary format and extract meshes, material settings, light settings to rebuild the scene.

  3. In QGIS, WITH the help of the libtinygltf, we could avoid the use of temporary files by inheriting from QNetworkAccessManager (as suggested here) to handle a specific uri (ie. b3dm://...). This uri will be pass to Qt3DRender::QSceneLoader and via our QNetworkAccessManager we could return a valid gltf/JSON file to the scene loader.

Float precision

For reasons of precision tests on matrix computation, the qgsmatrix4x4 and qgsvector4d classes have been added by using double instead of float.

Unloading

In our example, may be due to implementation, some objects persist even when unzooming but the Qt3DCore::QEntity are no more active. We will have to investigate further to resolve this point.

Further Considerations/Improvements

Further Considerations:

Backwards Compatibility

The need of backwards compatibility may not be releavant if, in the short term, QGIS evolves to Qt6.

Votes

(required)

nyalldawson commented 3 years ago

The gltf support via Qt3D only allow to read specific format via url/uri.

QgsModelPoint3DSymbolHandler::addSceneEntities already handles this situation by automatically storing the binary in a temporary file via QgsApplication::instance()->sourceCache()->localFilePath(...). Couldn't that approach be used here too?

nyalldawson commented 3 years ago

Exciting addition, this sounds great! I'd love for the introduced code to be kept as generic as possible to also allow for potential future support for ArcGIS scene layers :+1: .

benoitdm-oslandia commented 3 years ago

@nyalldawson I did not know this API part! Thanks for the tips I'll check this asap!

wonder-sk commented 3 years ago

Exciting stuff - looking forward to that!

Various things that jump to my mind:

3D tiles seem to have quite wide range of functionalities, so it would be good to further specify what will be supported and what not... Using the overview document - https://github.com/CesiumGS/3d-tiles/blob/master/3d-tiles-overview.pdf

The 3D stack to handle the chunk loading is missing documentation and samples to properly understood the usecases and the proper way to implement a new chunk loader.

Please let me know if you have concrete questions - will try to answer them (and hopefully also add to the API documentation).

properly extend the QgsChunkLoader, QgsChunkedEntity and QgsChunkLoaderFactory classes to support dynamic tile loading

Dynamic tile loading is already used for remote EPT datasets for point clouds... what did you have in mind?

extend the QgsAbstract3DRenderer and QgsMapLayer classes to display the tiles

In what way do they need to be extended?

benoitdm-oslandia commented 3 years ago

Thanks @wonder-sk!

what about 2D map rendering? what about APIs to access 3D Tiles data - will there be a way to access the hierarchy programmatically and to access the raw mesh data?

For what I know, the 2D rendering is not planned! But we could try to provide mesh data picking/selection but the main difficulty may be to maintain the selection and the mesh changes due to LOD.

how is it going to integrate with the concept of map layers - will there be a new layer type for this? or using mesh layer? or...?

In the current implementation example, we inherit from the mesh map layer but may be we will need a new one.

is it going to support local datasets or remote datasets (over HTTP(S)) or both?

Both dataset will be available.

how is styling going to work? Are you planning to add new shaders?

In 3D Tiles the mesh styling is in the embedded gltf objects. The style customization is a good idea but must be discussed further.

is identification of objects going to be supported?

What do you mean? If you are talking about selection of object, I would like to do it.

3D tiles seem to have quite wide range of functionalities, so it would be good to further specify what will be supported and what not... Using the overview document - https://github.com/CesiumGS/3d-tiles/blob/master/3d-tiles-overview.pdf

what types of bounding volumes?

the 3 types are already extracted from the JSON but from QGIS point of view, they are all boxes. which refinement strategies? The refinement strategies are (or seems to be) already handled by QgsChunkedEntity::setUsingAdditiveStrategy is the declarative styling going to be supported? This would be really nice but I do not know if we will have enough time to do it.

About the "extension" I was talking, it is only from the class inheritance point of view (french mistranslated) :)

wonder-sk commented 3 years ago

For what I know, the 2D rendering is not planned!

Hmm... IMHO that should be planned as well. All existing data types including point clouds are supported in 2D map rendering...

In the current implementation example, we inherit from the mesh map layer but may be we will need a new one.

This seems like an important implementation decision that should go to QEP - would it be a new map layer type (something like "scene layer" in ESRI (?)), that would be common for 3D Tiles, I3S and possibly other formats (implemented as data providers)? What would be the interface of data providers?

In 3D Tiles the mesh styling is in the embedded gltf objects.

Ah right - good point.

is identification of objects going to be supported?

What do you mean? If you are talking about selection of object, I would like to do it.

I mean support for identify map tool (in 2D and 3D view)...

benoitdm-oslandia commented 3 years ago

Hmm... IMHO that should be planned as well. All existing data types including point clouds are supported in 2D map rendering...

good point. we'll check how to do that.

This seems like an important implementation decision that should go to QEP

Ok I will add it!

I mean support for identify map tool (in 2D and 3D view)...

Ok I will add it!

Jean-Roc commented 3 years ago

good point. we'll check how to do that.

Keeping the faces based on normal orientation ? It would seems less costly than involving gltf's uv map. However, while it is a good feature to have, could it be a task planned for a later stage ?

benoitdm-oslandia commented 3 years ago

Hi,

so long without any updates! Let's try to fix that!

Introducing 3DTiles in QGIS was not as easy as I planned, mainly due to:

One can look at our presentation done for FOSS4G 2021/09. Here are the screencasts dragon, Japanese house and Nasa landscape.

glTF

As previously said, we have to use an external library to extract the data from the glb saved within the b3dm files to dump new files (json+bin) usable by Qt3D qsceneloader. Theses files load properly in blender.

Also If more than one mesh is defined, the Qt gltf importer failed at reading all meshes (only the first one). And as texture material cannot be retrieved/rendered properly with the generated qt scene we use our internal material.

To resolve those points, as we may modify the Qt and Assimp libraries hoping the changes will be backported in Qt 5.15 via compilation flag (like this one), thus available in future QGis builds.

Or we may have to retrieve from Qt some piece of code (eg. glTF/Assimp wrapper) and fix them in QGis (until we move to Qt6). Also it may be needed to add an updated Assimp library (via static linkage) in QGis. After that we should be able to backport the changes to Qt.

File cache

To load the 3Dtile files from remote location, I used, as suggested, the QgsSourceCache but sometimes the remote files cannot be loaded. Thus, I implemented one more file cache manager (as there are many ways in QGis to have a file cache). It should be improved or replaced by a better QGis solution.

Object orientation

By default 3dTiles objects are defined to be put on a globe thus the final object/mesh coordinates must be oriented accordingly. But we are not on a globe in QGis and the final object/mesh coordinates do not match our needs.

dragon_4978_bad_orientation

We need to re-orient the mesh to put it on a flat ground.

dragon_4978_good_orientation

But the meshes may be pre-oriented ie. the mesh coordinates are already set to match the globe location without the need to apply the tile transformation matrix. When the mesh is pre-oriented it is quite difficult to compute the reverse transformation matrix.

Currently when the globe transformation is brought by the tile transformation matrix (ie. the mesh is on flat ground and 0 centered), we are able to compute the reverse transformation matrix.

LOD

I encountered some issues or questions when loading the 3DTiles data within the QGis chunked loader mechanism.

event loop

The Qt3D entity retrieval needs the QSceneLoader object to load a glTF file, but the setSource signal is never handled by the event loop thus the QSceneLoader object never load the file. I have to manually call the QCoreApplication::processEvents() in createEntity!

Qt tree

3Dtiles dataset can be huge and for now I use only one rootEntity for the whole dataset. The more I get deeper in the tileset, the more I enrich the rootEntiy with new Qt3DCore::QEntity (with mesh, material, etc.) and by the way the Qt object tree is getting bigger. This will soon induce a memory problem and I do not know how to change my loader or the QGis chunk loading mechanism to solve it.

Should I use multiple rootEntity, ie. by limiting the tree depth ? How to tell to QGis to switch from one rootEntity to another?

Or Should I chop off the Qt object tree of the furthest Qt3DCore::QEntity branches? How to do that?

Further works

We would like to thank here the Eurométropole de Lille which financed this work, thus opening the way to a direct visualization of the 3DTiles format in QGIS!

We have laid the groundwork for an implementation of 3DTiles and we are very happy about it :).

However, to make its use possible in QGIS, there is still work to be done to industrialize it and optimize its performance.

Here are the main points needing work:

Several actors have shown interest in contributing to these developments and we would also be happy to be able to work on it again!

vpicavet commented 1 year ago

@nyalldawson Great to see you managed a Cesium Grant for this work to be pursued. We would have appreciated to be contacted first to discuss the matter though, as @benoitdm-oslandia already put some significant efforts into this topic.

We are ready to collaborate to make 3D Tiles in QGIS a reality, but will need fair coordination and cooperation.

Jean-Roc commented 1 year ago

Hi, more details have been published on north road's blog but I do not see any mention of the present QEP or others about 3D. I would like to second Vincent's demand on this subject, on a personal point of view this does not encourage me to fund this kind of work again.

nyalldawson commented 1 year ago

@vpicavet @Jean-Roc

Thanks for your interest in our work! We'll be kicking off this project in early July after 3.32 final release (with the intention of finalising this work for QGIS 3.34), and more details will be made available then.

benoitdm-oslandia commented 1 year ago

Not needed anymore!