search

Geodan / i3dm.export

Export 3D Instanced Tiles from PostGIS table
MIT License
42 stars 22 forks source link

readme

i3dm.export

Console tool for exporting Instanced 3D Models (i3dm's or glb's with EXT_mesh_gpu_instancing), i3dm composites (cmpt), subtree files and tileset.json from PostGIS table.

The input table contains instance information like location (epsg:4326), binary glTF model (glb), scale, rotation and instance attributes.

The 3D tiles created by this tool are tested in Cesium JS.

Sample of trees in Cesium viewer using instanced 3D Tiles in Lyon:

image

For instanced 3D Model (i3dm) specs see https://github.com/CesiumGS/3d-tiles/tree/master/specification/TileFormats/Instanced3DModel

For composite tile (cmpt) specs see https://github.com/CesiumGS/3d-tiles/blob/master/specification/TileFormats/Composite/README.md

NuGet Status

Installation

Options:

1] Use pre-built binaries from release

2] Use .NET Tool

Prerequisite: .NET 8.0 SDK is installed https://dotnet.microsoft.com/download/dotnet/8.0

$ dotnet tool install -g i3dm.export

Or update

$ dotnet tool update -g i3dm.export

3] Use Docker

$  docker run geodan/i3dm.export

Live demo 3D instanced tiles

Trees sample with implicit tiling:

https://geodan.github.io/i3dm.export/samples/traffic_lights/cesium/

Input database table

Input database table contains following columns:

. geom - geometry with Point or PointZ (for example epsg:4326) for i3dm positions;

. scale - double with instance scale (all directions);

. rotation - double with horizontal rotation angle (0 - 360 degrees);

. tags - json with instance attribute information;

. model - byte[] or string column with glb or name of binary glTF model per instance. Should be valid path on tool runtime or valid uri on display in client;

. (optional) scale_non_uniform - double precision[3] - for scale per instance in 3 directions.

See testdata/create_testdata.sql for script creating sample table.

Parameters

Tool parameters:

-c: (required) connection string to PostGIS database

-t: (required) table with instance positions

-g: (optional - Default: 1000) Geometric error

-o: (optional - Default: ./tiles) Output directory, will be created if not exists

-q: (optional - Default: "") Query to add to the where clauses (for example: -q "id<5460019"). Be sure to check indexes when using this option.

--use_scale_non_uniform: (optional - default false) Use column scale_non_uniform for scaling instances

--max_features_per_tile (optional - default 1000). Maximum number of features/instances of tile

--geometrycolumn: (optional - default: geom) Geometry column name

--use_gpu_instancing (optional - default false) Use GPU instancing (only for Cesium)

--boundingvolume_heights (option - default: 0,10) - Tile boundingVolume heights (min, max) in meters. The heights will be added to the z_min and z_max values of the input geometries.

--tileset_version (optional - default "") - Tileset version

--use_i3dm (optional - default false) Use I3dm format  - only first model will be processed (false creates Cmpt - Only when creating I3dm's)

--use_external_model: (optional - default false) Use external model instead of embedded model (Only when creating I3dm's)

--use_clustering: (optional - default false) If tile contains more than max_features_per_tile instances, its number of instances will be reduced to max_features_per_tile by clustering

Docker

See https://hub.docker.com/r/geodan/i3dm.export

To run in Docker mount drives:

$ docker run -it -v $(pwd)/output:/app/output -v $(pwd)/glb:/glb geodan/i3dm.export -c "database_connection_string" -t table

Sample running

$ i3dm.export -c "Host=localhost;Username=postgres;Password=postgres;Database=test;Port=5432" -t public.trees

Getting started

For getting started with i3dm.export tool see getting started.

Benchmarking

1] Benchmark on Daily Digital Obstacles files

Source: https://www.faa.gov/air_traffic/flight_info/aeronav/digital_products/dailydof/

588094 instances worldwide, Content tiles generated: 2345, Subtree files generated: 372

Time generating I3dm --use_gpu_instancing false: 0h 0m 43s 799ms

Time generating GLB --use_gpu_instancing true: 0h 0m 47s 879ms

Model

By default, the instance model will be stored in the i3dm payload. In the i3dm header the value 'gltfFormat' is set to 1. In this case, the model should be a valid file path to the binary glTF. Only the i3m files should be copied to a production server.

When parameter 'use_external_model' is set to true, only the model name will be stored in the i3dm payload. In the i3dm header the value 'gltfFormat' is set to 0. In this case, the model should be a valid absolute or relative url to the binary glTF. The client is responsible for retrieving the binary glTF's. Both the i3dm's and binary glTF's should be copied to a production server. Option 'use_external_model' is only available when '--use_gpu_instancing' is false.

Composites

Starting release 2.0, for every tile there will be a composiste tile (cmpt) - even if there is only 1 model available in the tile.
Specs see https://docs.opengeospatial.org/cs/18-053r2/18-053r2.html#249 . The composite tile contains a collection of instanced 3d tiles (i3dm), for each model there is 1 i3dm. When option --use_i3dm is set to true, only I3dm's are created. When there are multiple models in a tile only the first one is used.

Implicit tiling

Starting release 2.0, tiles are generated according to the 3D Tiles 1.1 Implicit Tiling technique. Tiles are generated in a quadtree, maximum number of features/instances is defined by parameter 'implicit_tiling_max_features'.

Content tiles will be generated in output folder 'content', subtree files will be created in folder 'subtrees'. In the root output folder file 'tileset.json' will be created.

Instance batch info

To add batch info to instanced tiles fill the 'tags' type json column in the input table.

For example:

[{"customer" : "John Doe"}, {"id" : 5454577}]

Note:

. all instance records per tile should have the same properties (in the above case 'customer' and 'id'). The list of properties is determined from the first instance in the tile;

. only key - value is supported, not more complex structures.

Sample to update the json column in PostgreSQL:

postgres=# update  i3dm.my_table set tags = json_build_array(json_build_object('customer','John Doe'), json_build_object('id',id));

The batch table information in the i3dm tile is stored as follows (for 2 instances):

{"customer":["John Doe","John Doe2"], "id": [5454577, 5454579]}

Bounding volume

The root bounding volume in Tileset.json is calculated from the input table in 3 dimensions (xmin, ymin, xmax, ymax, zmin, zmax).

Note: Positions outside xmin, ymin, xmax, ymax -180, -90, 180, 90 in wgs84 are not supported.

Scale non uniform

When the instance model should be scaled in three directions (x, y, z) use the --use_scale_non_uniform option (default false)

When using this option, the column 'scale_non_uniform' will be used for scaling instances.

Column 'scale_non_uniform' must be of type 'double precision[3]'.

Sample queries to create/fill this column:

Create column:

postgres=# ALTER TABLE traffic_signs_instances
postgres=# ADD COLUMN scale_non_uniform double precision[3]

Fill column:

postgres=# update traffic_signs_instances set scale_non_uniform = '{10.0, 20.0, 30.0}'

Spatial index

When using large tables create a spatial index on the geometry column:

psql> CREATE INDEX ON m_table USING gist(geom)

GPU Instancing

In 3D Tiles 1.1, GPU instancing is supported. This means that the same model can be used for multiple instances within a glTF (using EXT_mesh_gpu_instancing - https://github.com/KhronosGroup/glTF/blob/main/extensions/2.0/Vendor/EXT_mesh_gpu_instancing/README.md). Files like I3dm/cmpt are no longer created.

Attribute information can be added to the instances using glTF 2.0 extensions

There is an experimental option to create 3D Tiles 1.1 using GPU instancing: --use_gpu_instancing (default false).

This option is currently in development.

Live demo trees in Grenoble with GPU instancing and metadata:

https://bertt.github.io/cesium_3dtiles_samples/samples/1.1/grenoble_trees/

image

The following features should work:

To use this option, the input table should contain columns 'roll', 'pitch' and 'yaw' (column 'rotation' is not used).

Sql script to create the columns:

alter table instances add yaw decimal default 0
alter table instances add pitch decimal default 0
alter table instances add roll decimal default 0
alter table instances drop column rotation

The columns should be filled with radian angles (0 - 2PI).

Known limits:

Warning: When the input glTF model has transformations, the model will be transformed twice: once in the glTF and once for the instance translations. In some cases it's better to remove the transformations from the input model. For example tool 'gltf-tansform' - function clearNodeTransform (https://gltf-transform.dev/modules/functions/functions/clearNodeTransform) can be used to clear local transformations.

Known issues GPU Instancing:

Clustering

There is an experimental option to create 3D Tiles using clustering: --use_clustering (default false).

When this option is off, dense tiles with number of instances exceeding max_features_per_tile aren't rendered. With this option such tiles are rendered with number of instances that is exactly equal to max_features_per_tile. Number of instances is reduced in the following way:

Performance benchmark

number of instances: 2500
max_features_per_tile: 100

tileset generation time:

Developing

Run from source code:

$ git clone https://github.com/Geodan/i3dm.export.git
$ cd i3dm.export/src
$ dotnet run -- -c "Host=myserver;Username=postgres;Password=postgres;Database=test;Port=5432" -t public.trees

To develop in Visual Studio Code, open .vscode/launch.json and adjust the 'args' parameter to your environment

"args": ["-c" ,"Host=myserver;Username=postgres;Database=test;Port=5432", "-t", "my_table"],

Press F5 to start debugging.

History

2024-10-31: release 2.10.0: add multiple mesh support in input model + gpu instancing tags null checking + vertical precision

2024-10-15: release 2.9.0: add tileset version option

2024-08-01: release 2.8.3: fix release

2024-08-01: release 2.8.2: fix for I3dm using rtc_center for high precision positions

2024-07-22: release 2.8.1: fix release

2024-07-22: release 2.8.0: to .NET 8.0

2024-06-20: release 2.7.4: fix for composite tiles when using GPU instancing

2024-06-13: release 2.7.3: add support for other input source EPSG codes than 4326

2024-06-10: release 2.7.2: fix gpu instancing yaw, pitch, roll, remove mapbox code (parameter -f --format)

2024-06-08: release 2.7.1: fix for disappearing instances

2023-03-28: release 2.7.0: add support for EXT_structural_metadata

2023-11-08: release 2.6.0: Add support for GPU instancing (experimental), removed option -r RTC_CENTER

2023-10-18: release 2.5.0: Improved root bounding volume calculation, improved batch table handling

2023-09-27: release 2.4.5: Get boundingvolume z from input table + option boundingvolume_heights removed

2023-09-27: release 2.4.4 change default -r (relative positions) from false to true

2023-06-20: release 2.4.3 fix for generating 1 tile

2023-02-21: release 2.4.1 fix version number

2023-02-21: release 2.4 split root subtree file in multiple subtree files

2023-01-30: release 2.3.2 improve spatial index performance

2022-10-05: release 2.3.1 bug fix for showing all instances per tile

2022-10-05: release 2.3 add support for byte[] type column (in addition to string) for glb's from database

2022-08-31: release 2.2 renamed parameter 'implicit_tiling_max_features' to 'max_features_per_tile', fix skewed bounding volumes

2022-08-09: release 2.1 use 1 geometric error as input parameter

2022-08-08: release 2.0 adding 3D Tiles 1.1 Implicit Tiling.

Breaking change:

Parameters removed: -e and -s (extent tile and super extent tile)

Parameters added: implicit_tiling_max_features (default 1000)

2021-10-04: release 1.9 adding Cesium support

2020-11-12: release 1.8 add external tileset support

2020-11-12: release 1.72. to .NET 5.0

2020-11-05: release 1.7.1 with bug fix wrong instances per i3dm when multiple models used.

2020-11-05: release 1.7 add support for composite tiles (cmpt). Breaking change: parameter -m --model is removed. Model can now be defined per instance in the input table.

2020-10-28: release 1.6 add scale_non_uniform support

2020-10-21: release 1.5 add query, use_external_model parameters

2020-10-20: release 1.4 add batch info support

2020-10-20: release 1.3 adding instance rotation + rtc_Center support

2020-10-19: release 1.2 with instance scale support

2020-10-19: add support for uri references to external glTF's.

2020-10-15: Initial coding