search

CesiumGS / cesium-native

Apache License 2.0
416 stars 210 forks source link

Support .i3dm tile payloads #458

Open argallegos opened 2 years ago

argallegos commented 2 years ago

Add support for Instanced 3D Models.

Once this is completed, some work will need to be done in Cesium Native integrations (e.g. Cesium for Unreal, Cesium for O3DE) before the feature is fully operational in each engine.

argallegos commented 2 years ago

Pull request addressing this issue: https://github.com/CesiumGS/cesium-native/pull/290

argallegos commented 2 years ago

Cesium for Unreal issue: https://github.com/CesiumGS/cesium-unreal/issues/671

Cesium for O3DE issue: https://github.com/CesiumGS/cesium-o3de/issues/32

lilleyse commented 2 years ago

Useful diagram from https://github.com/CesiumGS/cesium/pull/10302 showing the expected transform order between i3dm and gltf + EXT_mesh_gpu_instancing

image

lilleyse commented 7 months ago

See https://github.com/CesiumGS/3d-tiles-tools/pull/52 for ideas for upgrading i3dm to EXT_mesh_gpu_instancing. It's not trivial because of transformation order differences (see diagram above).

CC @javagl

javagl commented 7 months ago

One of the insights from the linked PR was that it is sometimes plainly (mathematically) not possible to upgrade I3DM to EXT_mesh_gpu_instancing. An I3DM could contain an animated GLB that was non-uniformly scaled via the instancing transform. This can not be represented as a standard glTF animation.

It could be possible to lay out the map of "what is and what isn't possible" more strictly and systematically. But for the main case of I3DMs that contain "standard (non-animated) geometry", the approach from https://github.com/CesiumGS/3d-tiles-tools/pull/52#issuecomment-1679433253 turned out to be relatively easy, and could probably be applied in cesium-native as well, when there is a function to "bake" the transform of glTF nodes into the attached meshes.

lilleyse commented 7 months ago

It could make sense to implement this in three phases:

Sample data:

CC @timoore @kring

timoore commented 7 months ago

One of the insights from the linked PR was that it is sometimes plainly (mathematically) not possible to upgrade I3DM to EXT_mesh_gpu_instancing. An I3DM could contain an animated GLB that was non-uniformly scaled via the instancing transform. This can not be represented as a standard glTF animation.

It seems to me that even relatively simple animations can't be directly supported directly by EXT_mesh_gpu_instancing, for example wind turbines with the blade rotating around its local axis. This may be less useful in practice than as a thought experiment.

It could be possible to lay out the map of "what is and what isn't possible" more strictly and systematically. But for the main case of I3DMs that contain "standard (non-animated) geometry", the approach from CesiumGS/3d-tiles-tools#52 (comment) turned out to be relatively easy, and could probably be applied in cesium-native as well, when there is a function to "bake" the transform of glTF nodes into the attached meshes.

Animations aren't supported by the native clients yet AFAIK, but if it became important one could imagine cesium-native creating "extended instances" from standard glTF. That is, it could identify regular nodes that reference the same mesh and create instances from them, including animated instance data.

javagl commented 7 months ago

This may be less useful in practice than as a thought experiment.

For https://github.com/CesiumGS/3d-tiles-tools/issues/84 I actually considered to create a draft/proof of concept that visualizes actual wind power plants from one of the public databases, and ... then noticed that this may be an actual limitation. Not that having them animated would be "useful", but ... at this point, it's not about "useful", but about "looking cool" 😎

In terms of the phases that @lilleyse mentioned:

  • Add support for EXT_mesh_gpu_instancing
  • Transcode .i3dm to EXT_mesh_gpu_instancing

One concern could be: Iff the implementation is (too) focussed on the EXT_mesh_gpu_instancing extension, then there's the risk that it will "soak up" the limitations of this extension. I think that technically, having "individual(ly) animated instances that use GPU instancing" could be possible (and correct me if I'm wrong).

Ideally, the fact that it is not possible to represent such animations in the extension should not necessarily restrict the implementation. Very roughly speaking: The implementation could be "powerful enough" to represent such animations. When it is fed with EXT_mesh_gpu_instancing information, then this capabilitiy will not be used. If it is fed with I3DM (or some hypothetical EXT_mesh_gpu_instancing_with_animations data...), then it could use that information.

NOTE: This is really just brainstorming. I have no idea whether trying to support such animations may imply a prohibitively large implementation effort, or whether it would just boil down to passing in some optional animation data to an existing function...

andek714 commented 7 months ago

Just a short comment about this: Our users would benefit a lot from i3dm support in Unreal, even without animation support. That would make trees, power line pylons, street lights etc. so much better in a terrain exported from Remoscape. I assume we're not the only ones using static non-animated i3dm either, so implementing it without animation first could be a good idea.