Spatial filtering : clipping/cropping tiles or mesh data via a given primitive (AABB, OBB, projected polygon, clipping planes collection etc)

[jo-chemla]

Following a recommendation to open a dedicated thread https://github.com/NASA-AMMOS/3DTilesRendererJS/issues/401#issuecomment-1895465946_ , this issue is meant to discuss what would be a good implementation/considerations to clip or crop a 3d-tiles tileset based on a given clipping/cropping entity.

For context, we are using this implementation of the 3d-tiles threejs renderer + other building blocks to sync multiple typologies of 3D-data: OGC-3D-tiles (hence also google 3d-cities), Potree tiled pointclouds (photogrammetric 3d-scans), gaussian splats, oriented images and orthos, etc. All these datasets live in a single threejs scene and are placed in a unified geo-referenced coordinate system. Example here - you might have to go to the Layers panel, toggle visibility for the basemap layer and select Google 3D Tiles. Note this is based on the potree+cesium implementation, syncing a cesium viewport in the background with a transparent potree/threejs viewer, not the NASA-AMMOS implementation which is being worked on.

Since there is strong overlap between our 3d-scan (for example, a monument) and the google 3d-tiles (the basemap/background of the whole area or city), we would like to filter out the 3d-tiles tileset, clipping or cropping the inside (or the outside). This clipping could happen

• at the tile level: do not parse the tile and subsequent children if its bounds lie outside the clip area
• or at the geometry level, like the below potree or cesium implementation, discard vertices that lie outside the clip area

Existing reference implementations:

• Potree Clipping Volumes are (one-or-many) oriented boxes, with the ability to use these for filtering inside or outside (clipping task) the intersection or union of the volumes (clipping method) - see the Clipping panel in the Tools tab, plus one box has visibility state to off.
• Potree also can clip via clipping polygons, which are associated to a view projection and cull points in/outside the polygons projected along the view direction. Both these impl. compare the local coords of the vertices to the clipping box/clipping polygon expressed in the local coord system, at the vertex shader level.
• Cesium clippingPlanes collections make it easy to define any arbitrary shape. The globe vs 3d-tileset implementation differ in that one accepts convex or concave and the other does not - see this issue

Hence the question: what would be the best way to do clipping/cropping with 3DTilesRendererJS? The original thread would involve editing the calculateError(tile) logic to check if the tile bbox don't resides within the described clipping entity, then set its error to a large value.

[jo-chemla]

[Edited the original post to be more precise describing how clipping/cropping in/out a 3d-tileset with oriented bounding boxes, clipping planes, polygons etc could work]

[gkjohnson]

So to clarify - you basically want to clip polygon shaped holes (or inverted) into tiles so that other data sets can fit into those holes, correct? The first step is just clipping the geometry based on the desired shapes. Filtering out downloaded tiles based on those clip shapes is just an optimization.

For clipping, three.js' clippingPlanes feature should support this for convex shapes. You can modify the global clip planes on the Renderer or on individual materials in the onLoadModel to apply them more specifically. Have you tried this? Concave shapes will need something more complicated, though, if you need that.

Cesium clippingPlanes collections make it easy to define any arbitrary shape. The globe vs 3d-tileset implementation differ in that one accepts convex or concave and the other does not - see https://github.com/CesiumGS/cesium/issues/8751

Looking briefly at the Cesium code I don't see where concave polygons are supported. Do you have an example? I don't see the ability to make clip shapes in this example here.

[jo-chemla]

Hi Garett, Yes indeed, filtering the geometry is good enough - just thought discarding tiles based on the fact that their bounds were yes or no within the clipping entity would be easier, but it is indeed an optimization.

three.js clippingPlanes was indeed the way to go, thanks for that pointer!

• Note that when clippingPlanes are defined on the renderer itself it is only possible to clip 3d-tiles that are inside a convex bounding volume (inside a box rather than outside)

  
  clippingPlanes_insideRenderer = [
  new Plane(new Vector3(1, 0, 0), -4000000), 
  new Plane(new Vector3(0, 1, 0), -4000000), 
  new Plane(new Vector3(0, 0, 1), 1000000), 
  new Plane(new Vector3(-1, 0, 0), 4500000), 
  new Plane(new Vector3(0, -1, 0), 5000000), 
  new Plane(new Vector3(0, 0, -1), 1000000), 
  ];

renderer.localClippingEnabled = true; renderer.clippingPlanes = clippingPlanes;

![image](https://github.com/NASA-AMMOS/3DTilesRendererJS/assets/16822841/411becc3-834d-4730-9223-ab8b2e6f7020)

 - In order to be able to clip either inside or outside of the clipping volume, it was necessary to do so at the material level, setting the attribute `clipIntersection = true` to do the intersection of clipping Planes positive half-spaces rather than their union: 
```js
clippingPlanes_outsideMaterial = [
  new Plane(new Vector3(-1, 0, 0), 4000000),  
  new Plane(new Vector3(0, -1, 0), 4000000),
  new Plane(new Vector3(0, 0, -1), -1000000), 
  new Plane(new Vector3(1, 0, 0), -4500000), 
  new Plane(new Vector3(0, 1, 0), -5000000), 
  new Plane(new Vector3(0, 0, 1), -1000000), 
];
tilesRenderer.onLoadModel = function (scene) {
  scene.traverse((c) => {
    if (c.isMesh) {
      c.material.clippingPlanes = clippingPlanes;
      c.material.clipIntersection = true;
    }
  });
};

Filtering by convex clipping volumes defined by clipping planes on the material is therefore the way to go for us, thanks! Finally, regarding the last points,

• the two Cesium clipping planes sandcastle examples are here (doc) for 3d-tileset clipping and here for terrain clipping.
• Regarding concave clipping with cesium, I never experienced with it myself, and only saw it referenced in the mentioned issue.
• The Polygon clipping example with a concave polygon I had in mind is implemented in Potree itself (see below screenshot for the location of the tool on potree ui, not accessible through the GUI on our app). I could not have it work here, but this is where it is defined in the vertex shader code - boolean test that should work for concave polygons if I remember correctly.
• PS just saw this example of clipping with your three-mesh-bvh library, impressive visualizations by the way! https://discourse.threejs.org/t/three-mesh-bvh-a-plugin-for-fast-geometry-raycasting-and-spatial-queries/26394/18

[gkjohnson]

The Polygon clipping example with a concave polygon I had in mind is implemented in Potree itself (see below screenshot for the location of the tool on potree ui, not accessible through the GUI on our app). I could not have it work here, but this is where it is defined in the vertex shader code - boolean test that should work for concave polygons if I remember correctly.

It looks like this works with basic concave polygons using a point containment by checking the number of edges on one side of the point. This is something that can be implemented in a custom three.js shader.

PS just saw this example of clipping with your three-mesh-bvh library, impressive visualizations by the way!

Thanks - that example is modifying the geometry itself which can have some pros and cons compared to shader-based clipping but generally it will be more complicated, i think.

just thought discarding tiles based on the fact that their bounds were yes or no within the clipping entity would be easier, but it is indeed an optimization.

If this is an optimization you need and would like to add to the project I'm happy to talk through how that could be done. It would boil down to adding the ability to use custom volumes for triggering / filtering out tiles to download instead of just using the camera frustums. Especially if trying to render tiles only within a volume, as you've shown here, it could be a significant improvement in the number of tiles that need to be downloaded.

[gkjohnson]

We can reopen if it looks like something else needs to change to support this or you want to discuss other query mechanisms.

[jo-chemla]

Thanks again for the help, all is good as is!

• Indeed we can at some point implement the point-in-polygon test on the shader-side to allow for concave shapes/polygons rather than only clipping planes and boxes - although boxes is already enough atm for us.
• And finally regarding that optimization, most of the time we will do the opposite, clipping the inside of the clipping volume. This will be interesting to do at some point, I'll get back if we try to tackle that!

[jo-chemla]

For reference, Paul from our team just pushed a PR to Potree showing integration of a Potree georef'ed dataset (all datasets stored on app.iconem.com are georeferenced, stored in a projected CRS) with your 3d-tiles renderer: https://github.com/potree/potree/pull/1408

Will probably craft a demo tomorrow for socials. Again, thanks a lot for your help!

Live example here : https://3d.iconem.com/_dev/3dtilesrenderer/examples/3dtilesrenderer.html Pointcloud URL: https://3d.iconem.com/france/Paris_Sacre-Coeur_lowquality_189M_202006/pointclouds/index/cloud.js Height Offset: 156 Google 3d-tiles api key required

[jo-chemla]

For reference, here is the above example with

• Google 3D-Tiles cities loaded within NASA-AAMOS 3DTilesRenderer as a basemap
• and a georef'ed Potree stored in EPSG:32631, a 3d-scan of the Sacré-Coeur, Paris by Iconem

https://github.com/potree/potree/assets/16822841/d4aaaad7-0995-464d-b6ba-cbe0f6009fbd

[gkjohnson]

Looks great! Seems like the clipping plane worked well.

[jo-chemla]

Yes that's really gorgeous, plus replicable for any georef'ed scene - and depth integration is a real bonus compared to simply compositing a cesium viewer renderer behind the threejs/potree scene as in the potree-cesium example.

We also had some fun computing a rigid transform from one Coordinate System to the other since both datasets are expressed in different CRS (our is country-wide CRS or UTM etc, while Google 3D Cities is ECEF EPSG:4979), by using proj4 to convert from CRS1 to CRS2 coords of a local origin and three axes vectors, using the resulting coords to compute a basis rotation of the axes. Seems that it does work most of the case, but having some trouble I'm investigating.