Provide more general spatial query functionality to get tiles localized to a region

[antocorr]

This is a common problem for every Tile Renderer I tried. I'm building a very basic no-code game/experience editor and wanted to implement some position based games where I can input latitude and longitude and I get the highest resolution tiles from Google Photorealistic tiles service. On Unity and Unreal Google released Geospatial Creator but I'm using Three.js so I would like this 3D tiles renderer.

Unfortunately there are a lot of automations going on, requesting tiles seems to be related to camera pan/zoom, raycasting and so on.

It would be great to have some sort of API to request and position, I don't know, 4 siblings tiles close to location x,y,zoom without having to continuously requesting tiles as the camera moves.

I've also considered traversing the tileset but I don't know if I'm missing something from the spec, It's not working as expected.

This example uses real data from Google root node but if you check the console the first check returns true and the other four return false (at least one should be true).

I tried sphere.containsPoint and Box3.containsPoint

https://codesandbox.io/s/frosty-glitter-ghkcl2?file=/src/index.ts:86-768

[gkjohnson]

The issue is that you're expecting the tiles renderers to provide generic spatial query capabilities but they're just designed with rendering in mind - the spec primarily refers to the format with rendering as the primary use case, as well.

Unfortunately there are a lot of automations going on, requesting tiles seems to be related to camera pan/zoom, raycasting and so on.

The renderer loads the tiles up to the set screen space error requirement based on the camera position and screen resolution. These values are set with the setCamera and setResolutionFromRenderer function. Raycasting is not used to determine which tiles to load at all. The child tiles are only loaded once the necessary parent tiles and necessary sub tile sets have been loaded. These algorithms are required to load the tile sets performantly.

I've also considered traversing the tileset but I don't know if I'm missing something from the spec, It's not working as expected.

It's not clear to me how the boxes in your demo are supposed to be oriented but at the least it does not look like the box bounding volumes are interpreted incorrectly. The spec specifies how the box bounding volume vectors describe an oriented bounding box - not an AABB.

---

If you just want to query the height of the highest resolution tile at a specific point similar to a raycast you can do the following:

• Place an orthographic camera with a narrow view well above the lat / lon you want and face it in the direction you want to query a ray height. Ensure the frustum is long enough to encapsulate the tiles that may load.
• Set the camera on the TilesRenderer using the above-referenced functions.
• Set the errorTarget to 0 so the highest LoD tiles are loaded and set errorThreshold to 0 so no parent tiles are loaded.
• Call "update" and wait until all tiles are loaded (likely need to continue calling update until the tiles are loaded and displayed)
• Run a raycast from the camera position to sample the surface point.

I won't be able to work on this but if this is something you want to help improve the ergonomics of I'm happy to talk through API improvements and help PRs along. It would be a unique feature that sets it apart from other tiles renderers, I think. These types of spatial queries will always have to asynchronous, though.

[antocorr]

https://github.com/NASA-AMMOS/3DTilesRendererJS/assets/1094862/e318574a-1384-48a6-ab3b-e92b8fe5ca07

Unfortunately, I still believe that the most effective approach for me to achieve the desired outcome is to obtain the necessary tiles without relying on camera-related automations. When I attempt to manually position the game elements to the correct location in order to acquire the accurate tiles, numerous problems with precision arise, as you can observe.

[gkjohnson]

Using a shape instead of a camera frustum will not magically solve precision issues - it's worth understanding the problem before blinding trying a different approach.

That aside I don't understand what I'm looking at in your video. That kind of artifact can be caused by a number of different things especially with skinned geometry.

[antocorr]

I'm using a normal approach using camera frustum.

export default function initGoogle3DTiles(rainRenderer, pos) {
    if(tiles){
        console.log(tiles.group);
        return;
    }
    renderer = rainRenderer.renderer;
    camera = rainRenderer.camera;
    scene = rainRenderer.scene;
    scene.position.copy(pos);
    tiles = new GoogleTilesRenderer(params.apiKey);
    tiles.group.rotation.x = -Math.PI / 2;
    tiles.errorTarget = 0;
    tiles.errorThreshold = 0;

    let cameraWP = new THREE.Vector3();

    camera.getWorldPosition(cameraWP);

    console.log("camera wp", cameraWP);

    // Note the DRACO compression files need to be supplied via an explicit source.
    // We use unpkg here but in practice should be provided by the application.
    const dracoLoader = new DRACOLoader();
    dracoLoader.setDecoderPath("https://unpkg.com/three@0.153.0/examples/jsm/libs/draco/gltf/");

    const loader = new GLTFLoader(tiles.manager);
    loader.setDRACOLoader(dracoLoader);   
    tiles.manager.addHandler(/\.gltf$/, loader);
    scene.add(tiles.group);

    tiles.setResolutionFromRenderer(camera, renderer);

    tiles.setCamera(camera);
    renderLoop();
}
function renderLoop() {
    requestAnimationFrame(renderLoop);
    tiles.setResolutionFromRenderer(camera, renderer);
    tiles.setCamera(camera);
    // The camera matrix is expected to be up to date
    // before calling tilesRenderer.update
    camera.updateMatrixWorld();
    tiles.update();
    //renderer.render(scene, camera);
}

The problem lies here in my case scene.position.copy(pos);

But to solve the precision issue I could in theory get the tiles and move it to 0,0,0 later.

The ideal api in my case (I thereby understand that this library may not be the right fit for my project) would be something like this

//pseudocode getTileGroup(latitude : number, longitude : number, zoom: number | null, quantity : number | 1);

It would be a valid alternative to what happens using Geospatial creator in Unity.

[gkjohnson]

It's not clear what pos is and where it comes from. And you still haven't explained what's happening or what the apparent issue is in the above video.

I'm happy to point you in the right direction for some of these issues but I need you to make an effort to clearly explain what the issue is and look into the problem to understand what's happening yourself. Otherwise I'm left guessing which is not something I'm interested in spending my time on.

It would be a valid alternative to what happens using Geospatial creator in Unity. The ideal api in my case (I thereby understand that this library may not be the right fit for my project) would be something like this

I'm not sure why you're so confident in there being a problem with this library. As I mentioned above the jittering is a common issue with large floating values particularly with animated vertices. It's your responsibility to make sure the transforms and distances you're dealing with are at a reasonable scale for the calculations being done. This is completely independent of this project - so far I haven't seen any reason to believe this repo has an issue.

If you don't understand why this is happening for skinned meshes you can read this post here or ask another question at the forum.

[antocorr]

I think there is a misunderstanding at the moment. I don't believe there is a problem with the library; in fact, I find it fantastic. I apologize if I may have used the wrong words (I myself used the word "problem"). In this case, I jumped to conclusions without explaining my use case.

As you know, I am building a small multiplayer game editor. In this editor, you can import your own scenarios, models, etc. Among the various options, there is the ability to set a latitude and longitude and play both from home using currently generated meshes from osmbuildings and directly in the actual location using WebXR (all of this in multiplayer and without writing code).

As you can see in the example with osmbuildings, the meshes are pre-calculated and saved in a cache because the playable area cannot extend beyond what is pre-calculated.

Now I wanted to eliminate the implementation with osmbuildings in favor of Google's Photorealistic 3D Tiles.

In the current implementation of this library (which has many automations under the hood), I cannot simply request, for example, 4 tiles with a given resolution near the specified coordinates. I have to necessarily use setCamera to make a query and receive the correct tiles in the right position.

So, I thought about moving the scene in a way that the WorldPosition of the camera would correspond to the necessary query. However, as I anticipated, there were precision issues that I believe can be resolved by resetting the scene to the center of the world after receiving the tiles.

I indicated what the ideal APIs would be for my use case, but nothing more. I don't expect anything, and I'm not sure if they would benefit other users of the library. I'm just experimenting and opened an issue. That's it. Unfortunately, I didn't explain myself well in my eagerness to learn more and understand how I could proceed, and I apologize for that.

So pos is the result of this

        const { lat, lon } = coords;
        const wpos = new THREE.Vector3(0, 0, 1);
        WGS84_ELLIPSOID.getCartographicToPosition(lat * THREE.MathUtils.DEG2RAD, lon * THREE.MathUtils.DEG2RAD, 0, wpos);
        initGoogle3DTiles(this, wpos);

https://github.com/NASA-AMMOS/3DTilesRendererJS/assets/1094862/ded9ab71-e87f-4586-aa54-7111d3a778a2

[gkjohnson]

Understood - let me clarify a few things.

which has many automations under the hood

These algorithms are necessary to find the tiles regardless of query type. No matter what kind of spatial query is implemented (geo coordinate, shape-based, camera-based) - it's needed to be able to effectively traverse the tileset and fetch intermediate tileset jsons. It is not the case that all this logic is unnecessary for your usecase.

In the current implementation of this library (which has many automations under the hood), I cannot simply request, for example, 4 tiles with a given resolution near the specified coordinates.

It's not as convenient but I have described how to do this with an orthographic camera in https://github.com/NASA-AMMOS/3DTilesRendererJS/issues/378#issuecomment-1598448420. What I proposed effectively amounts to a query using a bounding box and will only load the highest resolution tiles within. Once the tiles have loaded you have access to all the necessary meshes via the visible tiles. It might be nice to add a callback that denotes when new loads have started and when all loads have finished to make this easier to respond to.

So, I thought about moving the scene in a way that the WorldPosition of the camera would correspond to the necessary query. However, as I anticipated, there were precision issues that I believe can be resolved by resetting the scene to the center of the world after receiving the tiles.

There's no need to move the scene after the tiles have loaded. You can move the tiles group ahead of time so the desired lat / lon is centered at 0, 0, 0 based on the known on the radius of the ellipsoid and further adjust based on loaded tiles if needed. I've just updated the docs to be more clear but the setLatLonYUp() function is a convenience function that does just that and is used in the GoogleMapsAerial example. And regardless of query type in order to position the loaded tiles relative to each other correctly the hierarchical transforms need to be applied and respected which means they will be placed at the surface of the earth. I haven't used the API but Unity likely centers these tiles for you but imo opinion it's the applications responsibility to place the tiles in the appropriate spot to avoid any application precision issues that may be incurred.

As I've mentioned I'm not against updating the API to support more convenient methods for querying tiles in this way that build upon the existing camera-based model but I won't be working on it myself. If you're interested in contributing some code and PRs to improve this I can hash out what I think the steps are to get there.

[gkjohnson]

See related tweet: https://twitter.com/arthurschiller_/status/1713130477932994737