search

mrdoob / three.js

JavaScript 3D Library.
https://threejs.org/
MIT License
102.04k stars 35.33k forks source link

Raycaster ignores displacement maps and their effct on geometries #24477

Closed Yincognyto closed 2 years ago

Yincognyto commented 2 years ago

Describe the bug

I'm not sure if this is a bug or the expected behavior, but according to the Three.js manual, "a displacement map affects the position of the mesh's vertices" and "the displaced vertices can cast shadows, block other objects, and otherwise act as real geometry", so based on that, they should affect the results produced by a Raycaster. However, they do not.

To Reproduce

Steps to reproduce the behavior:

  1. Create a basic Three.js setup
  2. Rotate the camera around an object that is using a displacement map
  3. Use Raycaster to cast a ray towards the object
  4. Notice that the logged distance to the intersect point with the object doesn't change

Code

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <style>
      body {margin: 0;}
    </style>
  </head>
  <body>
    <script type="importmap">
      {
        "imports":
        {
          "three": "http://localhost:8080/three.module.js",
          "ArcballControls": "http://localhost:8080/ArcballControls.js"
        }
      }
    </script>
    <script type="module">
      import * as THREE from "three";
      import {ArcballControls} from "ArcballControls";
      var scene, camera, light, renderer, earth, controls, reqid, paused = false;
      function createcontrols()
      {
        controls = new ArcballControls(camera, renderer.domElement, scene);
        controls.addEventListener("change", function(e) {renderer.render(scene, camera); controls.update();});
      };
      function create()
      {
        renderer = new THREE.WebGLRenderer({antialias: true, alpha: true});
        renderer.setSize(window.innerWidth, window.innerHeight);
        document.body.appendChild(renderer.domElement);
        camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.1, 100);
        camera.position.set(0, 0, 1.75);
        light = new THREE.AmbientLight("rgb(255, 255, 255)");
        light.position.copy(camera.position);
        var geometry = new THREE.SphereGeometry(1, 360, 180);
        var material = new THREE.MeshPhongMaterial();
        material.map = new THREE.TextureLoader().load("http://localhost:8080/Earth.png");
        material.displacementMap = new THREE.TextureLoader().load("http://localhost:8080/Earth - Height.png");
        material.displacementScale = 0.002 * 99;
        earth = new THREE.Mesh(geometry, material);
        scene = new THREE.Scene();
        scene.add(camera);
        scene.add(light);
        scene.add(earth);
        createcontrols();
        document.addEventListener("keyup", function(e)
        {
          switch (e.code)
          {
            case "KeyP": console.log("P"); paused = !paused; if (!paused) {animate();} else {stagnate();}; break;
          };
        }, {passive: false});
      };
      function raycast()
      {
        var cameradirection = new THREE.Vector3();
        camera.getWorldDirection(cameradirection);
        cameradirection.normalize();
        var cameraorigin = new THREE.Vector3();
        camera.getWorldPosition(cameraorigin);
        var raycaster = new THREE.Raycaster(cameraorigin, cameradirection);
        var intersects = raycaster.intersectObjects([earth], false);
        if (intersects.length) {console.log(intersects[0].distance);};
      };
      function animate()
      {
        if (paused) {return;};
        requestAnimationFrame(animate);
        earth.rotation.y += 0.01;
        renderer.render(scene, camera);
        raycast();
      };
      function stagnate()
      {
        if (!paused) {return;};
        cancelAnimationFrame(reqid);
      };
      create();
      animate();
    </script>
  </body>
</html>

Live example

None

Expected behavior

I expect Raycaster to react to the effects that the displacement map "should" have on the geometry of the object (and not only on its material, according to the manual), and log different distances to the intersection point with the object, based on the "height of the terrain" being displaced. I intentionally set a considerable displacement scale in the code above to make it even more obvious. My actual usage scenario is preventing the camera to approach closer than the minimum distance to an object, but according to the altitudes / terrain on the object's surface. Let me know if Raycaster ignoring displacement maps is a bug or it's something expected to happen due to the displacement maps being just as "fake" as the bump maps (i.e. not affecting the real geometry of the object).

Screenshots

None

Platform:

WestLangley commented 2 years ago

displacement maps being just as "fake" as the bump maps (i.e. not affecting the real geometry of the object).

Displacement maps do affect the geometry vertex positions, but on the GPU only. Raycasting occurs on the CPU.

I'm not sure if this is a bug or the expected behavior

It is expected.

In the future, please use the three.js forum if you need help determining if something is a bug.

//

Tip: You are instantiating 60 THREE.Raycaster objects per second. Create one and reuse it, instead.

Yincognyto commented 2 years ago

displacement maps being just as "fake" as the bump maps (i.e. not affecting the real geometry of the object).

Displacement maps do affect the geometry vertex positions, but on the GPU only. Raycasting occurs on the CPU.

I'm not sure if this is a bug or the expected behavior

It is expected.

In the future, please use the three.js forum if you need help determining if something is a bug.

//

Tip: You are instantiating 60 THREE.Raycaster objects per second. Create one and reuse it, instead.

Alright, thanks for letting me know so fast that this is expected - I thought objects have a single geometry, whether it's about the GPU or the CPU, but apparently they do not, however strange that might seem... Now I can proceed further with my code, knowing it has no solution - thanks for your great advice on optimizing the process, much appreciated.

As for seeking help on the Three.js forum, it's not exactly that, but there was a similar question on StackOverflow from way back in Feb. 2016 that still has no answer or comment on it, hence me posting this here. It's also more convenient this way, as I already have accounts on SO and GitHub and don't intent to create a dozen more for every issue I may encounter. I'll consider your advice on that though. Too bad you left SO lately, you were great...

donmccurdy commented 2 years ago

I could see this maybe being a workable feature request for three-mesh-bvh? But I think sampling from the displacement map in realtime on the CPU in three.js' built-in Raycaster is more than we can really support. /cc @gkjohnson

WestLangley commented 2 years ago

@Yincognyto Thank you for the complements. Unfortunately, we do not have sufficient staff here to deal with help requests.

Depending on your use case, you can also consider GPU Picking. You should be able to get help with that at the forum.

Yincognyto commented 2 years ago

@Yincognyto Thank you for the complements. Unfortunately, we do not have sufficient staff here to deal with help requests.

Depending on your use case, you can also consider GPU Picking. You should be able to get help with that at the forum.

Ah, ok, no problem, I completely understand. Smaller crews where quality prevails are generally more dedicated than larger ones where marketing does, and that's something to be proud of.

GPU picking seems an clever idea if I correctly understand its principle, but I'm not sure how suited it is for distance calculations (my usage case) instead of object picking (its intended usage). I'll investigate the matter further though, maybe there is a possibility. I already managed to write 2 shaders for emissive night lights and building a realistic atmosphere so I'm ok with that route, if needed. For now I'll settle on stopping the camera at the greatest altitude on the object as that works flawlessly, but I'll look into alternative ways of properly following the real shape of the terrain irrespective of altitude.

Thanks again for the useful advice - I'll make the best out of it ;)

gkjohnson commented 2 years ago

I could see this maybe being a workable feature request for three-mesh-bvh? But I think sampling from the displacement map in realtime on the CPU in three.js' built-in Raycaster is more than we can really support. /cc @gkjohnson

This is such a niche use case, I think, that I'd prefer not to support it directly in BVH structure. I wouldn't be against a utility to make this kind of raycasting easier to implement but it's not something I would plan to work on myself. Though at some point if you're generating equivalent geometry on the CPU to support raycasting the value of a shader-based displacement map becomes more limited.

But that aside it's a fairly complicated thing to do since exact, full support requires implementing equivalent texture sampling techniques on the CPU and possibly new normal derivation. To implement this I would create a utility class that takes a geometry and data texture displacement map, bias, and scale and then generates a new geometry (or optionally modifies one in place) which can be used for raycasting. The displacement map would likely have to be read from the GPU into a data texture so it could be processed on the CPU. Here's what the utility might look like:

const textureReader = new TextureReader( renderer );
const displacementMap = textureReader.read( originalMap );

const displacedGeometry = new BufferGeometry();
const generator = new DisplacedGeometryGenerator();
generator.generator( originalGeometry, displacementMap, displacementBias, displacementScale, displacedGeometry );

From there you can raycast the new geometry and / or generate or refit a bvh for faster raycasting. But like I said - quite a bit of work.

It's also more convenient this way, as I already have accounts on SO and GitHub and don't intent to create a dozen more for every issue I may encounter.

You're able to log into the forum using your Github account.

Now I can proceed further with my code, knowing it has no solution

There's always a solution - it just depends on how much code you want to write to get there 😁

Yincognyto commented 2 years ago

This is such a niche use case...

Well, I understand what you mean in terms of specific coding purpose, but generally I don't think following terrain when "moving" (aka changing camera position) is a niche case, in practical terms. Simulations need that, games need that, basically everything 3D that you draw on the screen and want to even remotely resemble "reality" (or physics) would make use of that on a basic level. I don't think any user would want to get lost in a hill or mountain instead of climbing it or going around it, or walk through irregularly shaped walls instead of stopping in front of them. :)

But that aside it's a fairly complicated thing to do...

It looks like it, yeah. Nice things have the habbit to be like that.

To implement this I would create a utility class that takes a geometry and data texture displacement map, bias, and scale and then generates a new geometry (or optionally modifies one in place) which can be used for raycasting.

Apart from moving data to an entity available for the CPU in order to be able to do raycasting on it (which I didn't know I had to do) and using your BVH structure (which is at this point an unknown quantity for me), that's more or less the same approach in my actual code (yes, I know I do things 60 times a second, but my process is to first make things work and only then optimizing them to the max):

function regulate()
{
  mincam = 1 + 0.002 * w.scale + camera.near; maxcam = 1 - camera.near + camera.far;
  if (camera && earth && camera.position.distanceTo(earth.position) < mincam)
  {
    camera.translateZ(maxcam - camera.position.z);
    var scaledobject = earth.clone();
    scaledobject.scale.setScalar(mincam);
    scaledobject.updateMatrixWorld(true);
    var cameradirection = new THREE.Vector3();
    camera.getWorldDirection(cameradirection);
    cameradirection.normalize();
    var cameraorigin = new THREE.Vector3();
    camera.getWorldPosition(cameraorigin);
    var raycaster = new THREE.Raycaster(cameraorigin, cameradirection);
    var intersects = raycaster.intersectObjects([scaledobject], false);
    camera.translateZ(intersects.length ? - intersects[0].distance : mincam - maxcam);
  };
  v.look = camera ? camera.position.distanceTo(earth ? earth.position : new THREE.Vector3()) * radius : 42164;
};

In short, when I wrote this, I thought that moving the camera back sufficiently, casting a ray towards a scaled up version of my object, getting the intersect point and moving camera foward to that point would both keep the camera at a set minimum distance from the object and also follow the object's displacement map. While this works, if disregarding the now obvious redundancies, Raycaster is not able to take the (scaled or not) object's displacement map into account, so for now I'm left with following only the maximum overall altitude instead of the altitude at the intersect point (well, more or less, if the ray is not perpendicular on the surface). At least now I understand why, thanks to @WestLangley, and explore alternatives, thanks to everyone else sharing his take on it.

You're able to log into the forum using your Github account.

Good to know, thanks - that would make it easier for sure, for the future.

There's always a solution - it just depends on how much code you want to write to get there 😁

I'm of the exact same opinion: everything is possible, it only depends on how much you want it. I was referring to the "standard" (or "easy") ways of achieving it with my statement above, i.e. directly via Raycaster. Of course, when talking about things in general, there is always a way to get from A to B, even if it means traveling around the Earth in 80 days ... to get to the same place.

LeviPesin commented 2 years ago

Maybe you shouldn't use the displacement map but instead just change the geometry based on the map data?

Yincognyto commented 2 years ago

Maybe you shouldn't use the displacement map but instead just change the geometry based on the map data?

Valid point. I chose displacement map because it was the easy and logical option and I reckoned it would produce the expected results, i.e. have an effect not only on displaying the heights and giving the sense of terrain, but also when it came to the raycaster.

So, how would I change geometry based on the map data, apart from, I suppose, the DataTexture method, as mentioned earlier? Are there other ways, or is that the only (or best) way to do it? More importantly, can this be done in plain Three.js or I'd have to use some other additional library / tool (like @gkjohnson explained)?

LeviPesin commented 2 years ago

If you are using it for terrain - just generate a PlaneGeometry and then use setY on the position attribute in loop taking data from the texture.

Yincognyto commented 2 years ago

If you are using it for terrain - just generate a PlaneGeometry and then use setY on the position attribute in loop taking data from the texture.

Yes, it's for a terrain, but the terrain is not on a plane but projected on a sphere / globe. I'll take your advice into consideration though, thanks. In the interest of not polluting the thread further with things that are normally rated as asking for help, I'll stop it here - I will explore my options later on, based on the helpful suggestions I received, and for which I'm grateful.

As a last thought, maybe there should be a small note in the Three.js manual that would briefly explain why Raycaster won't produce the "logical" results when used on a displacement map. That would avoid further confusion and / or questions on that matter - just saying. ;)

Yincognyto commented 2 years ago

Maybe you shouldn't use the displacement map but instead just change the geometry based on the map data?

For the record and future readers of this (if any), after some other things I had to correct in my code, I followed your advice and the "nothing is impossible" principle I share with @gkjohnson and managed to do what I wanted to do, using a combination of vertex positions and normals that I suspect will work for most geometries (they do for plane and sphere ones from my tests):

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <style>
      body
      {
        margin: 0;
        width: 414px;
        height: 414px;
      }
    </style>
  </head>
  <body>
    <script type="importmap">
      {
        "imports":
        {
          "three": "http://localhost:8080/three.module.js",
          "ArcballControls": "http://localhost:8080/ArcballControls.js"
        }
      }
    </script>
    <script type="module">
      import * as THREE from "three";
      import {ArcballControls} from "ArcballControls";
      var scene, camera, light, renderer, earth, controls, reqid, paused = false, camup;
      function createcontrols()
      {
        controls = new ArcballControls(camera, renderer.domElement, scene);
        controls._up0.copy(camup); controls._upState.copy(camup);
        controls.addEventListener("change", function(e) {renderer.render(scene, camera);});
      };
      function create()
      {
        renderer = new THREE.WebGLRenderer({antialias: true, alpha: true});
        renderer.setSize(window.innerWidth, window.innerHeight);
        document.body.appendChild(renderer.domElement);
        camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.1, 100);
        camera.position.set(0, 0, 1.75);
        camup = new THREE.Vector3(0, 1, 0);
        light = new THREE.AmbientLight("rgb(255, 255, 255)");
        light.position.copy(camera.position);
        var geometry = new THREE.SphereGeometry(1, 360, 180);
        var material = new THREE.MeshPhongMaterial();
        material.map = new THREE.TextureLoader().load("http://localhost:8080/Earth.png");

        /* Displacement maps do not create "terrains" that can be used by a raycaster to give different distances to the surface of the object */
        // material.displacementMap = new THREE.TextureLoader().load("http://localhost:8080/Earth - Height.png");
        // material.displacementScale = 0.002 * 99;

        /* Changing the position of the original geometry vertices in the direction of the normals do yield correct raycaster distances though */
        var displacementtexture = new THREE.TextureLoader().load("http://localhost:8080/Earth - Height.png", function(texture)
        {
          var tercvs = document.createElement("canvas");
          tercvs.width = texture.image.naturalWidth;
          tercvs.height = texture.image.naturalHeight;
          var terctx = tercvs.getContext("2d");
          terctx.drawImage(texture.image, 0, 0);
          for (var j = 0; j < geometry.parameters.heightSegments + 1; j++)
          {
            for (var i = 0; i < geometry.parameters.widthSegments + 1; i++)
            {
              var imgpix = terctx.getImageData(Math.round(i * tercvs.width / (geometry.parameters.widthSegments + 1)), Math.round(j * tercvs.height / (geometry.parameters.heightSegments + 1)), 1, 1).data;
              var posidx = (geometry.parameters.widthSegments + 1) * j + i;
              var posval = new THREE.Vector3(geometry.attributes.position.getX(posidx), geometry.attributes.position.getY(posidx), geometry.attributes.position.getZ(posidx));
              var dirval = new THREE.Vector3(geometry.attributes.normal.getX(posidx), geometry.attributes.normal.getY(posidx), geometry.attributes.normal.getZ(posidx));
              var disval = 0.002 * 99 * imgpix[0] / 255;
              var terpos = new THREE.Vector3().addVectors(posval, dirval.multiplyScalar(disval));
              geometry.attributes.position.setXYZ(posidx, terpos.x, terpos.y, terpos.z);
            };
          };
          geometry.computeVertexNormals();
          geometry.attributes.position.needsUpdate = true;
        });
        /* End of the replaced code */

        earth = new THREE.Mesh(geometry, material);
        scene = new THREE.Scene();
        scene.add(camera);
        scene.add(light);
        scene.add(earth);
        createcontrols();
        document.addEventListener("keyup", function(e)
        {
          switch (e.code)
          {
            case "KeyP": console.log("P"); paused = !paused; if (!paused) {animate();} else {stagnate();}; break;
            case "KeyR": console.log("R"); controls.dispose(); controls = undefined; createcontrols(); renderer.render(scene, camera); controls.update(); break;
          };
        }, {passive: false});
      };
      function raycast()
      {
        var cameradirection = new THREE.Vector3();
        camera.getWorldDirection(cameradirection);
        cameradirection.normalize();
        var cameraorigin = new THREE.Vector3();
        camera.getWorldPosition(cameraorigin);
        var raycaster = new THREE.Raycaster(cameraorigin, cameradirection);
        var intersects = raycaster.intersectObjects([earth], false);
        if (intersects.length) {console.log(intersects[0].distance);};
      };
      function animate()
      {
        if (paused) {return;};
        requestAnimationFrame(animate);
        earth.rotation.y += 0.01;
        renderer.render(scene, camera);
        raycast();
      };
      function stagnate()
      {
        if (!paused) {return;};
        cancelAnimationFrame(reqid);
      };
      create();
      animate();
    </script>
  </body>
</html>

That settles the issue, I reckon - I supose I incorrectly expected this to be more complicated than it was. One difference I noticed compared to the displacementMap geometry on the GPU is that the terrain is smoother (i.e. less faceted) probably due to how Three.js is computing normals. Anyway, apart from sea to land zones where slopes are not as steep as they should be when mountains are around, I believe it looks nicer this way. Anyone is free to express what he thinks of this solution, if he feels like it.

LeviPesin commented 2 years ago

Exactly what I meant 👍

var terpos = new THREE.Vector3().addVectors(posval, dirval.multiplyScalar(disval)); geometry.attributes.position.setXYZ(posidx, terpos.x, terpos.y, terpos.z);

Can't you just write posval.addScaledVector(dirval, disval); geometry.attributes.position.setXYZ(posidx, posval.x, posval.y, posval.z);?

Yincognyto commented 2 years ago

Exactly what I meant 👍

Indeed. Too bad there aren't many examples of similar things, and those that exist either focus on a specific geometry or involve other unrelated additional code. The good part is that the situation pushed me to think and come up with what was needed for the case.

Can't you just write posval.addScaledVector(dirval, disval); geometry.attributes.position.setXYZ(posidx, posval.x, posval.y, posval.z);?

Of course, it can be done that way too, but I liked to have that vertical column of equal signs aligned one under another... :)

Joking aside, Three.js has on occasion different ways of doing the same thing, and I have no idea if choosing one syntax system over another brings any particular benefits (e.g. speed, or efficiency), other than being favored by some developers and unused by others. Usually, in these case I go for the shortest syntax like you suggested, unless code symmetry looks better - so it's a purely aesthetical choice.

LeviPesin commented 2 years ago

It is better because it avoids creating a new object in loop. You can also simplify var posval = new THREE.Vector3(geometry.attributes.position.getX(posidx), geometry.attributes.position.getY(posidx), geometry.attributes.position.getZ(posidx)); var dirval = new THREE.Vector3(geometry.attributes.normal.getX(posidx), geometry.attributes.normal.getY(posidx), geometry.attributes.normal.getZ(posidx)); - you can create two vectors at the start and then just do something like posval.fromBufferAtribute(geometry.attributes.position, posidx); dirval.fromBufferAttribute(geometry.attributes.normal, posidx);.

Yincognyto commented 2 years ago

It is better because it avoids creating a new object in loop [...] you can create two vectors at the start and then just do something like posval.fromBufferAtribute(geometry.attributes.position, posidx); dirval.fromBufferAttribute(geometry.attributes.normal, posidx);.

You're right - .fromBufferAttribute() does indeed simplify things, I will use that instead, thanks! Avoiding creating reusable entities is a good advice as well, unfortunately it's something that I personally only think of once the job of figuring out how to do something (i.e. the "hard" part) is done, since the former almost always needs the latter in order to get something out of it.