3D scene - 2D image correspondence fails.

[vikusiq]

Hi @mikedh! Thanks for such a great library! You've done a lot of invaluable work with your strong community! I've just wanted to report you an issue I am dealing this weeks playing around with trimesh.

So, what am I doing is:

1. load a basic mesh.
2. save a image of it using save_image utility from trimesh.
3. process that image (find some useful information there, represented by 2D points (x-y coordinates)). Note that I am not altering the size of it or scale. So, the place of each pixel remains intact.
4. And here is where the problems begin: I am supposed to recover 3D coordinates of the correspondent 2D points of the image. I guess I've tried everything which I found relevant:
   • scene.camera_rays() - which is returning a quite wrong 2D-3D correspondences.
   • trimesh.scene.cameras.camera_to_rays(cam) - which is doing exactly the same
   • xy, pixels = ray_pixel_coords(scene.camera) - this, unfortunately, had not the correct correspondences, too. But was really promising.
   • points, index_ray, index_tri = scan.ray.intersects_location(origins, vectors, multiple_hits=False) - and finally, I've tried the ray tracing and found the intersection of the camera rays with mesh - which haven't solve the problem, too. Actually it end with the same coordinates as previous methods.

Could you please take a look at this issue? Maybe the save_image and camera_rays/camera_to_rays/ray_pixel_coords/intersects_location are not supposed to work together and there another way to recover 3D coordinates?! I've also had a feeling that somewhere the image/the scene/or the projected rays are rotated or something very strange happens to them. Or maybe there is kind of x-y inversion?

PS: I've also tried to express mathematically the x and y coordinates from the 3D space (with the fixed z), based on the intrinsic parameters of the camera. Which unfortunately did not help as well - x - was approximately in the right region, but y was projected too far away.

Hope, you can give me some hints! Thanks, Victoria

[vikusiq]

Or maybe the camera which is used in save_image has different parameters than the default camera from the trimesh??!

[M1234Thomas]

@vikusiq Have you found a solution for this? I am facing the same problem.

[benmor20]

I am also experiencing incorrect 2D-3D correspondence with scene.camera_rays(). @vikusiq or @M1234Thomas - have either of you found a solution?