search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.7k stars 1.3k forks source link

Camera position optimisation 'diverging' issue. #1717

Closed arv8315 closed 8 months ago

arv8315 commented 8 months ago

I have taken the camera position optimisation with differentiable rendering notebook and made efforts to adapt it as follows given my circumstances. My circumstances are that I have a 3D model (.obj) and a corresponding 2D target image of that 3D model (rather that having to make one. I do not want to do that and use my a-priori 2D images). I want to determine the camera pose for that 2D image which I do not have.

  1. Here is my 3D object's 2D image raster: image

  2. Here is the same object resized with background removed. (Lots of background removal tools out there). I used opencv to load the image, used numpy.pad to resize the image and then removed the background. Here is the outcome.

image

using cv2 results in the img being loaded being a numpy.ndarray by default. I simply had to call expand_dims to end up with a shape of (1, 256, 256, 4) which is the same as the reference image. I also introduced an alpha channel to make it 4 channels rather than 3 like the reference image had by default. Now, in place of the original reference image, I substituted in my numpy array. Then, when I ran the code that compares the starting point with reference image silhouette, obtained the following: image

Note that I had to change the Model code to remove all zeros (blacks) instead of whites (1)

However, when I run the optimisation, it ends up shrinking and not iteratively improving the camera pose until it matches the one that would have resulted in the target silhouette. So far not sure why this is happening. Any suggestions?

image

arv8315 commented 8 months ago

Is the artefact around the edge of the silhouette confusing the solver? I will attempt to remove that artefact and try again.

bottler commented 8 months ago

I'm afraid this is a modelling question rather than a software question and we probably can't help.

arv8315 commented 8 months ago

Oh, it got closed! Finding equivalency between pre-existing images and the rendered reference image (e.g. silhouette match) might help if there is solver limitation with using images other than the other generated via reference to the 3D model itself (like what is there in this tutorial). If the equivalency is found, can use the rendered camera views now. The solver works when working with the rendered camera views.

arv8315 commented 8 months ago

Perhaps then, it is solving it twice. anyhow, know what to try out