How to add boundary constraint

[abeacco]

Hi again,

I'm still experimenting using your smooth_integration function, and I'm struggling with a problem when I try to add a boundary constraint.

Let me explain:

• I have a normal map (p and q) and a depth map (z0)
• I want to integrate the depth using the normals just within the silhouette, not at the boundary.
• My boundary constraint would be to have the same depth at the boundary than in my original depth map once the integration has been done.

As far as I understand, and from previous issues, to set how much the original depth is respected I have to play with lambda. And since lambda is defined for every pixel, I thought that setting a different lambda on the boundary pixels would be the way to define my constraint.

The problem is that when I do this, integration is not doing anything at all, it says that the initial guess is within the tolerance and then not iterating. If I change the tolerance to a smaller value, then it iterates but gives me an integrated depth that does not respect my constraint.

I think that the problem might come from me not understanding the units and/or format of the initial depth I have to use, and what is the range of the computed integrated depths. Is there a relation between the image size and the depth range? How should I prepare my data?

I am attaching a modified demo of your code that you should be able to run to test the problem. BoundaryConstraintDemo.zip

Do you think you know what is happening here? What can be my mistake or the problem? Thank you very much.

Alejandro Beacco

[yqueau]

Hi Alejandro,

This is an intersting problem actually, and I would have first followed the same approach as you (make lambda spatially-varying, with high values on the boundaries and low values elsewhere). It would be more elegant to directly put the boundary knowledge into the linear system being solved. This is not extremely difficult but a bit tedious, maybe if I have time someday I'll add this feature but...

Your approach seems to work in my case ! I just changed the precision for pcg to 1e-12 (as default value I had set 1e-4), that is the third argument of the call on line 53 in smooth_integration.m

Alternatively, if you have enough RAM, you can use the 'direct' solver (calls Cholesky factorization instead of conjugate gradient iterations), on line 30 in your script.

For checking, I just set lambda to 1e6 also outside the mask, the effect is visually easier to understand : so on line 41 I set : lambda(im_contour == 1 | mask==0) = 1e6;

Hope that helps ! Best, Yvain

[abeacco]

Hi Yvain,

Sorry I was out during the week-end. I have seen what you say, and test all the tolerances and options you mention. I think I understand, but it doesn't seem to work for what I need/want.

Let me send you again an example, but this time with the data I'm working with: SMPLTest.zip As you might remember, I am trying to work with humanoids and more precisely with renders of the SMPL model.

When you execute the demo you will see that the integrated depth seems really flattened compared to the one I have as z0. So I am really not sure about what is the problem. Could it be that my normals (and then p and q) are not in the proper format again? :(

Thank you for all your help and dedication!

Best,

Alejandro Beacco

[yqueau]

Hi Alejandro, I had a look, your handling of (p,q,z0,lambda) seems fine, and the script does what it is supposed to.

If you want it less "flattened", the simplest would just be to tweak a bit (p,q), e.g. do p=3p; q = 3q; will triple the slopes.

I guess the problem comes from the fact that I guess you get normals and depth from some rendering engine. The link depth-normals thus depends from the intrinsics of the camera i.e., the coordinates of the principal point and the focal. If you know this process then the definition of (p,q) should take this into account, cf. e.g. Eq. (17) in https://arxiv.org/pdf/1709.05940.pdf , where f is the focal length, and (u,v) the pixel coordinates with respect to principal point, and (p,q) is the gradient of the log-depth map.

Best, Yvain

[abeacco]

Hi Yvain,

You are absolutely right. Depth and normals are rendered by opendr in python, but I think it is using a weak perspective projection (from what I've red). Could that also be one of the problems? I'm now looking at your survey and trying to figure out if there is a better solution (following Eq. 17 like you suggest) than manually tweaking the slopes (which works well by the way).

I'll let you know if I make it work.

Thank you again!

Sincerely,

Alejandro Beacco

[yqueau]

I see. If rendering is done under perspective projection things are quite simple : basically you just need to multiply (p,q) by the ratio (distance to focus plane / focal length). This is around Sect. 2.2 in the survey mentioned earlier.

Good luck ! Yvain

[yqueau]

Woops, just read my answer again: I obviously meant weak perspective projection (Sect. 2.2 in survey), if it is full perspective scaling by distance to focus/focal is not enough, cf. 2.3 in survey.

Cheers, Yvain