Feature: Adding Surface Normals Sensor

[mathfac]

Author @danielgordon10: I have code to turn depth into surface normals, but ideally it should be something provided by habitat-api directly.

Here is my surface normals code. One nice thing about it is I can batch it and run on GPU, though you should at least be able to do the second part on habitat-sim side if necessary.

import torch import torch.nn.functional as F

surfnorm_kernel = None def depth_to_surface_normals(depth, surfnorm_scalar=256):

depth is torch tensor in N x C x H x W order.

global surfnorm_kernel
if surfnorm_kernel is None:
    surfnorm_kernel = torch.from_numpy(
        np.array([[[1, 2, 1],
                   [0, 0, 0],
                   [-1, -2, -1]],
                  [[1, 0, -1],
                   [2, 0, -2],
                   [1, 0, -1]],
                  [[0, 0, 0],
                   [0, 0, 0],
                   [0, 0, 0]]])
    )[:, np.newaxis, ...].to(
        dtype=torch.float32, device=depth.device)
with torch.no_grad():
    surface_normals = F.conv2d(depth, surfnorm_scalar * surfnorm_kernel, padding=1)
    surface_normals[:, 2, ...] = 1
    surface_normals = surface_normals / surface_normals.norm(dim=1, keepdim=True)
return surface_normals

[danielgordon10]

Formatting: I have code to turn depth into surface normals, but ideally it should be something provided by habitat-api directly.

Here is my surface normals code. One nice thing about it is I can batch it and run on GPU, though you should at least be able to do the second part on habitat-sim side if necessary.

import torch
import torch.nn.functional as F

surfnorm_kernel = None
def depth_to_surface_normals(depth, surfnorm_scalar=256):
    # depth is torch tensor in N x C x H x W order.
    global surfnorm_kernel
    if surfnorm_kernel is None:
    surfnorm_kernel = torch.from_numpy(
        np.array([[
            [1, 2, 1],
            [0, 0, 0],
            [-1, -2, -1]],
           [[1, 0, -1],
            [2, 0, -2],
            [1, 0, -1]],
           [[0, 0, 0],
            [0, 0, 0],
            [0, 0, 0]]])
             )[:, np.newaxis, ...].to(dtype=torch.float32, device=depth.device)
    with torch.no_grad():
        surface_normals = F.conv2d(depth, surfnorm_scalar * surfnorm_kernel, padding=1)
        surface_normals[:, 2, ...] = 1
        surface_normals = surface_normals / surface_normals.norm(dim=1, keepdim=True)
        return surface_normals

[Yash621]

@mathfac can I work on this ?

[mathfac]

Yes, sure. On Fri, Nov 5, 2021 at 10:13 AM Yash Goel @.***> wrote:

@mathfac https://github.com/mathfac can I work on this ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/habitat-lab/issues/24#issuecomment-962070840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGWHAE3SPJK2G72EZTNDFDUKQGELANCNFSM4HDONC2Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.