Render Imgs for NeRF dataset

[StephenYangjz]

Hi,

I am trying to convert the generated AI2Thor into a NeRF dataset. However the generated room does not seem to be at the origin, and if I sample a couple of views from the center, its looking into the outside. Do you by any chance have ideas of a better way to capture a NeRF dataset from the generated room?

Thank you.

[YueYANG1996]

You need to change the camera position and angle based on the room. Here are two functions you can refer to get the images: https://github.com/allenai/Holodeck/blob/7094a014acc6ad0c8ddb9a739e3760940e1f3e21/modules/utils.py#L153 https://github.com/allenai/Holodeck/blob/7094a014acc6ad0c8ddb9a739e3760940e1f3e21/modules/utils.py#L265

[StephenYangjz]

@YueYANG1996 Thank you for your reply. Tho how may I generate imgs in the room's coordinate system given a camera transform? I tried to sample the images in a hemisphere and output them for a NeRF reconstruction using AI2thor APIs, but the "background" gets well reconstructed but the room is getting broken into pieces (as attached). I think the room and the background lie in different coordinate systems that is not compatible. It would be great if I can get any pointers to solve the issue. Appreciate it!

In the image, the squares are the supervised camera angle images and the pink traces are the pieces of the room.

[YueYANG1996]

Could you show me the code to generate this image?

[StephenYangjz]

Hi @YueYANG1996 This is a NeRF rendering from images like shown below. Those imgs are generated with AI2Thor API:

    event = controller.step(
        action="AddThirdPartyCamera",
        position=dict(x=point[0], y=point[1], z=point[2]),
        rotation=dict(x=rotation[0], y=rotation[1], z=rotation[2]),
        fieldOfView=120
    )
    # rendering imgs

    img = np.array(event.third_party_camera_frames)
    image = img.astype(np.uint8)[-1]

[YueYANG1996]

The position and rotation of the camera are off. Please refer to this line to calculate the correct position: https://github.com/allenai/Holodeck/blob/7094a014acc6ad0c8ddb9a739e3760940e1f3e21/modules/utils.py#L314C9-L314C22

[StephenYangjz]

@YueYANG1996 Thank you for your reply. Is that just shifting cam center or shall or the angles also be converted?

[YueYANG1996]

Depends on your usage; if you want to rotate 360 degrees at a fixed position, please refer to here: https://github.com/allenai/Holodeck/blob/7094a014acc6ad0c8ddb9a739e3760940e1f3e21/modules/utils.py#L186 https://github.com/allenai/Holodeck/blob/7094a014acc6ad0c8ddb9a739e3760940e1f3e21/modules/utils.py#L196