Possible in-correct undistortion for ARIA dataset in splatfacto?

[ziruiw-dev]

Describe the bug I recently tried to dump undistorted ARIA RGB images before splatfacto training starts, but the undistorted images look weird to me.

To Reproduce My training command:

# 1. process vrs data to nerfstudio-data
ns-process-data aria \
--vrs-file $aria_data_root/data/$scene_name/video.vrs \
--mps-data-dir $aria_data_root/data/$scene_name/mps/slam \
--output-dir $aria_data_root/ns-processed/$scene_name

# 2. train
ns-train splatfacto nerfstudio-data \
    --data $aria_data_root/ns-processed/$scene_name \
    --eval-mode interval

I am using Aria Digital Catalog dataset, object Airplane_B09WN2RN15_Black_1_active.

To dump undistorted ARIA RGB image, I added this code block

from PIL import Image
idx = data["image_idx"]
img_before = Image.fromarray(image)
img_before.save(f"debug/img_before_{idx}.png")
image = cv2.remap(image, map1, map2, interpolation=cv2.INTER_LINEAR)
img_after = Image.fromarray(image)
img_after.save(f"debug/img_after_{idx}.png")

around cv2.remap here: https://github.com/nerfstudio-project/nerfstudio/blob/cef95890f3e11196ddd6285952cb93064f7160ed/nerfstudio/data/datamanagers/full_images_datamanager.py#L538

Images before and after undistort First image: before undistort

Second image: after undistort

Expected behavior If I understand correctly, the undistorted image should not have a dark corner, and two black regions (highlighted by red circles)?

Additional context Printed camera parameter:

Cameras(
    camera_to_worlds=tensor(
        [[ 0.5399, -0.1729,  0.8238,  0.7783],
        [ 0.7732, -0.2848, -0.5666,  0.2183],
        [ 0.3326,  0.9429, -0.0200,  0.0284]]
    ), 
    fx=tensor([594.3605]), 
    fy=tensor([594.3605]), 
    cx=tensor([712.8746]), 
    cy=tensor([707.1136]), 
    width=tensor([1408]), 
    height=tensor([1408]), 
    distortion_params=tensor(
        [ 4.1283e-01, -4.6563e-01,  1.0973e-01,  1.1713e+00, -1.6854e+00,
          6.3521e-01, -1.0272e-03, -2.5742e-06,  1.0332e-03, -1.5531e-04,
          -3.8399e-04, -1.3548e-04]
    ), 
    camera_type=tensor([9]), 
    times=None, 
    metadata={'fisheye_crop_radius': 707.5}
)

Relevant PR: #2617 #2785

@brentyi, I apologise that I @'ed you directly and I appreciate your effort in adding the feature that supports training splats with ARIA data! I am not very familiar with the ARIA undistortion so I am pretty much lost here. I am wondering if I could get some comments?

Best, Zirui

[jb-ye]

can you also visualize the mask and overlay that with image? I think for fisheye, we cannot get a bbox ROI for valid pixels.

[ziruiw-dev]

@jb-ye Thanks for the fast response!

if I understand correctly, I overlaid this mask

https://github.com/nerfstudio-project/nerfstudio/blob/cef95890f3e11196ddd6285952cb93064f7160ed/nerfstudio/data/datamanagers/full_images_datamanager.py#L547

on the undistorted image. Results:

Raw mask dumped:

[ziruiw-dev]

I think this is not a bug. Closing the issue.

If anyone would like to have undistorted images with all pixels being valid (without dark corners and black regions), use a smaller FOV. For example, multiply 0.8 at the end of this line give me good results for now.