Support for LLFF dataset?

[AIBluefisher]

Hi, @tancik @liruilong940607

I wonder if nerfacc can have a test on the LLFF dataset since I found very weird results when training InstantNGP with nerfacc's pipeline. There is a related issue.

[liruilong940607]

Unfortunately we didn't play with LLFF with NDC. But I do know someone treated LLFF as an unbounded scene and can train it with our unbounded script.

If you are using NDC, I can imagine that those hyper-parameter such as render_step_size, aabb, near, far etc would need to be adjusted.

[AIBluefisher]

Thanks for your suggestion. Actually, I follow the dataset loading of ngp_pl and tried without NDC. The results show InstantNGP overfits on the training set. I also tried treating LLFF as an unbounded scene (data are loaded the same as mipnerf-360), the synthesized images are good on the validation set, but the depth images showed nothing. (gt, rendered image, rendered depth)

I did not find a suitable solution now.

[AIBluefisher]

It is the visualization issue for the depth map. Best viewed in the colorized inverse depth map.

[AIBluefisher]

Also tried to tune the bounding box size. The code is below:

# pip install trimesh
def visualize_poses(poses, size=0.1):
    # poses: [B, 4, 4]

    axes = trimesh.creation.axis(axis_length=4)
    # box = trimesh.primitives.Box(extents=(2, 2, 2)).as_outline()
    # box.colors = np.array([[128, 128, 128]] * len(box.entities))
    box = trimesh.creation.box(bounds=[[-5, -5, -5], [7, 5, 10]]) # the aabb corners that need to be tuned
    box.visual.face_colors = [0, 1., 0, 0.2]
    objects = [axes, box]

    for pose in poses:
        # a camera is visualized with 8 line segments.
        pos = pose[:3, 3]
        a = pos + size * pose[:3, 0] + size * pose[:3, 1] + size * pose[:3, 2]
        b = pos - size * pose[:3, 0] + size * pose[:3, 1] + size * pose[:3, 2]
        c = pos - size * pose[:3, 0] - size * pose[:3, 1] + size * pose[:3, 2]
        d = pos + size * pose[:3, 0] - size * pose[:3, 1] + size * pose[:3, 2]

        dir = (a + b + c + d) / 4 - pos
        dir = dir / (np.linalg.norm(dir) + 1e-8)
        o = pos + dir * 3

        segs = np.array([[pos, a], [pos, b], [pos, c], [pos, d], [a, b], [b, c], [c, d], [d, a], [pos, o]])
        segs = trimesh.load_path(segs)
        objects.append(segs)

    trimesh.Scene(objects).show()

Hope it can help someone else that has a similar issue as mine.

[aiyb1314]

How does nerfacc support LLFF datasets? assert params is None, "Only support pinhole camera model." AssertionError: Only support pinhole camera model.