nerfacto: Weird circular crops and ripples (when using no scene contraction and no render crop)

[f-dy]

Describe the bug When using no scene contraction, I get weird empty discs (no rgb, no accumulation, no depth) in renders, unless I enable "crop viewport" or enable scene contraction. This happens even if the camera is inside the crop box. Those are always perfect discs in 3D (ellipses in 2D), which are aligned and centered on the 3 axes. crop may happen inside or outside the disc (see screenshots)

Also ripples appear in normals when not cropping (see images in comment below)

No issue with tensorf.

To Reproduce Steps to reproduce the behavior:

dataset="blender/lego"
model=nerfacto
model_opts=(--pipeline.model.predict-normals True) # for mesh extraction
model_opts+=(--experiment-name "${dataset}")
viewer_opts=(--vis viewer --viewer.websocket-port=7007)
data_opts=(blender-data --data data/"${dataset}")

# Disable camera optimization
model_opts+=(--pipeline.datamanager.camera-optimizer.mode off)
# For white-background images
model_opts+=(--pipeline.model.background-color white)
model_opts+=(--pipeline.model.proposal-initial-sampler uniform)
#model_opts+=(--pipeline.model.near-plane 2. --pipeline.model.far-plane 6.)
# All images in a 360 have the same exposure etc.:
model_opts+=(--pipeline.model.use-average-appearance-embedding False)
# Distortion loss from mipnerf360, prevents floaters:
model_opts+=(--pipeline.model.distortion-loss-mult 0)
# Disable scene contraction,since the scene is bounded:
model_opts+=(--pipeline.model.disable-scene-contraction True)

# Train with blender lego dataset:
timestamp=$(date "+%Y-%m-%d_%H%M%S")
ns-train "${model}" "${model_opts[@]}" --timestamp "${timestamp}" "${viewer_opts[@]}" "${data_opts[@]}"

• launch training
• move around until the discs appear. They often appear when moving close.
• all the maps have the same holes (depth, accumulation, rgb)
• all holes disappear when cropping.

Expected behavior no holes

Screenshots without "crop viewport":

with crop viewport:

without crop:

with crop:

you can get two discs if you look along a diagonal:

or even three:

with crop:

 nvidia-smi
Wed Jun 21 23:56:30 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   42C    P0    26W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

[f-dy]

I think there's more. Here's a render of the pred_normals without cropping. Notice the ripples.

Same with crop viewport, I get much less ripples:

Looks like the no-crop render path has an issue

[f-dy]

I would tend to think that _intersect_with_sphere is the culprid, but the math looks ok

[gafniguy]

Hi @f-dy , have you had any success with Nerfacto on the object-centric datasets, without contraction, since posting?

[f-dy]

nope