Trained with Only Gray Results

[YuxinCHENGPE]

Hi, I tried the baseline on Blender dataset step by step under the guidance, started with ns-launch inpaint-blender and followed by ns-launch train-blender. The trained NeRF models and output rendering results are all in gray style and the camera outlines appear in the rendering results. I have no idea why it happened because I didn't change anything in code. I wander if you encouter with the same case and do you have any solution for this? Thanks so much.

[ethanweber]

Hey, so what looks to be happening is that the cameras are all cheating and the middle pixels in the frame are being closed really close up to each camera as "floaters". That's why you see the artifacts in the same location as the camera frame. If you open up the viewer while training, it should be even more apparent. Perhaps something changed in the base nerfstudio, but my suspicion is that the near plane is not being set properly. Can you trace down the specific command you ran to train this model? ns-launch will run everything in batch, but perhaps we can look at the specific command and track down why it did this. As for the inpaints, do they look reasonably consistent? For some of the baselines, I actually expect the result you are showing. Sometimes the training will fail completely when not using a good multi-view inpainting method.

[YuxinCHENGPE]

Thanks for your reply. I also noticed that the artifacts are the "floaters", which caused that the camera-path render results are still incomplete and the dataset render results are "overfitting". I followed the instruction to inpaint the blender datasets firstly with inpaint-blender and then train them with train-blender. Some command samples show below (with --dry-run option):

• Inpainting:

  --------------------------------------------------------------------------------
  Available GPUs:  [0, 1]
  --------------------------------------------------------------------------------
  Command:
  CUDA_VISIBLE_DEVICES=0 ns-inpaint grid-prior --nerfstudio-dataset data/nerfstudio/drums
  
  Command:
  CUDA_VISIBLE_DEVICES=1 ns-inpaint expanded-attention --nerfstudio-dataset data/nerfstudio/drums
  
  --------------------------------------------------------------------------------
  Available GPUs:  [0, 1]
  --------------------------------------------------------------------------------
  Command:
  CUDA_VISIBLE_DEVICES=0 ns-inpaint individual-sd-image --nerfstudio-dataset data/nerfstudio/drums
  
  Command:
  CUDA_VISIBLE_DEVICES=1 ns-inpaint individual-sd-text --nerfstudio-dataset data/nerfstudio/drums --prompt "a photo of drums"

• Train

  --------------------------------------------------------------------------------
  Available GPUs:  [0, 1]
  --------------------------------------------------------------------------------
  Command:
  CUDA_VISIBLE_DEVICES=0 ns-train nerfacto-nerfiller --data data/nerfstudio/ship-inpaint/grid-prior/2024-09-12_181410 --experiment-name ship-grid-prior --viewer.websocket-port=8082 --viewer.quit_on_train_completion True --vis viewer --viewer.default-composite-depth False --pipeline.model.camera_optimizer.mode off --pipeline.datamanager.masks-on-gpu True --pipeline.datamanager.images-on-gpu True --pipeline.model.background_color white --pipeline.model.disable_scene_contraction True --pipeline.model.distortion_loss_mult 0.0
  
  Command:
  CUDA_VISIBLE_DEVICES=1 ns-train nerfacto-nerfiller --data data/nerfstudio/hotdog-inpaint/grid-prior/2024-09-12_191908 --experiment-name hotdog-grid-prior --viewer.websocket-port=8082 --viewer.quit_on_train_completion True --vis viewer --viewer.default-composite-depth False --pipeline.model.camera_optimizer.mode off --pipeline.datamanager.masks-on-gpu True --pipeline.datamanager.images-on-gpu True --pipeline.model.background_color white --pipeline.model.disable_scene_contraction True --pipeline.model.distortion_loss_mult 0.0

  After training, I visualize some scenes with ns-viewer, all of them shows in gray color while the input images for training are indeed RGB color, which makes me confused.

As for the floaters, I guess this is due to inpaint inconsistency. But I do not know how to change the codes or settings to get better results and need your help. There are two videos I rendered with the comand render-blender your provide to show the inpaints with grid-prior method and NeRF renders, shown as follow:

• Dataset render:
• Camera-path render:

Looking forward to your help. Thanks :)

[ahmadsalimi]

I am also facing the same issue for the object-centric scenes. I guess something has changed in Nerfacto's implementation in more recent versions of nerfstudio that is causing the issue. Do you by any chance know the nerfstudio version used for the original experiments? I am using 1.1.3.

UPDATE: increasing the near plane helped with the floaters, but the reconstruction is still completely gray.

[ethanweber]

Hey all, I've been informed that Nerfstudio version 1.0.0 works as expected. I'm not sure what has changed between now and then, but I hope this helps to know in the meantime until figuring that out! (Please let me know if something learns why the results are gray for the latest Nerfstudio version.)