Editing 3d consistancy

[amirbarda]

Hi, thanks for the code release and interesting work!

I am trying to edit NeRFs using your method. my base model is the famous "spot" cow mesh, and I am attempting to add a chef's hat using the prompt "a photo of a cow with a chef hat" in the "dataset_name_to_prompt" list in the data.py file. I also added it to the synthetic blender dataset list.

my occlusion is the following:

I ran all the experiments,

and the one that seems to give the best results is 'individual-sd-text'.

https://github.com/user-attachments/assets/51038be6-d3b9-4429-98ca-f039a34de513

but I was wondering why there is 3d inconsistency in the renders. as they seem to all be rendered from the same model, at nerfstudio_models/step-000029999.ckpt in the 'individual-sd-text' folder.

thanks!

[amirbarda]

hi, additionally attaching my input data.

cow_chef_hat.zip

[ethanweber]

Hey! So the reason there is inconsistencies is because the NeRF is "cheating" at each viewpoint. If you train with the viewer open, you can move the camera around and you'll probably see artifacts floating in front of the camera. I'd try to look at the inpainted images from the model to see if they look consistent, otherwise I don't expect the NeRF result to be inconsistent either.

[amirbarda]

thanks! this is what I am getting in the inpainted images, is this ok?

[ethanweber]

Gotcha. I think these all look very different, which explains why the flicker happens. You could try doing the inpainting the NeRFiller to see if that works better, but you may have to do it by reference where you first inpaint one image and then try to propagate this image to the rest. We show an example like this in the paper -- it's probably what you're looking for?

[amirbarda]

I did not find the reference image for the cat demo (should be in data/nerfstudio/cat ?), but I used the notebook to create a reference image. just making sure that I am doing this right:

1. I move reference.py to the cow dataset folder in the data/nerfstudio folder
2. I insert "cow_chef_hat+grid-prior-du-reference": '--pipeline.text_guidance_scale 15.0 --pipeline.prompt "a cow with a chef hat" --pipeline.lower-bound 0.8'" to the "dataset_name_method_modifications" list

is that all? thanks!

[amirbarda]

updating that it used this inpainted reference:

and reran. Unfortunately, it does not seem to improve (still very 3d inconsistent). Would it be possible to get the reference.png used with the cat demo? thanks!