Low quality of the bear scene

[liru0126]

I use the following command to edit the bear scene: ns-train in2n --data ../dataset/bear/ --load-dir ./outputs/bear/nerfacto/2023-09-05_083218/nerfstudio_models/ --pipeline.prompt "Turn the bear into a grizzly bear" --pipeline.guidance-scale 6.5 --pipeline.image-guidance-scale 1.5 nerfstudio-data --downscale-factor 2 However, the results are unsatisfactory. I searched the issues of the git and found that the selection of guidance-scale and image-guidance scale is important. Could you please provide some advice about the training commands of the scenes in your paper? The following is the result when editing the bear scene.

[ayaanzhaque]

This is quite surprising, we used essentially the same guidance scales. I will debug the codebase this weekend

[liru0126]

Thanks for your reply. I re-trained the original NeRF using: ns-train nerfacto --data ../dataset/bear/ nerfstudio-data --downscale-factor 2 and the editing NeRF using the aforementioned command. The results are still unsatisfactory. Looking forward to your results.

[liru0126]

Thanks for your reply. I re-trained the original NeRF using: ns-train nerfacto --data ../dataset/bear/ nerfstudio-data --downscale-factor 2 and the editing NeRF using the aforementioned command. The results are still unsatisfactory. Looking forward to your results.

Original NeRF converges quickly and can generate high-quality videos.

[CaffeyChen]

I got the same low quality of the bear scene as yours. I think it might be a problem with iterating images.

[Friedrich-M]

I've also met the same problem of face scene. When I visualized the rendered and edited images every iteration, I found the editing results showed much worse quality. Maybe it occurs due to some modification of the new NeRFStudio version 0.3.3.

[ayaanzhaque]

It seems that it might be an issue with a newer version of diffusers. Let me confirm

[zhao-yiqun]

@ayaanzhaque , Hi~ Does it mean that it is beacause the high version of diffusers ? I wonder what is the diffusers version you used in you implementation. Also, I use the new version of diffusers on huggingface, the image version are already wrong. Is this result similar to yours?

[zhao-yiqun]

It seems that it might be an issue with a newer version of diffusers. Let me confirm

@ayaanzhaque Hi, I tried all six scenes in your paper, and found that only the bear scene lead to such low quality. Have you ever found the reason about that? Thanks in advance for your reply.

[BPAWD]

Hi, I try the campsite scene and the result is also different from the paper. With the following training script:

ns-train in2n \
  --data ./data/campsite-small/ \
  --load-dir ./outputs/campsite-small/nerfacto/original_scene/nerfstudio_models/ \
  --pipeline.prompt {"Make it look like it just snowed"} \
  --pipeline.image-guidance-scale 1.5 \
  --pipeline.guidance-scale 7.5 \

I got some result like: All the people disapear.

[BPAWD]

It seems that it might be an issue with a newer version of diffusers. Let me confirm

@ayaanzhaque Hi, I tried all six scenes in your paper, and found that only the bear scene lead to such low quality. Have you ever found the reason about that? Thanks in advance for your reply.

Hi, did you get good results on other scene like campsite and farm-small? My results on these two scenes are not as good as the paper.

[ttaa9]

It seems that it might be an issue with a newer version of diffusers. Let me confirm

Did you confirm that it is the diffusers version vs nerfstudio version vs something else?

[hrz2000]

I find it very easy to edit poor quality images as in the above picture for p2p for nerfstudio rendered images, which is very common(but not very common for non-rendered images), and I wonder why this is because

[ayaanzhaque]

@lsongx can u share what diffusers version you have?

[lsongx]

diffusers version 0.21.4

here is an example of the output from edit_image in ip2p.py

cond_im

output

command used:

ns-train in2n --data bear --load-dir bear/nerfacto/default/nerfstudio_models --pipeline.prompt "Turn the bear into a grizzly bear" --pipeline.guidance-scale 6.5 --pipeline.image-guidance-scale 1.5  nerfstudio-data --downscale-factor 2

[ayaanzhaque]

What is your nerfstudio version?

[ayaanzhaque]

And can you share your versions for transformers and accelerate?

[lsongx]

nerfstudio 0.3.4 with e8cee3fd transformers 4.34.1 accelerate.version '0.23.0'

[ayaanzhaque]

Can you share your results after training the entire thing?

[lsongx]

I see the issue now... After 3500iters

https://github.com/ayaanzhaque/instruct-nerf2nerf/assets/12553253/86e194c9-93ba-4f7c-a01a-8179153e432c

[ayaanzhaque]

Ya this seems like an issue with the images being edited weirdly, can you try printing some of the images around this many iterations

[lsongx]

Hi @ayaanzhaque , I saved the images out: some are good, some are bad.

I tried the original version of ip2p (not updated since Jan, https://github.com/timothybrooks/instruct-pix2pix)

My understanding is that it is fine if the instructp2p doesn't get good results on some views, as long as it works on some views. Do you think this is really due to instructp2p diffuser pipeline? I tend to assume that something in nerfstudio has changed.

[lsongx]

@ayaanzhaque I think I found out the reason. It is due to camera optimizer. Now after 3500 iters.

https://github.com/ayaanzhaque/instruct-nerf2nerf/assets/12553253/84cbc8ce-03fe-4687-975a-176d0fc3e6ea

[ayaanzhaque]

Ah, thank you!! So turning off the camera optimizer seems to fix the issue?

[lsongx]

I submitted a PR, but not sure if this can solve other guys' problem. Maybe merge after get confirmed by them.

@ttaa9 @liru0126 @CaffeyChen @BPAWD @zhao-yiqun Do you mind giving this a try: https://github.com/ayaanzhaque/instruct-nerf2nerf/pull/74

[hrz2000]

I just used the strategy you proposed, and I still have the problems you encountered before (but it is better than my previous results) I wonder if anyone else has the same problem as me

update：I remember, when I loaded the optimizer parameters, I reported an error, but I ignored it. I'll check again

[lsongx]

@hrz2000 Thanks for testing! Could you pull that PR with git clone https://github.com/lsongx/instruct-nerf2nerf.git, git checkout scheduler-fix and then do pip install -e . again?

[hrz2000]

@lsongx , I did as you said, but the result is still not good

My instructions are：

ns-train in2n --data '/root/picasso/hrz2000/workplace/DreamEditor/data/bear' \
--load-dir ../instruct-nerf2nerf/outputs/bear/nerfacto/2023-10-24_120015/nerfstudio_models \
--pipeline.prompt "Turn the bear into a grizzly bear" \
--pipeline.guidance-scale 6.5 \
--pipeline.image-guidance-scale 1.5  \
nerfstudio-data --downscale-factor 4

If the down-sampling factor is 1, this result will be produced

[hrz2000]

When I retrained the nerf model 2.2w step and edited it 2k steps, the result seemed much better (this time I didn't get error when loading optimizer)

https://github.com/ayaanzhaque/instruct-nerf2nerf/assets/79900945/3fc86dd4-e686-41d5-9bfa-9458506c00c3

[ayaanzhaque]

Ok cool, looks like the camera optimizer is the issue. I'll go ahead and merge the PR!

[Iliceth]

I have done several attempts with:

ns-train in2n-small --data data/nerfstudio/bear --load-dir outputs/bear/nerfacto/2023-11-08_215410/nerfstudio_models --pipeline.prompt "make it a grizzly bear" --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5 nerfstudio-data --downscale-factor 2

But the results always turn to something similar as below:

The code used it the one after the update from a few days ago.

[lsongx]

@Iliceth Could you try completely removing existing in2n and nerfstudio, and reinstalling them again? You can use

git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
pip install --upgrade pip setuptools
pip install -e .

for nerfstudio. Also, did you try in2n? I didn't test in2n-small.

[Iliceth]

@Iliceth Could you try completely removing existing in2n and nerfstudio, and reinstalling them again? You can use

git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
pip install --upgrade pip setuptools
pip install -e .

for nerfstudio. Also, did you try in2n? I didn't test in2n-small.

@lsongx I was under the impression I had the latest version, as I had done my re-install just after the camera optimizer update was done. But created a new env, installed the dependencies, nerfstudio and in2n anyhow, as you requested. Ran in2n-small (as I only have 12 GB) again first and success, got the output I expected. :-) Have extra changes been implemented in the last days?

[lsongx]

Yeah there is a PR merged last week

[ayaanzhaque]

So is this issue resolved @lsongx?

[lsongx]

@ayaanzhaque Yeah, I'd say so. It's working great for me now. Not sure about everyone else on this thread, though.