TypeError - Githubissues

yyhyyhyyhyyh commented 2 months ago

Hi!I meet a problem.

[NOTE] Not running eval iterations since only viewer is enabled. Use --vis {wandb, tensorboard, viewer+wandb, viewer+tensorboard} to run with eval. Disabled comet/tensorboard/wandb event writers Printing profiling stats, from longest to shortest duration in seconds GaussCtrlTrainer.train_iteration: 0.0805
Traceback (most recent call last): File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/bin/ns-train", line 8, in sys.exit(entrypoint()) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint main( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main launch( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch main_func(local_rank=0, world_size=world_size, config=config) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop trainer.train() File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 201, in train loss, loss_dict, metrics_dict = self.train_iteration(step) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner out = func(*args, *kwargs) File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 275, in train_iteration self.grad_scaler.scale(loss).backward() # type: ignore File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/autograd/init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/autograd/function.py", line 288, in apply return user_fn(self, args) TypeError: backward() takes 2 positional arguments but 3 were given

So I want to ask if this has anything to do with the version issue?Thank you!

yyhyyhyyhyyh commented 2 months ago

and I want to ask you torch version.Thank you!

jingwu2121 commented 2 months ago

We are using torch 2.1.2, see here

pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

yyhyyhyyhyyh commented 2 months ago

We are the same version, do you know what the error above is and why there is an extra parameter

jingwu2121 commented 2 months ago

Are you able to perform the 2D editing? What command are you trying to run, can you share the command? Are you getting the error when doing splatfacto training?

yyhyyhyyhyyh commented 2 months ago

This error occurred after generating the view done editing.The command is: ns-train gaussctrl --load-checkpoint output/bear/splatfacto/2024-09-07_140514/nerfstudio_models/step-000029999.ckpt --experiment-name bear --output-dir editbear --pipeline.datamanager.data data/bear --pipeline.prompt "a photo of a golden bear statue in the forest" --pipeline.guidance_scale 5 --pipeline.chunk_size 1 --pipeline.langsam_obj 'bear'

yyhyyhyyhyyh commented 2 months ago

train 3DGS is successful.This error occurred Edit model

jingwu2121 commented 2 months ago

If I understand correctly, you have no problem running the code until here? The error happens when continuing the 3DGS training?

yyhyyhyyhyyh commented 2 months ago

Sorry,I am unable to debug on the server,but I think maybe the key is: File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 201, in train loss, loss_dict, metrics_dict = self.train_iteration(step) The complete shell for this section is as follows： Start Editing: Reference views are [5, 12, 30, 32] ############################# Generating view: [0] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:21<00:00, 1.07s/it] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [1] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [2] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [3] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [4] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [5] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [6] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [7] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [8] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [9] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [10] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [11] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [12] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [13] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [14] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [15] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [16] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [17] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [18] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [19] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [20] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [21] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [22] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [23] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [24] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [25] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [26] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [27] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [28] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [29] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [30] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [31] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [32] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [33] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [34] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [35] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [36] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [37] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [38] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. Generating view: [39] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:18<00:00, 1.06it/s] Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed. ############################# Done Editing ############################# ╭─────────────── viser ───────────────╮ │ ╷ │ │ HTTP │ http://0.0.0.0:7007 │ │ Websocket │ ws://0.0.0.0:7007 │ │ ╵ │ ╰─────────────────────────────────────╯ [NOTE] Not running eval iterations since only viewer is enabled. Use --vis {wandb, tensorboard, viewer+wandb, viewer+tensorboard} to run with eval. Disabled comet/tensorboard/wandb event writers Printing profiling stats, from longest to shortest duration in seconds GaussCtrlTrainer.train_iteration: 0.0805
Traceback (most recent call last): File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/bin/ns-train", line 8, in sys.exit(entrypoint()) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint main( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main launch( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch main_func(local_rank=0, world_size=world_size, config=config) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop trainer.train() File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 201, in train loss, loss_dict, metrics_dict = self.train_iteration(step) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner out = func(*args, *kwargs) File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 275, in train_iteration self.grad_scaler.scale(loss).backward() # type: ignore File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/autograd/init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/autograd/function.py", line 288, in apply return user_fn(self, args) TypeError: backward() takes 2 positional arguments but 3 were given

yyhyyhyyhyyh commented 2 months ago

Maybe something wrong at return user_fn(self, args) the args but when I print *args,I just see some tensors

jingwu2121 commented 2 months ago

Hi there, can you try these things?

Comment out these lines, so what it does is that it doesn't perform 2D editing anymore, you just use your unedited images to continue the splat training. Have a look at if using original images to train splat works
If it works then, the problem might be the edited images. Here you didn't successfully edit the images, since all the images you got were black. You might want to visualize those edited images. see this line.
Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed.
- A few things to try are:
- removing the NFSW detected in diffusers. You can do so by cloning diffusers to your local machine and doing pip install -e . in the diffusers lib directory, then the local diffusers lib is installed in your conda environment. Then in the diffusers lib, find the corresponding controlnet pipeline file, comment out the NSFW part. Refer to here
- trying another seed/prompt and make sure you have correct edited images first

yyhyyhyyhyyh commented 2 months ago

ok,I will have a try

yyhyyhyyhyyh commented 2 months ago

I comment out these lines you marked,but meet the same problem. Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 7.57it/s] Done loading Nerfstudio checkpoint from output/bear/splatfacto/2024-09-07_140514/nerfstudio_models/step-000029999.ckpt ╭─────────────── viser ───────────────╮ │ ╷ │ │ HTTP │ http://0.0.0.0:7007 │ │ Websocket │ ws://0.0.0.0:7007 │ │ ╵ │ ╰─────────────────────────────────────╯ [NOTE] Not running eval iterations since only viewer is enabled. Use --vis {wandb, tensorboard, viewer+wandb, viewer+tensorboard} to run with eval. Disabled comet/tensorboard/wandb event writers (viser) Connection opened (0, 1 total), 469 persistent messages Printing profiling stats, from longest to shortest duration in seconds GaussCtrlTrainer.train_iteration: 69.1761
Traceback (most recent call last): File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/bin/ns-train", line 8, in sys.exit(entrypoint()) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint main( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main launch( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch main_func(local_rank=0, world_size=world_size, config=config) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop trainer.train() File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 201, in train loss, loss_dict, metrics_dict = self.train_iteration(step) File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 112, in inner out = func(*args, *kwargs) File "/mnt/cloud_disk/yyhyyh/gaussctrl/gaussctrl/gc_trainer.py", line 275, in train_iteration self.grad_scaler.scale(loss).backward() # type: ignore File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/autograd/init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/mnt/cloud_disk/yyhyyh/anaconda3/envs/gaussctrl/lib/python3.8/site-packages/torch/autograd/function.py", line 288, in apply return user_fn(self, args) TypeError: backward() takes 2 positional arguments but 3 were given terminate called without an active exception Aborted (core dumped)

jingwu2121 commented 2 months ago

It might be a version problem, but I cannot reproduce the error. I tried your command on my machine, it works fine. A way to work around is that, you manually continue the splat training. A simpler way to start is that you edit all the images with GaussCtrl pipeline, then continue 3D editing with the splatfacto training.

use the load_all argument. Something like this. I just fixed a bug in gc_datamangaer, be sure to update.

ns-train gaussctrl --load-checkpoint unedited_models/bear/splatfacto/2024-07-10_170906/nerfstudio_models/step-000029999.ckpt --experiment-name bear --output-dir editbear --pipeline.datamanager.load_all True --pipeline.datamanager.data data/bear --pipeline.prompt "a photo of a golden bear statue in the forest" --pipeline.guidance_scale 5 --pipeline.chunk_size 1 --pipeline.langsam_obj 'bear'

Here, you save all the images to your local images folder. With the same file structure

.
├── camera_paths
└── images
├── sparse_pc.ply
└── transforms.json

Continue the 3DGS training. In GaussCtrl, we just continue training for 500 iters. So here, you can do the same.

ns-train splatfacto --load-checkpoint unedited_models/.../step-000029999.ckpt --output-dir temp --experiment-name bear_try --steps-per-save 500 --max-num-iterations 500 --viewer.quit-on-train-completion True nerfstudio-data --data data/bear_edited

Note: In our paper, we didn't edit all the images, in order to speed up the training, i.e. load_all is disabled. If you want to reproduce our paper experiments with this work-around method, besides saving the images, you also need to edit the pose information in transforms.json. Simply doing this by removing pose elements that don't have a corresponding image in the key frames. If you load all the images, then you don't need to deal with pose

jingwu2121 commented 2 months ago

Hi there, also have a look at this. They are getting the same error as yours

yyhyyhyyhyyh commented 2 months ago

Thank you!I will have a look

ActiveVisionLab / gaussctrl

TypeError #11