search

SpectacularAI / 3dgs-deblur

[ECCV2024] Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion
https://spectacularai.github.io/3dgs-deblur/
Apache License 2.0
125 stars 8 forks source link

Questions about custom data training #7

Closed Vonjor closed 1 month ago

Vonjor commented 1 month ago

Hi. This is a great job. I wanted to train it with my own data. I shot a video using the Spectacular Rec app on my Android phone, And use the script./scrpts/process_and_train_sai_custom.sh /PATH/TO/spectacular-rec-MY_RECORDING, but it reports errors as follows: (nerfstudio) x@x:~/Desktop/prog/3dgs-deblur$ ./scripts/process_and_train_sai_custom.sh ./3dgs-deblur/data/myvideo/spectacular-rec_2024-05-29_15-27-29.zip ++ basename./3dgs-deblur/data/myvideo/spectacular-rec_2024-05-29_15-27-29.zip

output written to /tmp/tmpniwln2cj/sai-cli/spectacular-rec_2024-05-29_15-27-29 COMMAND: ['python', 'run_colmap.py', '/tmp/tmpniwln2cj/sai-cli/spectacular-rec_2024-05-29_15-27-29', '/tmp/tmpniwln2cj/colmap-sai-cli-imgs/spectacular-rec_2024-05-29_15-27-29'] Processing /tmp/tmpniwln2cj/sai-cli/spectacular-rec_2024-05-29_15-27-29 4 images (would be) copied in a temporary directory ['ns-process-data', 'images', '--data', '/tmp/tmppywzee7v', '--output-dir', '/tmp/tmpniwln2cj/colmap-sai-cli-imgs/spectacular-rec_2024-05-29_15-27-29'] Traceback (most recent call last): File "run_colmap.py", line 101, in process(case, args) File "run_colmap.py", line 42, in process ret = subprocess.run(cmd, check=True, capture_output=True) File "/home/x/anaconda3/envs/nerfstudio/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ns-process-data', 'images', '--data', '/tmp/tmppywzee7v', '--output-dir', '/tmp/tmpniwln2cj/colmap-sai-cli-imgs/spectacular-rec_2024-05-29_15-27-29']' returned non-zero exit status 1. Traceback (most recent call last): File "process_sai_custom.py", line 140, in process(args) File "process_sai_custom.py", line 112, in process maybe_run_cmd(colmap_cmd) File "process_sai_custom.py", line 36, in maybe_run_cmd if not args.dry_run: subprocess.check_call(cmd) File "/home/x/anaconda3/envs/nerfstudio/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['python', 'run_colmap.py', '/tmp/tmpniwln2cj/sai-cli/spectacular-rec_2024-05-29_15-27-29', '/tmp/tmpniwln2cj/colmap-sai-cli-imgs/spectacular-rec_2024-05-29_15-27-29']' returned non-zero exit status 1.

oseiskar commented 1 month ago

Hi @Vonjor : the main error seems to be

subprocess.CalledProcessError: Command '['ns-process-data', 'images', '--data', '/tmp/tmppywzee7v', '--output-dir', '/tmp/tmpniwln2cj/colmap-sai-cli-imgs/spectacular-rec_2024-05-29_15-27-29']' returned non-zero exit status 1.

that is, COLMAP failed for some reason. A bit earlier, there's

Processing /tmp/tmpniwln2cj/sai-cli/spectacular-rec_2024-05-29_15-27-29 4 images (would be) copied in a temporary directory

Which means that only 4 images passed the pipeline, which seems low. You can try these things

  1. Run the pipeline without COLMAP 

    SKIP_COLMAP=ON ./scripts/process_and_train_sai_custom.sh ./3dgs-deblur/data/myvideo/spectacular-rec_TIMESTAMP.zip

  2. Try again with another recording. The main thing to keep eye on is that it's currently not recommended to shoot extremely close to objects (keep a distance of 1m or more). However, there's also other reasons for it to relatively randomly fail so just trying the same thing again may help too

  3. Modify the script to support a smaller scale: In particular you can add something like --key_frame_distance=0.02 here https://github.com/SpectacularAI/3dgs-deblur/blob/b5bbc895e81c91c494faeb2c13be7f6ba5c3a41e/scripts/process_and_train_sai_custom.sh#L36C8-L36C26 (the default is 0.1, which mean 10cm, that is, the different camera poses selected for 3DGS training must be 10cm apart).

Vonjor commented 1 month ago

Thanks to your reply, I try these two and it worked

  1. Run the pipeline without COLMAP 
    SKIP_COLMAP=ON ./scripts/process_and_train_sai_custom.sh ./3dgs-deblur/data/myvideo/spectacular-rec_TIMESTAMP.zip
  2. Try again with another recording. The main thing to keep eye on is that it's currently not recommended to shoot extremely close to objects (keep a distance of 1m or more). However, there's also other reasons for it to relatively randomly fail so just trying the same thing again may help too

Now it can be trained and save a checkpoint model,but i can't use this to metric my model:

python parse_outputs.py

I check my folder and there is no file named metrics.json, so how to metric the model like PSNR、SSIM

oseiskar commented 1 month ago

I check my folder and there is no file named metrics.json, so how to metric the model like PSNR、SSIM

You can remove the --no_eval and --train_all flags here. https://github.com/SpectacularAI/3dgs-deblur/blob/b5bbc895e81c91c494faeb2c13be7f6ba5c3a41e/scripts/process_and_train_sai_custom.sh#L37

However, if your goal is to get the best possible reconstruction from your data, then it's best to leave those in an not waste input data on the test set. With most scenes, evaluation metrics using automatically/evenly selected evaluation sets are dominated by random cruft at the edges of the images. The numbers are useful for comparing different methods on the same dataset, but not for evaluating the quality of a particular scene/model.

Vonjor commented 1 month ago

Thank you very much for your answer, this is a great work！