search

NVlabs / CF-3DGS

Other
404 stars 43 forks source link

CF-3DGS Workflow #40

Open lh-rovco opened 3 months ago

lh-rovco commented 3 months ago

I tried to run the code following the example but the Tanks/Francis model could not be trained with my GPU, because it only has 8GB VRAM. So I ran again with a smaller custom dataset of 50 images. The training finished successfully.

After training I assumed we should run: python run_cf3dgs.py --source data/custom --mode eval_pose --data_type custom --model_path output/progressive/data_custom/chkpnt/ep00_init.pth and then python run_cf3dgs.py --source data/custom --mode eval_nvs --data_type custom --model_path output/progressive/data_custom/chkpnt/ep00_init.pth ?

For the eval_pose my result was

Downloading: "https://github.com/intel-isl/MiDaS/zipball/master" to /home/user/.cache/torch/hub/master.zip
Total images:  50
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Downloading: "https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt" to /home/user/.cache/torch/hub/checkpoints/dpt_hybrid_384.pt
100%|█████████████████████████████████████████████████████████████████████████████████| 470M/470M [00:42<00:00, 11.5MB/s]
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Traceback (most recent call last):
  File "/work/working/run_cf3dgs.py", line 53, in <module>
    trainer.eval_pose()
  File "/work/working/trainer/cf3dgs_trainer.py", line 663, in eval_pose
    trans_gt_align, trans_est_align, _ = self.align_pose(poses_gt[:, :3, -1].numpy(),
  File "/work/working/trainer/cf3dgs_trainer.py", line 704, in align_pose
    raise ValueError("Input matrices must contain >1 unique points")
ValueError: Input matrices must contain >1 unique points

I went ahead and ran eval_nvs mode in case eval_pose is optional:

Downloading: "https://github.com/intel-isl/MiDaS/zipball/master" to /home/user/.cache/torch/hub/master.zip
Total images:  50
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Training progress: 100%|█████████████████████████████████████████████████| 1200/1200 [01:33<00:00, 12.99it/s, PSNR=20.62]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /home/user/.cache/torch/hub/checkpoints/vgg16-397923af.pth
100%|█████████████████████████████████████████████████████████████████████████████████| 528M/528M [00:47<00:00, 11.7MB/s]
Downloading: "https://raw.githubusercontent.com/richzhang/PerceptualSimilarity/master/lpips/weights/v0.1/vgg.pth" to /home/user/.cache/torch/hub/checkpoints/vgg.pth
100%|███████████████████████████████████████████████████████████████████████████████| 7.12k/7.12k [00:00<00:00, 1.82MB/s]
Number of 000 to 006 frames: PSNR : 20.993, SSIM : 0.824, LPIPS : 0.406                      | 0.00/7.12k [00:00<?, ?B/s]
Training progress: 100%|█████████████████████████████████████████████████| 1200/1200 [02:30<00:00,  7.98it/s, PSNR=20.62]
Duration: 0:02:30.791701

eval_nvs seemed to train 6 images from my dataset.

It is not clear to me what the output is meant to be; is this a correct result? If there a splat file I can load into a viewer?

Any help is appreciated. For your information, I am running in a docker image.

rolpotamias commented 4 weeks ago

I have a similar issue. Did you manage to solve this?

hankeceli commented 3 weeks ago

I tried to run the code following the example but the Tanks/Francis model could not be trained with my GPU, because it only has 8GB VRAM. So I ran again with a smaller custom dataset of 50 images. The training finished successfully.

After training I assumed we should run: python run_cf3dgs.py --source data/custom --mode eval_pose --data_type custom --model_path output/progressive/data_custom/chkpnt/ep00_init.pth and then python run_cf3dgs.py --source data/custom --mode eval_nvs --data_type custom --model_path output/progressive/data_custom/chkpnt/ep00_init.pth ?

For the eval_pose my result was

Downloading: "https://github.com/intel-isl/MiDaS/zipball/master" to /home/user/.cache/torch/hub/master.zip
Total images:  50
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Downloading: "https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt" to /home/user/.cache/torch/hub/checkpoints/dpt_hybrid_384.pt
100%|█████████████████████████████████████████████████████████████████████████████████| 470M/470M [00:42<00:00, 11.5MB/s]
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Traceback (most recent call last):
  File "/work/working/run_cf3dgs.py", line 53, in <module>
    trainer.eval_pose()
  File "/work/working/trainer/cf3dgs_trainer.py", line 663, in eval_pose
    trans_gt_align, trans_est_align, _ = self.align_pose(poses_gt[:, :3, -1].numpy(),
  File "/work/working/trainer/cf3dgs_trainer.py", line 704, in align_pose
    raise ValueError("Input matrices must contain >1 unique points")
ValueError: Input matrices must contain >1 unique points

I went ahead and ran eval_nvs mode in case eval_pose is optional:

Downloading: "https://github.com/intel-isl/MiDaS/zipball/master" to /home/user/.cache/torch/hub/master.zip
Total images:  50
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Using cache found in /home/user/.cache/torch/hub/intel-isl_MiDaS_master
Training progress: 100%|█████████████████████████████████████████████████| 1200/1200 [01:33<00:00, 12.99it/s, PSNR=20.62]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /home/user/.cache/torch/hub/checkpoints/vgg16-397923af.pth
100%|█████████████████████████████████████████████████████████████████████████████████| 528M/528M [00:47<00:00, 11.7MB/s]
Downloading: "https://raw.githubusercontent.com/richzhang/PerceptualSimilarity/master/lpips/weights/v0.1/vgg.pth" to /home/user/.cache/torch/hub/checkpoints/vgg.pth
100%|███████████████████████████████████████████████████████████████████████████████| 7.12k/7.12k [00:00<00:00, 1.82MB/s]
Number of 000 to 006 frames: PSNR : 20.993, SSIM : 0.824, LPIPS : 0.406                      | 0.00/7.12k [00:00<?, ?B/s]
Training progress: 100%|█████████████████████████████████████████████████| 1200/1200 [02:30<00:00,  7.98it/s, PSNR=20.62]
Duration: 0:02:30.791701

eval_nvs seemed to train 6 images from my dataset.

It is not clear to me what the output is meant to be; is this a correct result? If there a splat file I can load into a viewer?

Any help is appreciated. For your information, I am running in a docker image.

Would you mind sharing your Dockerfile?

Songxinlei commented 3 weeks ago

have you solved this problem?