search

MoyGcc / vid2avatar

Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition (CVPR2023)
https://moygcc.github.io/vid2avatar/
Other
1.2k stars 102 forks source link

Too long training time #26

Open surheaven opened 1 year ago

surheaven commented 1 year ago

I train on single NVIDIA RTX 3090 GPU. The data is only 270 frames, 1400 epochs run for five days. But github and the supplemental files say it only takes 48h, why is that? I see in the code that the maximum epoch of train is 8000. Is it necessary to run so many epochs.

MoyGcc commented 1 year ago

What about your iters per second? It's about 6 iter/s on my local PC and around 3.5 iter/s on my institutional server. In both cases, 48 hours should be enough for a decent result.

surheaven commented 1 year ago

Epoch 2449: 100%|██████▉| 259/260 [02:15<00:00, 1.91it/s, loss=0.0283, v_num=iox1, rgb_loss=0.0277, eikonal_loss=0.00417, bce_loss=0.0273, opacity_sparse_loss=0.000, Epoch 2868: 7%| | 19/259 [00:11<02:26, 1.63it/s, loss=0.0215, v_num=iox1, rgb_loss=0.0211, eikonal_loss=0.00268, bce_loss=0.0202, opacity_sparse_loss=0.000, in_shape_loss=0.0126]Epoch 2973: 84%|▊| 218/259 [01:57<00:22, 1.86it/s, loss=0.0263, v_num=iox1, rgb_loss=0.0258, eikonal_loss=0.00245, bce_loss=0.0346, opacity_sparse_loss=0.000, in_shape_loss=0.0011

hakanErgin commented 8 months ago

I am testing the preprocessed demo on my work laptop with quadro RTX 3000 6gb. had to change pixel_per_batch to 256 image

Extremely slow, but I think I may have a misconfiguration or something.