search

pHaeusler / codef-experiments

10 stars 0 forks source link

Some question about training #4

Open FSYFSYFSY opened 1 year ago

FSYFSYFSY commented 1 year ago

Thanks for the work and note! it really helps me understand the original project, but I still have some questions about training by run.py.

I was running it on A5000, and I set batch size to 4, but the speed barely improved, although I did see the iteration of each epoch was only a quarter of before.

And what are the differences between training with canonical images and without canonical images? What's the train/loss of your successful case?

The final question is whether it is possible to reduce the times of validation Dataloader? I see it runs every epoch and cost almost half of the time, do we need to load our dataset again and again?

Thanks!

FSYFSYFSY commented 1 year ago

And my deform image ara totally black or like this

image
pHaeusler commented 1 year ago

You can reduce the validation frequency by increasing this option in pytorch lighning: check_val_every_n_epoch=1

Training without a canonical model is faster and more stable, but not as flexible. Essentially you just train a model to warp the canonical image across time.

I found the most success without a canonical model and with very few frames (sub sampling the frames). Then reconstructing the video with a higher frame rate to make it smooth again.