search

tarun005 / FLAVR

Code for FLAVR: A fast and efficient frame interpolation technique.
Apache License 2.0
455 stars 69 forks source link

Finetune problem #20

Closed ECEMACHINE closed 3 years ago

ECEMACHINE commented 3 years ago

Hello, Thanks for your brilliant work but I have a problem about the finetune. When I finetune your model on my own dataset, the finetuned model predicted twinkled videos and I output the predicted frame, I found that the predicted frame was darker than the adjacent frames. Then I tried train the model from the start using Unet34, but got the similar results that darker. And the PSNR and training loss were improving, but the inference results were worse. Could you please explain to me a little? It's the training details python main.py --batch_size 8 --test_batch_size 8 --dataset vimeo90K_septuplet --loss 1L1 -max_epoch 200 --lr 0.00001 --n_outputs 1 Namespace(batch_size=8, beta1=0.9, beta2=0.99, checkpoint_dir='.', cuda=True, data_root='/vimeo_septuplet', dataset='vimeo90K_septuplet', exp_name='exp' , joinType='concat', load_from=None, log_iter=60, loss='1L1', lr=1e-05, max_epoch=200, model='unet_18', n_outputs=1, nbr_frame=4, nbr_width=1, num_gpu=1, num_workers=16, pretrained='FLA VR_2x.pth', random_seed=12345, resume=False, resume_exp=None, start_epoch=0, test_batch_size=8, upmode='transpose', use_tensorboard=False, val_freq=1

tarun005 commented 3 years ago

I am not sure what is causing the problem. Did you check the pre-processing and post processing steps match that in the code?

ECEMACHINE commented 3 years ago

I am not sure what is causing the problem. Did you check the pre-processing and post processing steps match that in the code? I don't think the infer code has problems since the result of your model seems really good. And I checked the training datasets, the data has no problem. It seems really wired. The training PSNR even pass 38.

tarun005 commented 3 years ago

It typically happens when the mean and std used for preprocessing do not match that of evaluation. Urge you to check if the dataloader pipelines are consistent.

ECEMACHINE commented 3 years ago

It typically happens when the mean and std used for preprocessing do not match that of evaluation. Urge you to check if the dataloader pipelines are consistent.

I checked my pipelines and checked the images in every dataloader, it seems good. And I tried the Vimeo90k using your code,it turned out the PSNR was around just 15. Then I output the images of the dataloader, I found that the images of the same video were flipped by different angles or randomly cropped which means that the previous frame was not consistent with next frame. I don't know if it's right? https://github.com/tarun005/FLAVR/blob/d07e162970ba985b7f46303775bc81c1ba97bd06/dataset/vimeo90k_septuplet.py#L32

tarun005 commented 3 years ago

I did fix the random seed before applying the transformation. It is somehow not working in your case. What versions of pytorch, torchvision and PIL are you using?

ECEMACHINE commented 3 years ago

I did fix the random seed before applying the transformation. It is somehow not working in your case. What versions of pytorch, torchvision and PIL are you using?

I am so sorry I did't notice the requirements. Thanks so much!

Astrostellar commented 5 months ago

Hi, could you please share the pretrained model? The current links seem to be unavailable