search

ckkelvinchan / RealBasicVSR

Official repository of "Investigating Tradeoffs in Real-World Video Super-Resolution"
Apache License 2.0
906 stars 136 forks source link

Process after saving the checkpoint #23

Closed fellow-tom closed 2 years ago

fellow-tom commented 2 years ago

Hi. First of all, thank you very much for your project. The quality is impressive. I'm trying to train a neural network, but after every save checkpoint it starts some long process for 300 iterations. It looks like evaluating, but I couldn't find a value of 300 in the config file. I train on the REDS dataset (24k images) and that process takes longer than the training itself for 10k iterations. What is it? Is there any way to reduce this value (300)? Is it possible to disable it and what is the risk?

Example: screen

ckkelvinchan commented 2 years ago

It is validation. What validation data did you use? Here 300 corresponds to 300 sequences.

fellow-tom commented 2 years ago

It is validation. What validation data did you use? Here 300 corresponds to 300 sequences.

I used the REDS validation dataset (3k images). In my first attempt to train RealBasicVSR, I used this REDS validation dataset for both validation and training. It took much less time. When I downloaded the REDS training dataset for training, the validation process became very long. Now I wait 2 hours for 10k iterations of training and 12 hours of validation.

ckkelvinchan commented 2 years ago

If you use the official validation set. There should be 30 sequences instead of 300. I suspect that there is something wrong in the configuration file. Did you modify it?

fellow-tom commented 2 years ago

If you use the official validation set. There should be 30 sequences instead of 300. I suspect that there is something wrong in the configuration file. Did you modify it?

I executed the command: python crop_sub_images.py --data-root ./data/REDS --scales 4 and the dataset became 30k instead of 3k

ckkelvinchan commented 2 years ago

I suggest using the provided configuration to test on UDM10. In this case you only need to test on 10 clips during validation. Also you can reduce the frequency of validation.

fellow-tom commented 2 years ago

I suggest using the provided configuration to test on UDM10. In this case you only need to test on 10 clips during validation. Also you can reduce the frequency of validation.

Thank you very much. I'll give it a try.

fellow-tom commented 2 years ago

It helped. Thanks!