search

ChenyangLEI / deep-video-prior

[NeurIPS 2020] Blind Video Temporal Consistency via Deep Video Prior
323 stars 39 forks source link

Processing larger videos #14

Closed Saartjes closed 3 years ago

Saartjes commented 3 years ago

HI, I am trying to process a large video (10 min) to deflicker after colorization. I have left my system on to run for about 24 hours after witch it trained for 50 epochs. After 50 epochs the result is still quite poor and looks more like the original black and white video than the colorized one. Did you ever test on longer video’s? Is the result going to improve with more training epochs? Or is it impossible to use your model on a video this long? Thanks in advance for your answer!

ChenyangLEI commented 3 years ago

It is recommended to first extract frames from the video using a scene detection technique such as pyscenedetect https://pyscenedetect.readthedocs.io/en/latest/ . I.e., for each scene, it would be better to train a model. Can you try it first?

Saartjes commented 3 years ago

Thank you for your swift response!

I have tried splitting the video into scenes and that seems to do the trick! I now have to figure some way of dealing with the color consistency between scenes.

Also if I compare the result of 100 epochs vs 50 epochs it seems that there is a certain ‘sweet spot’. When trained for 100 epochs the color in the resulting video seems to flicker quite a bit more compared to the video result of 50 epoch training. Is this something you have experienced as well? I would assume at a certain point the model starts to overfit the data,

ChenyangLEI commented 3 years ago

It depends on the number of training frames. For example, we use 25 epochs for 30-200 frames. However, if the number of frames are large and the motion is large, then more epochs are required.

Saartjes commented 3 years ago

I have been playing around with the number of training epochs but I am not yet satisfied with the results.

For example I took a scene where a part of a shirt changes color as a shadow passes over. This color shift is omitted after 400 epochs. The overall colors however still do look washed out:

afbeelding

After training for 500 epochs the colors are just as good as the processed video but the color shift is now back. The results seems to completely mimic the processed video:

afbeelding

Quite unsatisfying as the results you where able to achieve in your paper look downright marvelous!

ChenyangLEI commented 3 years ago

What is resolution and numbers of your vframes？Many things can lead to unsatisfying performance. Send the video to my email. Thanks.