search

tukilabs / Video-Compression-Net

A new approach to video compression by refining the shortcomings of conventional approach and substituting each traditional component with their neural network counterpart. Our proposed work consists of motion estimation, compression and compensation and residue compression, learned end-to-end to minimize the rate-distortion trade off. The whole model is jointly optimized using a single loss function. Our work is based on a standard method to exploit the spatio-temporal redundancy in video frames to reduce the bit rate along with the minimization of distortions in decoded frames. We implement a neural network version of conventional video compression approach and encode the redundant frames with lower number of bit.
24 stars 12 forks source link

Regularization for optical flow values? #5

Closed heydarshahi closed 1 year ago

heydarshahi commented 1 year ago

Hi!

Did you use any regularization for optical flow values? I'm trying to train the model end-to-end but optical flow values get arbitrarily large. I wonder if you've faced a similar problem.

Thanks!

Amin

prasangadhungel commented 1 year ago

Hi Heydarshahi!

We also trained the model end-to-end. However, it is difficult to train all the components at once. So, what we did was, we first trained the optical flow component alone. Then we used this weight in the optical flow component of the video compression pipeline without freezing it. That means the optical flow component is still trained in the end-to-end setting.

For your reference, Here is the code to train the optical flow component only.

Regards, Prasanga

heydarshahi commented 1 year ago

Thanks for the explanation, Prasanga!