search

RLado / STB-VMM

STB-VMM: Swin Transformer Based Video Motion Magnification (official repository)
MIT License
34 stars 5 forks source link

Testing #9

Closed taodaxiong closed 8 months ago

taodaxiong commented 8 months ago

Hello, I am trying to use "ckpt_e49.pth.tar" for magnifying the video "baby.mp4", but the magnified frames become increasingly blurry from the first to the last one. Is this situation normal? Any hint is much appreciated !

RLado commented 8 months ago

Without seeing your result I cannot definitively assure that is normal, but this is expected behaviour.

Why?

By default magnification "mode" is set to "static" meaning that magnification takes as a reference the first frame of the video sequence to magnify against the rest. As time advances it is possible that the shot has moved and consequently larger movements are magnified leading to blurrier results.

model diagram

If you look at the model diagram you will see that the model actually only magnifies the difference between two frames and generally the further away in time those frames are, the larger the differences that you might find.

Alternatively you could magnify one frame against the next one using the "dynamic" mode, but magnification appears less natural/appealing.

Is there a solution?

Maybe... You could try and magnify shorter segments of a larger video to mitigate the problem described above. However, unless you choose the endings and beginnings of your segments very carefully the transition will be noticeable.

Conclusion

You are probably doing things right. This model is pretty good, but far from perfect.

jay-1104 commented 8 months ago

Hey there, I tried to test the model based on the directions you provided in the readme. The model amplifies "baby" video but is unable to magnify larger videos like of wheel and car. The process gets killed while I am using cpu. I also tried using cuda but it still has no output. The program is getting killed or terminated.

RLado commented 8 months ago

Hey there, I tried to test the model based on the directions you provided in the readme. The model amplifies "baby" video but is unable to magnify larger videos like of wheel and car. The process gets killed while I am using cpu. I also tried using cuda but it still has no output. The program is getting killed or terminated.

It seems to me that you are running out of memory when attempting to magnify higher resolution videos. There is no real fix for that, the model is rather memory hungry. But... You could try this thing, it is a total hack but at least it might finish without crashing.

jay-1104 commented 8 months ago

While using cuda, I get gpu out of memory error. So will it still be possible to use this thing or should I use the hack you gave?

taodaxiong commented 8 months ago

While using cuda, I get gpu out of memory error. So will it still be possible to use this thing or should I use the hack you gave?

I can magnify "baby" video by cpu, but when I magnify by CUDA, "CUDA out of memory"are displayed.

taodaxiong commented 8 months ago

Without seeing your result I cannot definitively assure that is normal, but this is expected behaviour.

Why?

By default magnification "mode" is set to "static" meaning that magnification takes as a reference the first frame of the video sequence to magnify against the rest. As time advances it is possible that the shot has moved and consequently larger movements are magnified leading to blurrier results.

model diagram

If you look at the model diagram you will see that the model actually only magnifies the difference between two frames and generally the further away in time those frames are, the larger the differences that you might find.

Alternatively you could magnify one frame against the next one using the "dynamic" mode, but magnification appears less natural/appealing.

Is there a solution?

Maybe... You could try and magnify shorter segments of a larger video to mitigate the problem described above. However, unless you choose the endings and beginnings of your segments very carefully the transition will be noticeable.

Conclusion

You are probably doing things right. This model is pretty good, but far from perfect. Thank you for your explanation, I will try again! In addition, there are some unnatural fractures in the magnified frames. Is there any solution? STBVMM_static_000143

RLado commented 8 months ago

In addition, there are some unnatural fractures in the magnified frames. Is there any solution?

No, this is more or less the same problem as before. I have not found any solutions for that issue.

taodaxiong commented 8 months ago

In addition, there are some unnatural fractures in the magnified frames. Is there any solution?

No, this is more or less the same problem as before. I have not found any solutions for that issue.

Thank you, you've really helped me a lot!

RLado commented 8 months ago

Happy to help!