feinanshan / M2M_VFI

Many-to-many Splatting for Efficient Video Frame Interpolation
Other
79 stars 5 forks source link

Question about the fileTimes meaning #11

Closed Amazingren closed 1 year ago

Amazingren commented 1 year ago

Hey @feinanshan,

Hope you doing well!

I have a question regarding the parameter fltTimes, I noticed it was set to 0.5 in your code. Does it mean we want to Interpolate a new frame between the correct middle between img0 and img1? if it is, why did you first set the order as (t1, t0) to the function forwarp_mframe_mask() here:https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L359 but then you used it as the opposite order t0 and t1 within the function forwarp_mframe_mask()https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L14

For 0.5, it would always be fine, but I am curious what if I want to do JUST the FORWARD SPLATTING task with img0 and flow_forward to get img1_estimate, instead of the video frame interpolation task from both sides? How should I change it?

I am looking forward to your suggestion.

Bests,

feinanshan commented 1 year ago

Hi, the t1 and t2 here are Temporal Relevance coefficients.
https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L14

Please refer to the following section of the paper.

image

Amazingren commented 1 year ago

Hi @feinanshan

Oh, my bad~ thanks a lot for the quick and kind explanation :)

By the way, If it is possible and when you are free, let's have a quick discussion here about the splatting task via M2M:

For splatting, let's use I_0 and forward optical flow flow_forward to warp I_0 to I_1.

Hence, the fltTime t0 and t1 should be set to t0=1 and t1=1-t0=0, right?

Based on this, now the temporal relevance coefficientst1_temporal = t1 = 0 and t2_temporal = t0 = 1,

so for the flowing line: https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L17

The inputs for softsplat_func.apply(tenIn, tenFlow) now are:

Now the tenIn is the point that I am not totally clear about. So if I want to do splatting with M2M, is there any suggestion on how to solve this problem?

Bests,

feinanshan commented 1 year ago

Hi, yes in this extreme case of t0=1, tenIn (from I_0 to t0) will be zero which means no information will be warped from I_0.
Yet note that we are interpolating with two frames, hence at the same time, the tenIn (from I_1 to t0) will be equivalent to I_1 itself.

Amazingren commented 1 year ago

Hi, yes in this extreme case of t0=1, tenIn (from I_0 to t0) will be zero which means no information will be warped from I_0. Yet note that we are interpolating with two frames, hence at the same time, the tenIn (from I_1 to t0) will be equivalent to I_1 itself.

Got it! thanks a lot! :)