Closed Amazingren closed 1 year ago
Hi, the t1 and t2 here are Temporal Relevance coefficients.
https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L14
Please refer to the following section of the paper.
Hi @feinanshan
Oh, my bad~ thanks a lot for the quick and kind explanation :)
By the way, If it is possible and when you are free, let's have a quick discussion here about the splatting task via M2M:
For splatting, let's use I_0
and forward optical flow flow_forward
to warp I_0
to I_1
.
Hence, the fltTime
t0 and t1 should be set to t0=1
and t1=1-t0=0
, right?
Based on this, now the temporal relevance coefficientst1_temporal = t1 = 0
and t2_temporal = t0 = 1
,
so for the flowing line: https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L17
The inputs for softsplat_func.apply(tenIn, tenFlow)
now are:
torch.cat([tenIn * 0 * (tenMetric).clip(-20.0, 20.0).exp(), 0 * (tenMetric).clip(-20, 20).exp()], 1)
flow_forward
Now the tenIn
is the point that I am not totally clear about. So if I want to do splatting with M2M, is there any suggestion on how to solve this problem?
Bests,
Hi, yes in this extreme case of t0=1, tenIn (from I_0 to t0) will be zero which means no information will be warped from I_0.
Yet note that we are interpolating with two frames, hence at the same time, the tenIn (from I_1 to t0) will be equivalent to I_1 itself.
Hi, yes in this extreme case of t0=1, tenIn (from I_0 to t0) will be zero which means no information will be warped from I_0. Yet note that we are interpolating with two frames, hence at the same time, the tenIn (from I_1 to t0) will be equivalent to I_1 itself.
Got it! thanks a lot! :)
Hey @feinanshan,
Hope you doing well!
I have a question regarding the parameter
fltTimes
, I noticed it was set to 0.5 in your code. Does it mean we want to Interpolate a new frame between the correct middle between img0 and img1? if it is, why did you first set the order as (t1, t0) to the functionforwarp_mframe_mask()
here:https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L359 but then you used it as the opposite ordert0
andt1
within the functionforwarp_mframe_mask()
https://github.com/feinanshan/M2M_VFI/blob/5d4d75345689885a8314a98bd7ad9c18c324d64c/Test/model/m2m.py#L14For 0.5, it would always be fine, but I am curious what if I want to do JUST the FORWARD SPLATTING task with
img0
andflow_forward
to getimg1_estimate
, instead of the video frame interpolation task from both sides? How should I change it?I am looking forward to your suggestion.
Bests,