Feature Request: (RIFE) Please add support for rational frame rate changes (like vs-rife has)

[Samhayne]

Unfortunately multi is an int.

vs-rife replaced multi with factor_num, factor_den with version 3.0.0 to support rational frame rate changes.

see: https://github.com/HolyWu/vs-rife/releases

[hooke007]

Unfortunately, the feature in https://github.com/AmusementClub/vs-mlrt/commit/2f6d5989a33833042a7a5f49ea0d22c0d30ec92d is not as safe as factor_num, factor_den in rife-ncnn-vk. If I tried with mpv, it would used up memory and make my program frozen.

[WolframRhodium]

How to reproduce it in vsedit/vs-preview?

[hooke007]

I cannot. I remembered holywu once pushed some commits too in his original repo to fix the similar issue. But the related issues are gone by deleting his repo.

[WolframRhodium]

Since we have used it successfully in encoding, without reliable reproduction we can not validate the claim.

This repository should contain the relevant commit. Do you mean this one?

[hooke007]

It is included i think.

[WolframRhodium]

Why do you think fps_num is better? It is equivalent to multi=target.fps / source.fps for CFR videos, and I don't think that implementation handles VFR videos.

[hooke007]

No, i never talked about fps_num... To make it clear, factor_num=5 factor_num_den=2 works fine in rife-ncnn-vk. multi=Fraction(5,2) makes the program frozen.

[WolframRhodium]

Then factor_num and factor_den, which is equivalent to multi=Fraction(factor_num, factor_den), isn't it?

multi=Fraction(5,2) makes the program frozen.

I can't reproduce that with vsedit or vs-preview.

[hooke007]

I can't reproduce that with vsedit or vs-preview.

I knew, that's why I said But the related issues are gone by deleting his repo. The more details explaination are only recorded in those deleted issue.

[WolframRhodium]

Then we can do nothing.

And we are using different implementation here compared to the original approach.

[aloola18]

multi=Fraction(5,2)

I can confirm this, I tried with MPV and MPC, and both froze. MPC sometimes froze for ~3 minutes and then ran successfully but it froze most of the time.

[WolframRhodium]

but it froze most of the time

Could you provide more information? For example, does the freezing occur at the same frame? Is it related to scene change, and what if you disable scene change detection?

[aloola18]

but it froze most of the time

Could you provide more information? For example, does the freezing occur at the same frame? Is it related to scene change, and what if you disable scene change detection?

It froze when opening any video file or when seeking(when it ran successfully). it's eating a lot of RAM while freezing

I disabled scene change detection, still froze.

[WolframRhodium]

Are you using SVP with "Duplicate frames removal" enabled?

By "opening any video file", do you mean it does not return any frame at all?

[aloola18]

yeah, just a black screen and froze.

no, I'm not using SVP.

[WolframRhodium]

Does the frozen time depends on the number of frames of the video?

[aloola18]

Does the frozen time depends on the number of frames of the video?

seems so, video > 20 minutes took forever to load.

[WolframRhodium]

That's a new information. What if you change your script to the following:

• before:

  output = vsmlrt.RIFE(xxx)

• after:

  output = vsmlrt.RIFE(xxx).std.BlankClip(color=[0.5] * 3)

Does the playback still froze?

[aloola18]

cut2 = RIFE(cut1, multi=Fraction(5,2), model=46, backend=BackendV2.TRT(force_fp16=True, num_streams=2, output_format=1, static_shape=True)).std.BlankClip(color=[0.5] * 3)

video still froze

[WolframRhodium]

Do you know Python? I want to know which line of code in the RIFE() function of vsmlrt.py blocks.

This can be checked by inserting a line return clip before each function calls (i.e. line 1147, 1164-1167, 1169, 1176-1179). With this change the original video is returned, but some insertions may not have immediate display.

[aloola18]

I inserted return clip into: line 1146: video play normal w/o interpolation line 1163: froze about 5 seconds then started playing normally w/o interpolation line 1168: video froze

so these lines cause issues?

left_clip = core.akarin.PickFrames(clip, left_indices)
right_clip = core.akarin.PickFrames(clip, right_indices)
tp_clip = core.std.BlankClip(clip, format=gray_format, length=len(timepoints))
tp_clip = tp_clip.akarin.PropExpr(lambda: dict(_tp=timepoints)).akarin.Expr('x._tp')

[WolframRhodium]

line 1163: froze about 5 seconds then started playing normally w/o interpolation

This is alarming because it should not froze.

In the original program, what if you repeat line 1142-1162 https://github.com/AmusementClub/vs-mlrt/blob/2f6d5989a33833042a7a5f49ea0d22c0d30ec92d/scripts/vsmlrt.py#L1142-L1162 several times? Will each repetition increase the frozen time by 5 seconds?

[aloola18]

Will each repetition increase the frozen time by 5 seconds?

yes, it freeze for a very long time. around 30~40 seconds

[WolframRhodium]

That's a very useful information.

What if you simply call len(timepoints) multiple times in the loop without assignment, does it affect performance?

Alternatively, insert the following piece of code

with open("num_frames.txt", "w") as f:
    f.write(str(clip.num_frames))

This will create a file. What's the value in it?

[aloola18]

called len(timepoints) 5 times took 10 seconds to play a 90 seconds video 25s for 20 minutes video

What's the value in it?

10810800

[WolframRhodium]

10810800

Thanks, now I fully understand what happens.

Video playback is different from encoding in that the player will create a clip with a large number of frame, while for encoding the value is at most hundreds of thousands. This make the initialization much more demanding for vudeo playback.

I will start to fix the issue now.

[aloola18]

input = VpsFilterSource.std.Trim(length=500000) #mpc input = video_in.std.Trim(length=500000) #mpv

this prevents the freezing forever, but still slow

the value 10000 is fast at least, still don't know about the side effects thought.

[WolframRhodium]

The effect of 10000 may be the player closed after playing 10000 frames?

Another side effect with frame interpolation in video players is that, when it jumps to another frame and returns to the previous frame, the player may renders differently.

How is the variable video_in defined?

[hooke007]

How is the variable video_in defined?

https://github.com/mpv-player/mpv/blob/master/video/filter/vf_vapoursynth.c

the value 10000 is fast at least, still don't know about the side effects thought. The effect of 10000 may be the player closed after playing 10000 frames?

The player would destroy the vf_vapoursytn filter after 10000 frames. User have to re-enable the filter again to make it actually work.

when it jumps to another frame and returns to the previous frame, the player may renders differently.

When users are seeking, mpv would destroy the filter and re-activate it again automatically.

[WolframRhodium]

Thanks for the support.

Please try this initial implementation with RIFE(video_player=True).

This code may change later.

[hooke007]

https://gist.githubusercontent.com/WolframRhodium/ec584a966e67b877c09c9c495c792555/raw/310bb2eb0dd2e258c15f6d2bef92b2cede705cff/rife_test.py No frozen but it becomes a slowmo clip instead.

[WolframRhodium]

What do you mean by a slowmo clip? Why this change is related?

[hooke007]

Tried (5,2) Looks like duration is wrong. It feels like turing down the playback speed.

[WolframRhodium]

This line is necessary to prevent integer overflow when multi > 16, however.

[hooke007]

No, I mean the version I tested had the problem. Not this line caused the problem. I also tested the inital version, it also had the same issue.

[WolframRhodium]

What about multi=2? Will it slows down the video?

[hooke007]

2 works fine.

[WolframRhodium]

What if you change this line to src_fps = Fraction(24, 1) or other values?

[hooke007]

Emmm, doesn't help. mpv has a quirk on fps and duration.

For example, the original change_fps function doesn't work correctly in mpv. I made a little bit tweak to make it match the duration properly.

So I finally used a workaround below.

...
clip = RIFE(...multi=Fraction(5, 2), video_player=True...)
clip = clip.std.AssumeFPS(fpsnum=5 * container_fps * 1000, fpsden=2 * 1000)

[netExtra]

What difference does setting video_payer=True make?

[WolframRhodium]

Allows the fractional RIFE to be usable in video players.

[netExtra]

Allows the fractional RIFE to be usable in video players.

So won't make any difference with mpv using svp? I've tried it by the way but didn't see anything. I assume SVP is not using fractional rife. Sorry one more question, how does fractional rife help?

[aloola18]

Allows the fractional RIFE to be usable in video players.

So won't make any difference with mpv using svp? I've tried it by the way but didn't see anything. I assume SVP is not using fractional rife. Sorry one more question,

SVP has its method to handle Fraction and it works very well. I don't know if this will conflict with SVP though.

how does fractional rife help?

like if you want 24fps video to 60fps, then use a fraction(5, 2).

clip = RIFE(...multi=Fraction(5, 2), video_player=True...) clip = clip.std.AssumeFPS(fpsnum=5 container_fps 1000, fpsden=2 * 1000)

this works well for me. btw do you know any way to improve MPV seek time when using RIFE? MPV reload vapoursynth on every seeking, makes it slow compared with MPC-HC which instantly on seeking.

[hooke007]

btw do you know any way to improve MPV seek time when using RIFE?

IMO, This requires an expert to rewrite the implement of vs in mpv. Any heavy filters suffer the similar pain.

[WolframRhodium]

No difference and conflict with existing svp.

Yes svp should be rewritten.

[netExtra]

btw do you know any way to improve MPV seek time when using RIFE?

IMO, This requires an expert to rewrite the implement of vs in mpv. Any heavy filters suffer the similar pain.

Tbh I don't have major mpv seek issues any more. V1 with the akarin update is as good as V2 which has always been good.

[netExtra]

Allows the fractional RIFE to be usable in video players.

So won't make any difference with mpv using svp? I've tried it by the way but didn't see anything. I assume SVP is not using fractional rife. Sorry one more question,

SVP has its method to handle Fraction and it works very well. I don't know if this will conflict with SVP though.

how does fractional rife help?

like if you want 24fps video to 60fps, then use a fraction(5, 2).

clip = RIFE(...multi=Fraction(5, 2), video_player=True...) clip = clip.std.AssumeFPS(fpsnum=5 container_fps 1000, fpsden=2 * 1000)

this works well for me.

Where do I place this code? I assume I have to replace some existing code?

[hooke007]

Tbh I don't have major mpv seek issues any more

Are you sure you can seek as fast as playing without svp? Or you do not think it's a 'major' problem here?

Where do I place this code?

I assume you were using svp, you have to wait for its support.

[aloola18]

btw I tested with multi = Fraction(3, 2) but it not working

3.18.18

[netExtra]

Tbh I don't have major mpv seek issues any more

Are you sure you can seek as fast as playing without svp? Or you do not think it's a 'major' problem here?

SVP with Rife v4.13(v2) At 1080p x 2 (3D) downscaled to 1080p with catmull_rom dscale + antiring lanczossharpest cscale, correct and linear downscaling. adaptive-sharpen shader

hwdec=no : 1080p or 4K is about a second or less between each seek. hwdec=d3d11va-copy : 1080p seek is almost instant. See attached. The last couple of seeks are chapter seeks.

https://github.com/AmusementClub/vs-mlrt/assets/140253706/5a2f11cc-c9d3-4ff8-99c9-f6e0e2714f70