Artefacts described in this ticket are improved in SVP with Rife v4.15 but still present

[netExtra]

When using SVP, Rife tends to have less artefacts than the other mode. I notice that Rife 4.7 has even less artefacts than 4.6. But the problem is that the artefacts that remain tend to be worse. The biggest one I've seen recently is a character appearing and disappering every other frame. Is there any use for me to point out the location of these artefacts ? Is there anyone I should send them to?

[jensdraht1999]

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

[netExtra]

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

https://github.com/hzwer/Practical-RIFE/assets/140253706/756ebfe3-84cf-42ce-8cea-bd402660855f

[jensdraht1999]

@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.

If you test out all different models and all of them exhibit this error, then this might be a rife problem.

It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.

BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.

[netExtra]

@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.

If you test out all different models and all of them exhibit this error, then this might be a rife problem.

It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.

BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.

I've tested 4.6 and 4.7 , both have the same issue and since 4.7 > 4.6 > 4.4 I'm guessing that 4.4 will be the same. Maybe I'm missing something here but even the pay version of Flowframes seems to be only using 4.4 which has even more artefacts and errors.

[jensdraht1999]

FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.

[netExtra]

FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.

Installed Flowframes. Says GPU: Virtual Desktop Monitor even though I have an Nvidia GPU. Tried changing GPU id makes no difference. When interpolating CPU is basically idling for scene extraction and frame extraction. If it's going to be this slow I really would need a realtime mode.

[jensdraht1999]

@netExtra OK if all of them have it, then it's probably RIFE Model, probably needs more training.

[Q8sh2ing]

Hello,

I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.

If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.

But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

https://github.com/hzwer/Practical-RIFE/assets/140253706/756ebfe3-84cf-42ce-8cea-bd402660855f

cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.

[netExtra]

cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.

Thanks but there were other ugly errors in Rife 4.6 that have been fixed in 4.7/4.8 so I would hope that this eventually gets fixed soon. Especially since these large and ugly errors don't occur in SVP when not using Rife.

[netExtra]

4.9 is better. One remainder of an artefact that was improved in 4.7 is now completely gone in 4.9. The Spider-man artefact I show above seems to have improved slightly. I have another similar artefact that has also improved to the point where I can remove it by decreasing the scene detection threshold. It's also a little more efficient than 4.7 too Rife is getting better and better 👍

[netExtra]

Update 04/11/2023 I don't know if this helps but I can remove the Spider-man artefact shown above by significantly reducing rife-scene change threshold in SVP. But it is so low that it cause stuttering issues elsewhere.

[hooke007]

The 'scene change threshold in SVP' you mentioned is out of rife's control. Because it use a different 'solution' to detect scene changes. Once some frames were marked by this 'solution', rife do nothing on them.

[hooke007]

Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.

[netExtra]

Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.

I disabled the sc algorithm. I double checked it was off by setting the sc threshold to 5 which I know fixes the issue in SVP manager GUI. When I run the clip it's broken just like the video attached above. When I re-enable the algorithm the artefact disappears but causes major problems elsewhere so it's not a solution.

I'm not sure how testing a clip without scene-changes would help because there are no issues with those and never have been.

[hooke007]

choose a clip without scene-changes to test.

It's not talking about the movement of len.

Could you give the raw clip of your test src?

[netExtra]

choose a clip without scene-changes to test.

It's not talking about the movement of len.

Could you give the raw clip of your test src?

Hi. Sorry I am confused about your request for a clip without scene changes. I don't have any issues with that. What do you mean by "movment of len"?. Also, my raw clip is the one that is attached to this issue. Thanks.

[hooke007]

I only see the screen recording here. https://github.com/hzwer/Practical-RIFE/issues/34#issuecomment-1747789930

I want to test its raw clip on my side.

[netExtra]

I only see the screen recording here. #34 (comment)

I want to test its raw clip on my side.

OK I'm doing it now. Will upload it in a couple of minutes. Thanks.

[netExtra]

Attached. I had to convert it to MP4.

https://github.com/hzwer/Practical-RIFE/assets/140253706/49bbf0e0-cfd1-4957-aa44-3364894a9060

[hooke007]

Thanks. It could be reproduced by inference_video.py with model4.9. https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687

[netExtra]

Thanks. It could be reproduced by inference_video.py with model4.9. https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687

Now it can be replicated does that help? BTW the attached video doesn't work :)

[hooke007]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

[netExtra]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

OK Thanks. I've raised a new issue for Rife 4.11 but I'm not sure if I should be since it probably doesn't affect the model training.

[netExtra]

OK so this artefact seems to be gone now and I am confused because up until this morning it was still there with Rife 4.9. I have another annoying artefact similar to this which was getting better but seems to have regressed a little. SVP has been updated and new code added so I will have to investigate further.

[netExtra]

Update: The problem artefact is back again and the one that had regressed is back to being improved again. Very strange! Not going to reopen at the moment.

[netExtra]

OK there was a weird issue where the player I was using wasn't registering with SVP after the first file. I'm guessing that is why the artefacts went away because SVP and therefore RIFE was not running.

[netExtra]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.

I didn't try legacy v3/2 models so I have no ideas more.

I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans

BTW the issue applies to Rife v4.6 as well so it probably goes back some way.

[banjaminicc]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec. I didn't try legacy v3/2 models so I have no ideas more.

I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans

BTW the issue applies to Rife v4.6 as well so it probably goes back some way.

At this point you're just confusing SC thresholds. Setting a scene change threshold literally outright disables RIFE on those frames, so obviously there wouldn't be artifacting, and hence why it will obviously be stuttery at some parts.

[netExtra]

BTW the attached video doesn't work

It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec. I didn't try legacy v3/2 models so I have no ideas more.

I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans BTW the issue applies to Rife v4.6 as well so it probably goes back some way.

At this point you're just confusing SC thresholds. Setting a scene change threshold literally outright disables RIFE on those frames, so obviously there wouldn't be artifacting, and hence why it will obviously be stuttery at some parts.

I have disabled the SC threshold in SVP and the artefact is improved but it is still there. But as I said above, I can fix the artefact by setting SVP SC to below 6 but setting SC this low causes stuttering for slow pans.

Here is a a direct quote from the SVP Rife AI FAQ: Scene change threshold - Higher values gives more interpolation artifacts when scene change was not detected. Lower values give more "stuttering" (i.e. repeated frames instead of interpolated) when there was a false detection.

FYI with SC disabled, slow pans are almost perfect but fast object movement causes double images in Rife 4.9 and below. But the double images are gone in Rife 4.12, so there is a visible improvement between 4.9 and 4.12.

[artem-frolov]

In my experience, Rife 4.1 is the best model for panning. 4.12 is slightly worse than 4.1. And 4.13 is slightly worse than 4.12. All other v4 models have significantly more artifacts in fast panning scenes.

[netExtra]

In my experience, Rife 4.1 is the best model for panning. 4.12 is slightly worse than 4.1. And 4.13 is slightly worse than 4.12. All other v4 models have significantly more artifacts in fast panning scenes.

What I was talking about with Rife 4.12/4.13 was fast object movement in live action scenes. Fast panning seems to be identical between the two.

Rife 4.6 has more artefacts than 4.9 so I wouldn't consider using Rife 4.1.

[netExtra]

OK so I did a bit of reading up on Rife today and from what I read it made sense to increase multi which I think is also known as the implementation factor. Once I did that, the double images went away from the Spider-man clip shown above as well as another movie with a similar issue. I would like to close this but I'm just wondering else what impact increasing multi will have on anything else. Besides I assume increased resource usage.

Update: It fixes the issue but when run Rife is loaded then after a few seconds the screen goes blank then returns but I don't think Rife is running anymore because the GPU usage goes down a lot.

[netExtra]

With the current Rife onnx models, the only way to fix the double images (in fast motion) is to increase the fps. At 72fps the issue is reduced a little. At 90 or more fps it disappears. With my GPU I can just about do 72fps but with some ugly stuttering on slow pans.

[netExtra]

Hello, I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors. If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live. But yeah, you could add your videos probably to here and we could if those errors still appear.

Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.

Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.

Rife.error.mov

The artefact seen in the certain types of fast movement like the one shown in Amazing Spider-man clip above has been reduced in v4.15. It's not fixed but it definitely has been improved :)

[carycary246]

Any way to reduce panning artefacts? My TV motion interpolation never has issues with straight pans like this but rife often has issues with these kind of shots.

https://github.com/hzwer/Practical-RIFE/assets/31323333/22d0d0ce-1fb6-45c9-b53c-bc1baa7c00a7

[netExtra]

@carycary246 Maybe I'm missing something but I can't see any major artefacts in that clip. But if it is there, then it looks very minor. Also which version of Rife are you using?

[carycary246]

@carycary246 Maybe I'm missing something but I can't see any major artefacts in that clip. But if it is there, then it looks very minor. Also which version of Rife are you using?

Look at the right side with the building fences. I think Rife doesn't know how to handle those kind of objects (thin lines?) even when it is a straight pan with no other motion. I am using 4.14 lite now, but all previous versions are the same. Here is another example, in motion it is distracting because it warps a large area over several frames.

https://github.com/hzwer/Practical-RIFE/assets/31323333/d45d6c04-6714-404e-b5da-62429463a732

[netExtra]

@carycary246 The artefacts are not being captured clearly in your clips, but they are clear in the still image. I don't watch anime, but in live action this type of horizontal and vertical pan straight line warping is much reduced by Rife v4.9 and mostly gone by Rife v4.13, with only a couple of minor artefacts visible if you look for them. The main issue remaining is fast movement artefacts but v4.15 is an improvement. Sorry I can't be more help.

[hiredd]

4 frame interpolation model may resolve the problem

[hexiaoyi95]

@carycary246 can you share these two original videos without interpolation?

[carycary246]

@hexiaoyi95 I've uploaded the two clips without interpolation here.

https://mega.nz/file/9ToS0KgQ#S03DLc5YPMCPnDhPmsjN-bLpb4O3MR2xQjVJpC4HNhg https://mega.nz/file/Zf5X1IAR#HesJScsyt7iBT0b4zM8Rtp7iwDXvJL-Z8Crc0Bgd0pM

[netExtra]

@hexiaoyi95 I've uploaded the two clips without interpolation here.

https://mega.nz/file/9ToS0KgQ#S03DLc5YPMCPnDhPmsjN-bLpb4O3MR2xQjVJpC4HNhg https://mega.nz/file/Zf5X1IAR#HesJScsyt7iBT0b4zM8Rtp7iwDXvJL-Z8Crc0Bgd0pM

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

[carycary246]

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

Your Name is a standard 24fps film, not sure why you wouldn't be able to play it. Panning motion wouldn't be any different from any other kind of 24 fps media.

[netExtra]

SVP can't play "Your Name" clip properly because it might be HFR?. But I was able to play the "Solo" clip and the artefacts are identical to the ones I used to see in the introduction to Blade Runner 2023 before Rife 4.9.

Your Name is a standard 24fps film, not sure why you wouldn't be able to play it. Panning motion wouldn't be any different from any other kind of 24 fps media.

SVP doesn't like it and it's default config treats it like HFR but I still got a good idea of the artefacts for both clips.

[netExtra]

Rife v4.15 was a major improvement for fast motion but Rife v4.16 is a major regression. I hope @hzwer does NOT use this as the backbone for future versions because for live action fast motion, it would be like going back to Rife 4.9 or worse.

[TNTwise]

4.16 is a lite model, so it is similar to 4.9 in terms of quality. I'm assuming he means that it will be the base for future lite models.

[netExtra]

Maybe. But 4.16 being pretty much like 4.9 for quality is going in the wrong direction. Rife 4.9 is a nice all round model but is pretty bad for live action. Rife 4.15 is the best for quality.

[TNTwise]

Well, in terms of speed, it's faster than 4.13-lite, the lite models are more for speed than quality. Keep using 4.15 for the best results.

[hzwer]

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a heavier version to see the effect.

[netExtra]

@netExtra Interesting, since recently, friends who care about 2d or 3d scenes almost always give me the opposite feedback. I plan to release a “heavy” version to see the effect.

I've checked the SVP forums and the consensus is that v4.16 lite is a good all round replacement for v4.9. But v4.15 has the least artefacts, especially for fast motion. Maybe your friends are talking about animation? I'm talking about video and movies.