Why was ensemble removed for models 4.7+?

[Samhayne]

Hey. Actually I found ensemble to be very useful for fighting pattern artifacts (fences, carpet patterns, etc). Besides using higher scales.

Can the ensemble code just be re-added to IFNet_HDv3.py or is there more to it to make ensemble work again?

[banjaminicc]

I second this, ensemble was vey useful in increasing the quality of prior models. Especially for real life situations instead of cgi/anime. As of now 4.6 ensemble seems to still be more stable and warp less than 4.7+ models. Maybe that is the reason?

[hzwer]

https://drive.google.com/file/d/1o9cdJusMrws0FDUv6kC-lc4itG6x26Gq/view?usp=sharing

Sorry, because I didn’t know the use of ensemble before, so I didn’t realize it because I was lazy. This is a 4.12 version with ensemble. If there are no problems, I will merge this change into 4.7-4.11.

@banjaminicc @Samhayne

[banjaminicc]

Could you please make the drive link public? Thanks @hzwer

[netExtra]

https://drive.google.com/file/d/1o9cdJusMrws0FDUv6kC-lc4itG6x26Gq/view?usp=sharing

Sorry, because I didn’t know the use of ensemble before, so I didn’t realize it because I was lazy. This is a 4.12 version with ensemble. If there are no problems, I will merge this change into 4.7-4.11.

@banjaminicc @Samhayne

I'm not sure how the .onnx files are created but if you do make ensemble public then Rife v4.9 would make the most sense to do first because it fixes some of the issues with v4.7 and v4.8 just like Rife v4.12 fixes the issues with v4.11.

[banjaminicc]

https://drive.google.com/file/d/1o9cdJusMrws0FDUv6kC-lc4itG6x26Gq/view?usp=sharing Sorry, because I didn’t know the use of ensemble before, so I didn’t realize it because I was lazy. This is a 4.12 version with ensemble. If there are no problems, I will merge this change into 4.7-4.11. @banjaminicc @Samhayne

I'm not sure how the .onnx files are created but if you do make ensemble public then Rife v4.9 would make the most sense to do first because it fixes some of the issues with v4.7 and v4.8 just like Rife v4.12 fixes the issues with v4.11.

With my understanding of RIFE, ensemble doesn't require models to be re-exported, but simply how the model calculates flow has to be appended. And hwzer has mentioned that they will merge the changes into 4.7-4.11 if this version works. So once the link is public we could test if it works well

[hzwer]

The v4.9 and v4.12 inference codes are updated and tested.

[banjaminicc]

The v4.9 and v4.12 inference codes are updated and tested.

The v4.12 zip is unfortunately still not publicly accessible The google drive link requires permission to view

[Samhayne]

Thanks, @hzwer!

That means, if one would modify IFNet_HDv3.py for the 4.10 model, we could enable ensemble for it too?

(Would reduce mental load and parameter switching if all the 4.x models are ensemble capable)

[banjaminicc]

Thanks, @hzwer!

That means, if one would modify IFNet_HDv3.py for the 4.10 model, we could enable ensemble for it too?

(Would reduce mental load and parameter switching if all the 4.x models are ensemble capable)

Seems like ensemble does work for 4.7+ by modifying the IFNET_HDv3

[Samhayne]

Thanks for testing, @banjaminicc !

[banjaminicc]

However I am encountering a very weird issue when I adapt the implementation from 4.9 onto 4.12, where objects on screen seem to "vibrate" and the solution is simply increasing scale(My input is 1080p and I had to use scale=2) to solve this issue. Might still be best to wait when the proper implementation is public (Hopefully hwzer remembers to publicize the gdrive link haha)

[WolframRhodium]

The ensemble implementation in v4.9.1 and v4.12.1 might be wrong, because in IFNet_HDv3.py:114 and IFNet_HDv3.py:119, respectively,

fd, mask = block[i](torch.cat((warped_img0[:, :3], warped_img1[:, :3], wf0, wf1, timestep, mask), 1), flow, scale=scale_list[i])

the mask variable on the LHS shadows definition on the RHS.

As a consequence, for IFNet_HDv3.py:116 and IFNet_HDv3.py:121, respectively,

f_, m_ = block[i](torch.cat((warped_img1[:, :3], warped_img0[:, :3], wf1, wf0, 1-timestep, -mask), 1), torch.cat((flow[:, 2:4], flow[:, :2]), 1), scale=scale_list[i])

here the mask variable refers to the new definition, which should be the old one.

[banjaminicc]

The ensemble implementation in v4.9.1 and v4.12.1 might be wrong, because in IFNet_HDv3.py:114 and IFNet_HDv3.py:119, respectively,

fd, mask = block[i](torch.cat((warped_img0[:, :3], warped_img1[:, :3], wf0, wf1, timestep, mask), 1), flow, scale=scale_list[i])

the mask variable on the LHS shadows definition on the RHS.

As a consequence, for IFNet_HDv3.py:116 and IFNet_HDv3.py:121, respectively,

f_, m_ = block[i](torch.cat((warped_img1[:, :3], warped_img0[:, :3], wf1, wf0, 1-timestep, -mask), 1), torch.cat((flow[:, 2:4], flow[:, :2]), 1), scale=scale_list[i])

here the mask variable refers to the new definition, which should be the old one.

I've noticed this as well. Perhaps changing the mask variable in :114 to m0 would fix the issue 🤔?

[hzwer]

Fixed and uploaded all. Hope everything will be ok...

[Samhayne]

Thanks, @hzwer! Can you please also update / re-share 4.10?

I guess you got bug reports for it, but I’m using it and even without ensemble I found it surprisingly stable with patterns in some situations where even 4.6 and 4.0 aren’t.

[banjaminicc]

Many thanks, @hzwer Now 4.12 with ensemble finally performs better or at least as well as 4.6 with ensemble Amazing work, thanks so much!

[netExtra]

Thanks @hzwer for the releases. Very much appreciated :)

I'm not sure what is happening for @banjaminicc. But with 4.10+ (SVP+mpv with hwdec=off or *-copy), GPU usage is always between 10-20% greater than v4.9 (which uses 20% more than 4.6). I checked and double checked and the same applies to the Ensemble. v4.12 seems to have very slightly better fast movement handling but that's not enough to offset the extra 20% GPU usage for me.

[hzwer]

@netExtra If you prefer faster model, I recommend v4.12.lite. It will be 30%+ faster than 4.12 with 20% less GPU usage.

[netExtra]

@netExtra If you prefer faster model, I recommend v4.12.lite. It will be 30%+ faster than 4.12 with 20% less GPU usage.

Yeah it works great and I was just about to thank you for it :) The Ensemble lite version is just as unusable but the main version is great thanks 👍

Question: What am I loosing by using the lite engine? I ask because 4.12 is only slightly better than 4.9 "for my use case" so I worry that 4.12_lite will effectively be 4.9.

Update: I just checked and the fast movement advantages of the original 4.12 are no longer there.

[hzwer]

The report I received was that the visual performance of 4.12.lite was similar to v4.9. Theoretically it will be faster than 4.9. This is just a preliminary trial of a lightweight model. After receiving feedback, we will try to launch a lighter version.

[netExtra]

The report I received was that the visual performance of 4.12.lite was similar to v4.9. Theoretically it will be faster than 4.9. This is just a preliminary trial of a lightweight model. After receiving feedback, we will try to launch a lighter version.

This matches my personal experience. There is a minor improvement between 4.9 and v4.12 and the Lite version looses that minor advantage and is basically 4.9. Performance wise both are similar but 4.9 is still slightly better. Thanks for the updates 👍

[ghost]

@hzwer is fast enabled on 4.14?

[TNTwise]

From my knowledge, fast is always enabled as contextnet is removed.