search

n00mkrad / flowframes

Flowframes Windows GUI for video interpolation using DAIN (NCNN) or RIFE (CUDA/NCNN)
GNU General Public License v3.0
1.54k stars 117 forks source link

FlowFrames 1.41-1.42-1.43 Changelog #275

Closed jensdraht1999 closed 3 months ago

jensdraht1999 commented 1 year ago

FlowFrames 1.41 Changelog

jensdraht1999 commented 1 year ago

FlowFrames Version 1.41

Please put this folder in the "FlowframesData" Folder, It will put the new rife.py, version.ini and natsort library which is needed into the right places. Normally this should work.

I just fear, it might not work because of following function, which we might have to comment out for now:

ShowTotalPerformance().

Because on non Nvidia GPU, this might indeed give an error, but there is not way to check this out unfornuately.

jensdraht1999 commented 1 year ago

FlowframesData

Please rename GIF to 7z and then extract it.

jensdraht1999 commented 1 year ago

This here: FlowframesData

https://github.com/n00mkrad/flowframes/assets/99474666/843a5bc3-8920-47f5-8213-99f7778d0718)

jensdraht1999 commented 1 year ago

Resetting does not work very well with interpolated frames, need to look into this.

jensdraht1999 commented 1 year ago

@n00mkrad Can you merge?

jensdraht1999 commented 1 year ago

@n00mkrad ????

jensdraht1999 commented 1 year ago

@n00mkrad Can you merge?

FelixKainz commented 1 year ago

@n00mkrad it would be cool to at least respond to @jensdraht1999. He obviously put a lot of effort into this. @jensdraht1999 thanks for your contributions!

Astalavista1655 commented 1 year ago

@n00mkrad Can you merge?

Is it now showing a conflict with the branch, or have they finally started merging your code?

jensdraht1999 commented 1 year ago

@Astalavista1655 This will not work anyway. He has ignored me, not even mentionion me. I left it with him.

n00mkrad commented 11 months ago

I will look into it once I finish the messy VFR changes

n00mkrad commented 11 months ago

I do have to look into the blending which appears to only work with PNG frames right now. Does not apply to RIFE-NCNN-VS though.

jensdraht1999 commented 11 months ago
  • RIFE NCNN tends to be faster than CUDA so I haven't looked into any of the CUDA related changes
  • QSV for JPEGs is obsolete since I am now using TIFF for proper RGB/YUV support
  • Writing the log only every 10s is a terrible idea, would make it hard if not impossible to debug crashes
  • I don't understand your issue about the output directory dragndrop, works fine here
  • I adjusted the temp folder settings and made %LOCALAPPDATA%\Temp\Flowframes\ the default setting f1a91a4
  • WEBP frame output was already supported, not sure why you added it somewhere separately
  • Stuff like CPU Temperature or disk write speed is out of scope for this program

I do have to look into the blending which appears to only work with PNG frames right now. Does not apply to RIFE-NCNN-VS though.

1.) Cuda is faster than Rife NCNN, but because of broken logger function, it seems the other way around. It actually is fast, but it gets slower, because flowframes log function is broken. Please debug in visual studio and look which function consumes everything. And this has been proven by all people here: https://github.com/n00mkrad/flowframes/issues/160

2.) QSV and jpeg are two different things. QSV decoder will extract images with jpeg faster from h264 videos faster than pure cpu image extraction.

2.5.) Jpeg interpolated frames are supported and faster too, because a lot of cpu cycles are consumed png creation. However creating jpeg files for interpolation is much much faster. Flowframes would if run for about 30 minutes, consume 15-20% of the cpu will all these things combined, however, after fixing logging function, using temp directory, using jpeg interpolated frames, it would not only not slowdown, it would even be much faster in the beginning.

3.) Please drag and drop to the place in the image and see what happens if you try to interpolate a video. Spoiler alert. It will not work:

!t

4.) WebP interpolated frames are supported.

5.) Stuff like cpu/gpu/disk support are good for people who want to know and see directly if there are problems and are totally okay to add as function.

6.) Bleding with anything other than png has been also fixed by me. You have to play with imagemagick library.

For you not to build everything, I will upload the exe file here so you can test out yourself.

https://anonymfile.com/d7qj/flowframes.7z

There are a few improvments too like chaning w-threads and whatnot. Please check this thread:

https://github.com/n00mkrad/flowframes/issues/160

PS:

  1. Only thing that can make it faster than this is using PytorchTRT, which does not work on Windows
  2. Please use Bing Chat GTP for programming, it helped me a lot for fixing those issues.
jensdraht1999 commented 11 months ago

All people that tried cuda after my release say that the whole video takes only half of the time. Please look at the threads.

jensdraht1999 commented 11 months ago
  • Writing the log only every 10s is a terrible idea, would make it hard if not impossible to debug crashes

I do not think, because if the program is gonna crash anyway, it would have not written it to the file either. For this purpose we will have to use the visual studio in debug mode to see, where it crashes.

n00mkrad commented 11 months ago

Use RIFE-NCNN-VS.

CUDA with frame extraction and writing is a dead end.

I benchmarked it on a 4090 and NCNN is still a bit faster.

NCNN VS: 16s total processing time

CUDA pre-patch: 60s total processing time

CUDA with your patch: 24s total processing time

jensdraht1999 commented 11 months ago

Use RIFE-NCNN-VS.

CUDA with frame extraction and writing is a dead end.

I benchmarked it on a 4090 and NCNN is still a bit faster.

NCNN VS: 16s total processing time

CUDA pre-patch: 60s total processing time

CUDA with your patch: 24s total processing time

Can you send me the test video? I would like to test it out.

I also need info: FP16? On or off Interpolated frames JPG or PNG? Rife version?

I have 1:16 min with NCNN/VS/Vapoursynth NCNN did not work. I have 0:18 min with Nvidia RIFE CUDA (with everything all together)

My settings: FP16 ON, Interpolated Frames JPG, Rife version 4.6, MP4 h264 (cpu).

The sample file: https://file-examples.com/wp-content/storage/2017/04/file_example_MP4_480_1_5MG.mp4

jensdraht1999 commented 11 months ago

1

Here are some more options I used.

Astalavista1655 commented 11 months ago

I’ll break the dialogue a little here, will there be an update to RIFE 4.13? In the meantime, I switched to SVP, Flowframes are still more familiar

jensdraht1999 commented 11 months ago

Hello another test with longer video,

Top 10 Best Anime Fights of Spring 2023 (1080p_24fps_H264-128kbit_AAC)

Video ID : 1 Format : AVC Format/Info : Advanced Video Codec Format profile : High@L4 Format settings : CABAC / 3 Ref Frames Format settings, CABAC : Yes Format settings, Reference : 3 frames Codec ID : avc1 Codec ID/Info : Advanced Video Coding Duration : 11 min 55 s Bit rate : 2 623 kb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 24.000 FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.053 Stream size : 224 MiB (95%) Title : ISO Media file produced by Google Inc. Writing library : x264 core 155 r2901 7d0ff22 Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 Codec configuration box : avcC

Audio ID : 2 Format : AAC LC Format/Info : Advanced Audio Codec Low Complexity Codec ID : mp4a-40-2 Duration : 11 min 56 s Bit rate mode : Constant Bit rate : 128 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 44.1 kHz Frame rate : 43.066 FPS (1024 SPF) Compression mode : Lossy Stream size : 10.9 MiB (5%) Title : ISO Media file produced by Google Inc. Language : English Default : Yes Alternate group : 1

with 3x interpolation in both cases and waited for both of them 5 minutes too cool down:

Rife Cuda: Done running RIFE - Interpolation took 26:23. Peak Output FPS: 39,70 - Waiting for encoding to finish... [ffmpeg] Frame: 1100 FPS: 64 QP: 25.0 Size: 24576kB Time: 00:00:15.25 Bitrate: 13201.8kbits/s Relative Speed: 0.88x Merging videos... [ffmpeg] Frame: 28474 FPS: 0.0 Size: 631552kB Time: 00:06:35.48 Bitrate: 13081.9kbits/s Relative Speed: 791x Deleting temporary files... Total processing time: 28:07

VRAM USED: 0,71 GB

NCNN VS: Video FPS: 24/1 (~24) - Total Number Of Frames: 17183 Input Resolution: 1920x1080 Running RIFE (NCNN-VS)... Interpolated 0/51547 Frames (0%) - Average Speed: 0,00 FPS In / 0,00 FPS Out - Time: 0ms - ETA: 08s Canceled interpolation. Running RIFE (NCNN-VS)... Interpolated 0/51547 Frames (0%) - Average Speed: 0,00 FPS In / 0,00 FPS Out - Time: 46:38 - ETA: 08s Done running RIFE - Interpolation took 32:18. Peak Output FPS: 0,00 Deleting temporary files... Total processing time: 32:24

VRAM USED: 3.46 GB

It's about 4 minutes faster than NCNN-VS. But I think Cuda could be much faster, if we somehow could directly process the video to video, so the cpu would not be bottlenecked by JPEG/PNG creation.

By the way, I already use Pytorch 2.0, which has reduced VRAM usage and still is fast it was. However one of the best option release with Pytorch 2.0. is Pytorch compile, which would increase the performance, but is sadly not available for Windows.

jensdraht1999 commented 11 months ago

Running RIFE (CUDA)... Interpolated 51547/51547 Frames (100%) - Average Speed: 10,34 FPS In / 31,01 FPS Out - Time: 27:43 - ETA: 00s - Encoding... Done running RIFE - Interpolation took 27:43. Peak Output FPS: 38,84 - Waiting for encoding to finish... [ffmpeg] Frame: 1095 FPS: 71 QP: 14.0 Size: 64512kB Time: 00:00:15.16 Bitrate: 34845.0kbits/s Relative Speed: 0.984x Merging videos... [ffmpeg] Frame: 43555 FPS: 21775 Size: 2610176kB Time: 00:10:04.98 Bitrate: 35343.9kbits/s Relative Speed: 302x Deleting temporary files... Total processing time: 29:25

This time I used NVENC Encoder too. The VRAM usage would go 1.01 GB when it was encoding and then going back to 0,71 GB.

n00mkrad commented 10 months ago

I’ll break the dialogue a little here, will there be an update to RIFE 4.13? In the meantime, I switched to SVP, Flowframes are still more familiar

4.13 is already working in the repo version, yes

n00mkrad commented 10 months ago

I will keep these things in mind but for now I prioritize the VapourSynth based implementations since they are a lot more flexible.

As soon as 1.41 is out I will look at some of the changes here.

jensdraht1999 commented 10 months ago

I will keep these things in mind but for now I prioritize the VapourSynth based implementations since they are a lot more flexible.

As soon as 1.41 is out I will look at some of the changes here.

@n00mkrad I also try my best to somehow manage to get PytorchTRT Version working with direct video to video, but I cannot promise anything. Let's see, if PytorchTRT will be ready at that time.

Astalavista1655 commented 10 months ago

I will keep these things in mind but for now I prioritize the VapourSynth based implementations since they are a lot more flexible.

As soon as 1.41 is out I will look at some of the changes here.

Why is it impossible to set a custom coefficient for RIFE 2.3? Is this a feature of the model?

ghost commented 7 months ago

Hello,

I followed the guide here https://github.com/n00mkrad/flowframes/issues/160#issuecomment-1776423203 and compiled without no errors.

Here is the beginning of the story...

GPU core use and GPU VRAM use are increasing but the progress does not show how many frames and the process looks like it's progressing, but it's not.

Can anyone help?

Some SS here:

00

01

jensdraht1999 commented 7 months ago

VRAM use are increasing but the progress does not show how many frames and the process looks like it's progressing, but it's not.

The changes I made are only for Nvidia Cuda Rife, not for the other ones. However, if I would be you, please put it into another folder, which does not contain special turkish chars. Then try again, perhaps it works.

There was this PR, which suppossed to correct this behavoir, but I could not test it out: https://github.com/n00mkrad/flowframes/pull/184

Trufel-Obiknovenniy commented 3 months ago

Здравствуйте, как скачать готовые файлы программы? я не очень понимаю что здесь за файлы и коды

Egor179 commented 3 months ago

Здравствуйте, как скачать готовые файлы программы? я не очень понимаю что здесь за файлы и коды

Никак, здесь исходный код, который нужно самому компилировать, либо покупать на патреоне, но вообще здешняя модель RIFE сильно не актуальна

jensdraht1999 commented 3 months ago

@Egor179 If you can compile this program yourself, then it might work, however cannot provide the executable, since the author of this programs earns his money this program. I think you can download the free version and just use RIFE Vulkan version. This is good enough and easy to understand for most people..

Egor179 commented 3 months ago

@Egor179 If you can compile this program yourself, then it might work, however cannot provide the executable, since the author of this programs earns his money this program. I think you can download the free version and just use RIFE Vulkan version. This is good enough and easy to understand for most people..

Wait, no, you can either buy an off-the-shelf version 1.40 I think, or you can compile and the executable will be compiled that way. I already compiled a long time ago, including your version of the code. Or has something changed now, I haven't used Flowframes for over a year now

jensdraht1999 commented 3 months ago

@Egor179 If you can compile this program yourself, then it might work, however cannot provide the executable, since the author of this programs earns his money this program. I think you can download the free version and just use RIFE Vulkan version. This is good enough and easy to understand for most people..

Wait, no, you can either buy an off-the-shelf version 1.40 I think, or you can compile and the executable will be compiled that way. I already compiled a long time ago, including your version of the code. Or has something changed now, I haven't used Flowframes for over a year now

You can compile it, but I am not sure, if the models are supported in this way. Because for the models there are some kind of checksum chcecks, just putting them into the folder won't work. You could compile the exe and then try it with 1.36 or the paid 1.40 version.

jensdraht1999 commented 3 months ago

Also I'm closing this PR, because this is not going anywhere.

FelixKainz commented 3 months ago

@n00mkrad, can you please get @jensdraht1999 some aknowledgement here? You said you'll check this PR out many months ago. Jens made some nice additions here. He deserves a bit of attention IMO.

jensdraht1999 commented 3 months ago

@FelixKainz I don't think FlowFrames is that important anymore. Don't get me wrong, it's very good at what it is doing. But this is just a theory of me (not confirmed in any way, shape or form):

NVIDIA will probably release (only for 40er series and up) RTX Video Interpolation as an option RTX Video Super Resolution. So people will be upscaling and frame interpolating in real time. To be honest, I can't understand, why NVIDIA didn't do it until this day. Probably waiting for 50er series of GPUS.