search

nihui / rife-ncnn-vulkan

RIFE, Real-Time Intermediate Flow Estimation for Video Frame Interpolation implemented with ncnn library
MIT License
823 stars 70 forks source link

fp16 or fp32 per channel image processing #13

Open talosh opened 3 years ago

talosh commented 3 years ago

Hi guys, I'm developing an open-source timewarp tool that runs within Autodesk Flame python subsystem and uses RIFE-2020. While looking for a faster backend then CPU-only currently avaliable in pytorch for MacOS I came across this implementation, managed to compile and test it and processing speed is promising. Thank you for the great and fast implementation!

While being rather novice amator in coding I briefly looked at the data types used and find quile a lot of assumptions for image channel data as "unsigned char". ncnn::VkMat seem to have it as 32bits, not sure about ncnn::Mat

The tasks we're working on in VFX field requires image channel data to be at least 16fp and to be processed without clamping negative values and preserving values over 1. This is currently achievable withing original RIFE pytorch implementation.

Do you think it is something that can be achieved with ncnn / vulkan, and if yes, could you please outline the areas in the code to potentially look at for such modification?

Thank you!

talosh commented 3 years ago

UPD: the data gets normalized later on padding so RIFE model works within 0-1 range After playing around a bit: ncnn::Mat::from_pixels() produce non-normalized result, float values is not within 0-1 range. Scaling it up and down produce the result that is close to original. Some clipping occures in high values though. The resulting image appears slightly brighter overall wich is not the case with original implementation

talosh commented 3 years ago

The clamping seem to happen in fusionnet. It i in the original RIFE-2020 code as well but can be easily commented out. Could you please share the process you guys using to convert the model to bin and param file as it would be very helpful in this case to get rid of clamping, thank you!