Faster QuickSR models?

[arianaa30]

Hi, I see multiple quicsr TFLite models available in the project. The default one is 540p which on GPU, currently takes ~200ms on my phone. Is there a faster one? Any of these maybe for 360p?

[Monaco12138]

You may be able to find the model you are looking for in this https://github.com/Monaco12138/AndroidFFmpegVideoPlayer. Here are some models from 270p to 540p. If you want to customize a model, you can use the methods in this repository to train and export your own model. For inference speed, you can try different proxy modes. We provide two methods: GPU proxy and NNAPI proxy. On mobile phones, NNAPI proxy is usually faster. Good Luck!

[arianaa30]

Thanks. Yeah there was a 270p model and a 540p model. No 360p though. And for the proxy, hmm on my Samsung S23 GPU delegate is much faster than NNAPI. There is no way to make it work in real-time (<35ms) right? So to get 30fps.

[Monaco12138]

Here is my measurement results at that time. Please note that in the first repository, the super-resolution images will be rendered to the camera for output (this process is not multi-threaded), resulting in a slower final time. In this repository, we have optimized the process by using multiple threads for streaming, preprocessing, and model inference. By running this repository, you can see real-time playback of rendering results from 540p to 1080p. Thank you for your attention to this work. If you find this code is useful for your research, please cite it and star it. 🤗

[arianaa30]

That's interesting. Thanks for sharing. In my usecase, I read from a stream and need to apply SR in realtime. Is this repo enough or I should use your new repo? Btw do you think converting model to shader would be much faster than TFLite?

[Monaco12138]

The latest repository is a video player that integrates super-resolution enhancement, using ffmpeg for streaming (RTSP), to enhance and render the resulting video stream, which can help you quickly get started. The SR method built using shader usually has fewer parameters, and the speed depends on your technical implementation and hardware support. You may need to use OpenGL to do this, which I have not tried yet.

[arianaa30]

Got it. In my case, the player uses surface and I have to apply SR to the surface. Is this the case with your video player too?

And btw, would int8 be generally faster? Your table doesn't have it

[Monaco12138]

The SR result is transformed to Bitmap, you can show it in ImageView or surfaceHolder. For int8 inference, it is usually faster than float type, but due to the small number of parameters in our model, their speed is not much different in my testing. You can directly use float type.