search

mueslimak3r / tv-intro-detection

This project tries to detect intros of tv series by comparing pairs of episodes to find the largest common subset of frames.
https://mueslimak3r.github.io/tv-intro-detection/
GNU General Public License v3.0
81 stars 3 forks source link

GPU decoding? #27

Open Yankees4life opened 2 years ago

Yankees4life commented 2 years ago

I know you're in the middle of retooling the app a bit but I believe there should be an option to use the GPU instead of CPU to run the intro detection. Not only it'll be faster but it'll also take less resources/electricity use of the computer. I ran this app and it usually takes days even on a i7 computer to finish.

mueslimak3r commented 2 years ago

If I can find a sensible way to use GPU decoding then I'm open to it. My understanding is that with the current pipeline (ffmpeg decode frame on cpu, encode on cpu in rgb24 format, and write it to a pipe), I think that most of those operations would run on the CPU anyway. So offloading a few of the steps to the GPU may just create IO bottlenecks. For instance, if I use ffmpeg -hwaccel videotoolbox -i videopath ... to use GPU decoding on my M1 mac, the frame extraction runs at only ~17fps. But if I keep it as it is now and use software decoding, it runs at around 120fps.

Maybe to effectively utilize the GPU, the entire process of getting the frames and turning them into a matrix of RGB24 would run on the GPU. I'm not sure how to do this rn.

For now, there may be some room for improving the frame extraction speed by running ffmpeg in parallel to extract frames from more than one video concurrently. I removed this a while ago because it was running slower multithreaded than it was single threaded. But with some recent changes, it's possible that it could be added back and perform well.

Another factor to consider is hard drive read speed, which may limit the gains possible from GPU processing or concurrency with CPU processing.

Yankees4life commented 2 years ago

You can use something like

ffmpeg -hwaccel qsv -c:v h264_qsv -i videopath ....

The codec of the video would have to identified in some way (I guess through ffprobe) tho and then it would adjust accordingly (eg. -c:v hevc_qsv, -c:v vp8_qsv, etc, etc)