getting scene score info

[ceewanna]

How do we get scene score from ffmpeg filter? I looked into showinfo() but I couldn't find it there.

[roninpawn]

After some quick searching, I figure you're talking about the scene-detection feature of FFmpeg when you say 'scene score?' And specifically, about outputting a float value from 0 to 1 representing the likeliness of the current frame being a new scene.

I found an old thread here that seems on topic. But the answer given directs the user toward a difficult to parse FFprobe command:

ffprobe -show_frames -of compact=p=0 -f lavfi "movie=$F,select=gt(scene\,.8)" | gsed -r 's/.pkt_pts_time=([0-9.]{8,})|./\1/' >> scenes-0.8

...which is claimed to output a list of timestamps, indicating where scene changes were detected. I honestly don't know if there's a way to output the internal floating value of FFmpeg or FFprobe's automatic scene detection feature. I'm completely unfamiliar with it, but it looks to me that invoking it at all requires you to specify a threshold -- and that the threshold dictates the output.

In other words, that's a very DEEP-cut FFmpeg feature that I know nothing about. And sadly, can't really help. Except to say that creating your own scene-detection feature with this Python class wouldn't be particularly difficult. The simplest starting point would be to take the sum value of frame 1 (every color channel of every pixel added together) and compare it against the sum value of frame 2. If the two values are significantly different from one another, then you probably switched camera angles, at least.

From there, you could build a list of timestamped sections with similar sums and try to detect scenes. In fact... now that I'm spinning on the idea, that might be a reason why you can't (if you can't) extract a running scene score from a single frame in FFmpeg. Because it probably builds a table of sum() or absdiff() values for each frame, then bakes the frame data down to timestamped sections representing hard camera cuts, and then recursively looks at the averages of those sections to match the same camera angles being cut to and away from within the same scene.

If that's about right, the process would have to do the actual scene detection after it had looked at each individual frame. And there wouldn't be any scene-score available while parsing the individual frames.

...

Boy, I hope that's what you were talking about. ;) I kind of just went off there. No really, though. That was a fun thought experiment. Thanks. And let me know if I made any sense at all.

[ceewanna]

Yes, the idea of getting a sum value of a frame and analyzing the sum to detect scene change sounds very interesting to me.

Let me share with you what I am trying to do.

I have several rtsp streams from my IP cameras. Currently I have them feeding to an rtsp server and ffmpeg taking the feeds producing 2-minute recordings. I write python codes running DVR-Scan app (based on OpenCV) through those files to detect motion. It's basically requires threshold as well. At the same time I am also exploring ffmpeg's scene score in doing similar thing.

While exploring the above, I have an idea of reading rtsp directly into python's threading through ffmpeg hoping to get the scene score to trigger capturing motion frames. Each camera having its own thread collecting frames into a queue. Once a thread sees a sustaining scene score or sum it creates/triggers a thread to process the event which collecting relevant picture frames and producing video file associated with the event.

Currently I have no idea how to use python to ride ffmpeg directly frame by frame and need to learn more on this. I only know some basic ffmpeg commands and getting results from the command as output mp4 or metadata print.

Based on my experiences so far, OpenCV via DVR Scan seems to require substantial computational load compared to ffmpeg scene score filtering. Using DVR Scan, it can produces event video files while ffmpeg scene score via command line only produces parallel video capturing smaller set of motion frames. I haven't tried any study on OpenCV but my feeling is that ffmpeg would be more simple to work with. I really want to see how it would look like if the sum is adopted for detection.