`min_scene_len` parameter isn't respected with AdaptiveDetector

ci21-cgarijo commented 1 month ago

Hello. First of all, thanks for all the work you've done in this project. I've been using PyScene for some months until now and it has been very helpful, intuitive and easy to use.

Description of the issue I'm having trouble with the parameter min_scene_len because it doesn't seem to do nothing if I change it.

I'm using the AdaptiveDetector module to differenciate shots in videos and one problem I get a lot is that when some frame surpases the threshold, normally there are some following frames also surpasing the threshold until it decays, and these additional frames trigger the scene detection. So I end up with 2 to 3 very short scenes whenever there is a scene change.

My approach to handle this was to use the min_scene_len parameter to filter this short scenes but, even with this parameter, the module generates clips with fewer frames than the specified parameter.

The code I'm running to get the scenes is this one:

scenemanager = SceneManager(stats_manager=StatsManager())
scenemanager.add_detector((AdaptiveDetector(adaptive_threshold=2, window_width=20, min_scene_len=24)))
scenemanager.detect_scenes(open_video(video))
scenes = scenemanager.get_scene_list()

Then I cut the scenes with this ffmpeg command:

for idx, scene in enumerate(scenes):
        output_name = os.path.join(output_basename, f"scene_{idx}.mp4")
        periods[idx] = {}
        start_time = scene[0].get_seconds()
        end_time = scene[1].get_seconds()
        subprocess.run(f"ffmpeg -y -i {video_name} -ss {start_time} -to {end_time} -c:v libx264 -preset medium -crf 30 -c:a copy {output_name}", shell=True)

Which writes all the clips in my project folder.

Some of the clips (especially the ones I told in shots boundaries) are very short (0.1 to 0.4 seconds) and it didn't seem to be coherent with the min_scene_len=24 parameter.

I checked the number of frames of these videos with ffprobe, obtaining this result:

ffprobe -v error -select_streams v:0 -count_frames -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 video_clip.mp4
4

So my suspicions were confirmed. I'm having clips shorter than the specified minimum scene length.

Question

Is there something am I doing the wrong way or maybe I don't understand correctly the use of this parameter? Or is this a bug I found? I didn't manage to find anything about this topic, especially with the AdaptiveDetector module.

Thank you!

Breakthrough commented 1 month ago

The code samples you posted here look correct at first glance, so this could possibly be a bug. Could you share what version you're using and any sample videos? In the latest version, the check for min_scene_len is done here:

https://github.com/Breakthrough/PySceneDetect/blob/main/scenedetect/detectors/adaptive_detector.py#L163

Consequently it should not emit any cuts until min_scene_len frames have passed. Are you able to share any clips that reproduce the issue?

I attempted to fix this in https://github.com/Breakthrough/PySceneDetect/commit/d8397bcdeea344009763ff31b7223fbcd62ab91d but perhaps that was not robust enough. If you are using the latest version, could you verify with an older one?

Thanks for the report!

Breakthrough commented 1 month ago

Looking at the code again with some fresh eyes, it seems that the check for if the minimum length was met uses the current frame number, but we emit a cut at the target. This definitely seems wrong, and we seem to be missing test cases that exercise different window sizes with AdaptiveDetector.

I'll make sure we add some for that before closing this out, and will try to reproduce with existing videos in the dataset.

Open Items:

[ ] Attempt to reproduce this in a test case using existing test material
[ ] Expand tests for AdaptiveDetector to exercise larger window sizes and values for min_scene_len

wjs018 commented 1 month ago

So, if I am understanding this issue correctly, it currently works like this:

Let's say a scene was detected at frame 100
At frame 124, the threshold is exceeded, meaning a second cut is triggered because min_scene_len = 24
However, with a window width of 20, the cut is actually marked down at frame 104

That would explain how the resulting video in the example code ended up as just 4 frames.

ci21-cgarijo commented 1 month ago

The code samples you posted here look correct at first glance, so this could possibly be a bug. Could you share what version you're using and any sample videos? In the latest version, the check for min_scene_len is done here:

https://github.com/Breakthrough/PySceneDetect/blob/main/scenedetect/detectors/adaptive_detector.py#L163

Consequently it should not emit any cuts until min_scene_len frames have passed. Are you able to share any clips that reproduce the issue?

I attempted to fix this in d8397bc but perhaps that was not robust enough. If you are using the latest version, could you verify with an older one?

Thanks for the report!

@Breakthrough When I found this issue I was using version 0.6.3, and I thought that maybe updating the version this was going to fix, so I upgraded to version 0.6.4 and still was happening.

This was the example when I was finding this problem: Graph of the adaptive score to choose the threshold. scene_0_graph

This is an extract of the video I was using https://github.com/user-attachments/assets/69a1dc62-2ab0-4c7d-9089-88d3b25f659b

Just after "federal gun trial" there is a change between shots. This change corresponds with the two peaks that are just about second 6 in the graph. In that moment, I get these following clips:

https://github.com/user-attachments/assets/d46e2ab3-6a45-4d78-9520-ad187f957500 https://github.com/user-attachments/assets/dbdec444-885b-4975-9a47-6d9cfcbff7af

For which the first one is the one I run ffprobe with, getting a length of 4 frames. Before and after that shot change seems to be working properly.

Thank you!!

ci21-cgarijo commented 1 month ago

So, if I am understanding this issue correctly, it currently works like this:

Let's say a scene was detected at frame 100

At frame 124, the threshold is exceeded, meaning a second cut is triggered because min_scene_len = 24

However, with a window width of 20, the cut is actually marked down at frame 104

That would explain how the resulting video in the example code ended up as just 4 frames.

@wjs018 Exactly

Breakthrough / PySceneDetect

`min_scene_len` parameter isn't respected with AdaptiveDetector #408

Open Items: