search

Breakthrough / PySceneDetect

:movie_camera: Python and OpenCV-based scene cut/transition detection program & library.
https://www.scenedetect.com/
BSD 3-Clause "New" or "Revised" License
3.11k stars 385 forks source link

Color histogram method #53

Open r1b opened 6 years ago

r1b commented 6 years ago

I noticed that you have a stub for a HistogramDetector. I have completed a proof-of-concept using this method in this notebook.

Do you think this is a sound approach? I don't have a lot of experience with this kind of problem so I am curious to hear your thoughts. I would love to integrate this into pyscenedetect.

Breakthrough commented 6 years ago

Very interesting approach. Have you given some thought to benchmarking it versus Albanie's shot detection benchmarks to see how it performs?

It would be quite interesting to have some additional detection techniques in PySceneDetect for sure. I haven't had much time to keep up with the project lately, but I'm always open to pull requests!

Breakthrough commented 6 years ago

Also @r1b, this definitely does seem to be a sound approach - there is plenty of literature referencing this method.

I will have to do some more research into your work, but definitely plan to get started on something like this in the future. I have added this to the backlog of issues, indicating it will be worked on for the next major release of PySceneDetect following v0.5 (the project is almost done a heavy refactor which should make developing detectors much easier).

Also just curious, what is your development environment like? I'm not too familiar with Python notebooks, but some of the analysis (esp. with regards to the graphs) would be immensely useful. Just curious what you're using, if it's an interactive environment.

r1b commented 6 years ago

Wonderful! When I have some free time I will play with it some more. I am using Jupyter as a development environment to take advantage of the inline plotting & exploratory workflow. Usually I use matplotlib for plots but here I use Seaborn mostly just to use distplot (but I also think they look nicer).

Breakthrough commented 6 years ago

Thank you for the references @r1b , will definitely be checking out Jupyter. Thank you also for the proof of concept, had some time to look into it briefly and will be following up on this after v0.5 is released. One thing I'm experimenting with is automatic threshold detection for the content/threshold detectors, and had a few high-level questions if you don't mind me picking your brain.

I noticed in your notebook that you're using the equation for threshold T = μ + ασ, where μ is the mean, σ is the standard deviation, and α is a constant set to 5.

Just curious, how did you arrive at a value of α = 5? Does it need to be adjusted for different types of source material?

For the histogram itself, have you experimented at all with changing the number of frequency bins in the histograms, or generating the colour code in HSV space instead of RGB? Just trying to flesh out the design by trying to get a feel for what options need to be presented to the user when calling detect-histogram, and what the exact equations/algorithms need to be in place.

I'm trying to use the same equation to automatically generate T for the ContentDetector algorithm, but I can't seem to find a good value for α that works across a good number of source material - specifically when dealing with shorter clips. I was also wondering if you had any ideas for approaching computing a good value for α (or if another equation would be better in this case) for the ContentDetector, which computes a single HSV-based frame score.

Also, feel to check out the upcoming API if you have a minute, any feedback you might have is most welcome.

ash2703 commented 4 months ago

What is the status of this issue, is it still open for help?

Also was any benchmarking done with opencv calcHist method after resizing?

Breakthrough commented 4 months ago

@ash2703 Thanks for the bump, feel free to take a look at @wjs018's PR for this: #295. I've not had the chance to go through it much yet, but he did some benchmarking there as well. Feel free to leave a comment on that PR, I would be happy to keep things moving along.

If you're keen to get involved, there's a lot that still remains: figuring out how to manage detection threshold parameters, improving performance, tests, documentation, etc.

Breakthrough commented 4 months ago

I spent some time going over that PR and merged it into the develop branch just now if you'd like to have a try at it. Thanks @wjs018 for taking that on. One issue right now is we need to normalize frame scores, since they change drastically if you modify the downscale factor. That should be a pretty easy fix, already added some TODOs in the code where we might want to tackle this. @ash2703 feel free to provide any input.

The following are the remaining open items before closing this out:

ash2703 commented 4 months ago

@Breakthrough I had some optimizations in mind, do we have some sort of golden dataset for me to benchmark not just the performance but say the accuracy of the algo as well.

ash2703 commented 4 months ago

Hi @Breakthrough Have added some optimizations and can pick up the TODOs as well Can you assign this to me?

Breakthrough commented 4 months ago

Done, that's awesome to hear. I'll have a more thorough look at your PR over the weekend, would like to see what @wjs018 thinks as well, but love where things are going with it!

There are some accuracy tests in tests/test_detectors.py. Right now there's just two for HistogramDetector, but feel free to expand upon them. If there's any other relevant test cases you think should be added, feel free to link them as well. Of course we will need to have permission to include them in the repo, or at least make sure they fall under fair use guidelines. Test videos are in the resources branch.