Scenedetect fails on mp4 video with multiple audio tracks

[tslater]

Bug/Issue Description: I get fatal errors using scenedetect with a multiaudio stream mp4. When I used ffmpeg to copy to an mkv with the same streams, or when I copy to an mp4 with only one audio stream, it works. It will not work with multiple streams in an mp4.

Required Information: Provide the following information to assist with reporting the bug:

1. Provide a full copy of the command line options you are using, for example:

scenedetect --input /Volumes/sources/sources/video/movie/chapter_5/landscape.mp4  --output /Volumes/sources/sources/movie/movie/chapter_5/scene_detect  --stats stats.csv detect-content list-scenes save-images            

attach the generated BUG_REPORT.txt file.

INFO: __init__.scenedetect_cli(): Version: v0.5.3
INFO: __init__.scenedetect_cli(): Info Level: debug
INFO: __init__.scenedetect_cli(): Output directory set:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect
DEBUG: context.parse_options(): Parsing program options.
DEBUG: context._init_video_manager(): Initializing VideoManager.
INFO: video_manager.__init__(): Loaded 1 video, framerate: 23.98 FPS, resolution: 886 x 480
INFO: video_manager.set_downscale_factor(): Downscale factor set to 3, effective resolution: 295 x 160
DEBUG: context.parse_options(): VideoManager initialized.
DEBUG: __init__.detect_content_command(): Detecting content, parameters:
  threshold: 30, min-scene-len: 14
INFO: context.list_scenes_command(): Scene list CSV file name format:
  $VIDEO_NAME-Scenes.csv
INFO: context.save_images_command(): Image output format set: JPEG
DEBUG: context.process_input(): Processing input...
INFO: context.process_input(): Detecting scenes...
INFO: context.process_input(): Processed 0 frames in 0.0 seconds (average 0.00 FPS).
DEBUG: context.process_input(): No frame metrics updated, skipping update of the stats file.
INFO: context.process_input(): Detected 1 scenes, average shot length 0.0 seconds.
INFO: context.process_input(): Writing scene list to CSV file:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect/landscape-Scenes.csv
INFO: context.process_input(): Scene List:
-----------------------------------------------------------------------
 | Scene # | Start Frame |  Start Time  |  End Frame  |   End Time   |
-----------------------------------------------------------------------
 |      1  |           0 | 00:00:00.000 |           0 | 00:00:00.000 |
-----------------------------------------------------------------------

INFO: video_manager.set_downscale_factor(): Downscale factor set to 1, effective resolution: 886 x 480
INFO: context._generate_images(): Generating output images (3 per scene)...
DEBUG: context.cleanup(): Cleaning up...

INFO: __init__.scenedetect_cli(): Version: v0.5.3
INFO: __init__.scenedetect_cli(): Info Level: debug
INFO: __init__.scenedetect_cli(): Output directory set:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect
DEBUG: context.parse_options(): Parsing program options.
DEBUG: context._init_video_manager(): Initializing VideoManager.
INFO: video_manager.__init__(): Loaded 1 video, framerate: 23.98 FPS, resolution: 886 x 480
INFO: video_manager.set_downscale_factor(): Downscale factor set to 3, effective resolution: 295 x 160
DEBUG: context.parse_options(): VideoManager initialized.
DEBUG: __init__.detect_content_command(): Detecting content, parameters:
  threshold: 30, min-scene-len: 14
INFO: context.list_scenes_command(): Scene list CSV file name format:
  $VIDEO_NAME-Scenes.csv
INFO: context.save_images_command(): Image output format set: JPEG
DEBUG: context.process_input(): Processing input...
INFO: context.process_input(): Detecting scenes...
INFO: context.process_input(): Processed 0 frames in 0.0 seconds (average 0.00 FPS).
DEBUG: context.process_input(): No frame metrics updated, skipping update of the stats file.
INFO: context.process_input(): Detected 1 scenes, average shot length 0.0 seconds.
INFO: context.process_input(): Writing scene list to CSV file:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect/landscape-Scenes.csv
INFO: context.process_input(): Scene List:
-----------------------------------------------------------------------
 | Scene # | Start Frame |  Start Time  |  End Frame  |   End Time   |
-----------------------------------------------------------------------
 |      1  |           0 | 00:00:00.000 |           0 | 00:00:00.000 |
-----------------------------------------------------------------------

INFO: video_manager.set_downscale_factor(): Downscale factor set to 1, effective resolution: 886 x 480
INFO: context._generate_images(): Generating output images (3 per scene)...
DEBUG: context.cleanup(): Cleaning up...

Expected Behavior: Expect the process to run with output and no error

Computing Environment: Dockerfile

FROM python:3.8-slim

RUN apt-get update && apt-get install -y \
  libglib2.0-0 \
  libsm6 \
  libxext6 \
  libxrender-dev \
  libgl1-mesa-glx \
  && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir scenedetect[opencv,progress_bar]==0.5.3

Additional Information: Just my gratefulness for making this!

[tslater]

Here's the error omitted when I run without BUG_REPORT.txt:

[PySceneDetect] Downscale factor set to 1, effective resolution: 886 x 480
[PySceneDetect] Generating output images (3 per scene)...

  0%|          | 0/3 [00:00<?, ?images/s]Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 792, in main
    ctx.exit()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 558, in exit
    raise Exit(code)
click.exceptions.Exit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/scenedetect", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/scenedetect/__main__.py", line 58, in main
    cli.main(obj=cli_ctx)   # Parse CLI arguments with registered callbacks.
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 792, in main
    ctx.exit()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 404, in __exit__
    self.close()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 494, in close
    cb()
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 395, in process_input
    self._generate_images(scene_list=scene_list, video_name=video_name,
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 200, in _generate_images
    timecode_list = [
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 201, in <listcomp>
    [
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 202, in <listcomp>
    FrameTimecode(int(f), fps=fps) for f in [
  File "/usr/local/lib/python3.8/site-packages/scenedetect/frame_timecode.py", line 115, in __init__
    self.frame_num = self._parse_timecode_number(timecode)
  File "/usr/local/lib/python3.8/site-packages/scenedetect/frame_timecode.py", line 251, in _parse_timecode_number
    raise ValueError('Timecode frame number must be positive and greater than zero.')
ValueError: Timecode frame number must be positive and greater than zero.

  0%|          | 0/3 [00:00<?, ?images/s]
    at ChildProcess.exithandler (child_process.js:308:12)
    at ChildProcess.emit (events.js:314:20)
    at ChildProcess.EventEmitter.emit (domain.js:486:12)
    at maybeClose (internal/child_process.js:1051:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

[Breakthrough]

Hi @tslater;

Are you running into this issue with all files with multiple audio tracks, or just a few? I'll try to replicate this on my end and keep you posted (may need a small sample if I'm unable to reproduce on my end). Looks like you might've found a couple bugs here, thanks for the report.

One of my concerns is I'm not sure at first glance how I would go about detecting this condition at runtime, without relying on another tool like ffprobe, but will definitely do some more investigation on my end and see if I can come up with a resolution. I haven't heard any issues like this before with OpenCV + video files, so hoping this might just be a one-off issue.

Thanks for the report!

[tslater]

You're very welcome. I'm seeing that it mkv videos with multiple audio tracks are working, but not mp4. Hopefully that helps.

[tslater]

I can send an offending video file, but privately because the video files I'm working with are all copyrighted. I have permission to work with the film, but distribution rights are currently in negotiation status. Just let me know an email I can send it to if that helps!

[Breakthrough]

@tslater If you could that would be great. Send me a message via the form on http://bcastell.com/contact/ and I'll get in touch with you. Thanks!

[tslater]

I think your contact form is broken? I get a 400 JS error.

[tslater]

[Redacted]

[Breakthrough]

Thanks for letting me know! Something is definitely up with my site, haven't updated it in a while, will have to do that. Have requested access for now, thank you for the link (have removed it from your comment to prevent spam).

On Fri, Aug 21, 2020 at 1:04 PM Tyler Sl8r notifications@github.com wrote:

Here's a link. It's restricted, but if you request access I can grant:

https://drive.google.com/file/d/1Ty8SxhkiwcmyhKZJElMjef7EiubKYgt9/view?usp=sharing

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Breakthrough/PySceneDetect/issues/179#issuecomment-678393940, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA6TBHRJ6BDTK4MDIJ6BBLSB2SK3ANCNFSM4QAQHF7A .

[tslater]

Thanks! I approved your request. You're the best!

On Sat, Aug 22, 2020 at 9:53 AM Brandon Castellano notifications@github.com wrote:

Thanks for letting me know! Something is definitely up with my site, haven't updated it in a while, will have to do that. Have requested access for now, thank you for the link (have removed it from your comment to prevent spam).

On Fri, Aug 21, 2020 at 1:04 PM Tyler Sl8r notifications@github.com wrote:

Here's a link. It's restricted, but if you request access I can grant:

https://drive.google.com/file/d/1Ty8SxhkiwcmyhKZJElMjef7EiubKYgt9/view?usp=sharing

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/Breakthrough/PySceneDetect/issues/179#issuecomment-678393940 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAA6TBHRJ6BDTK4MDIJ6BBLSB2SK3ANCNFSM4QAQHF7A

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Breakthrough/PySceneDetect/issues/179#issuecomment-678657058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADNWLSI7ZSANSLVOMJSNM3SB7SV3ANCNFSM4QAQHF7A .

[Breakthrough]

Awesome, many thanks for the sample. Now here's where things are going to get a bit tricky unfortunately - I was able to run the video successfully, as well as split it on my end. However, when playing the video back using VLC, I've noticed a few inconsistencies when playing it back on my end (it reports video length as ~15 minutes, and after I hit "play", the timestamp jumps to 30 seconds)...

So I'm wondering if it might be due to how the files are being generated/encoded (this isn't common, but has cropped up occasionally for some users). However, it's strange that PySceneDetect seems to work fine on my end (in particular, my OpenCV/ffmpeg libraries/binaries).

Could you provide some more details about the OpenCV package you're using, as well as particular OS distribution / machine? The more details you can provide the better. I'll try the same environment in a Linux VM to see if I can replicate it on my end, but the closer I can get to your setup, the better it would be to help track down the issue here.

Likewise, thank you for the material & feedback. In the meantime, is re-encoding your source material or using MKV as an input container a viable workaround for you? Just want to make sure you're not blocked by this bug while we figure out what's going on here.

---

Console output I get:

[PySceneDetect] Processed 253 frames in 0.1 seconds (average 3011.25 FPS).
[PySceneDetect] Detected 5 scenes, average shot length 2.1 seconds.
[PySceneDetect] Writing scene list to CSV file:
  C:\Users\Pietro\Documents\git-repos\PySceneDetect\blah\test-Scenes.csv
[PySceneDetect] Scene List:
-----------------------------------------------------------------------
 | Scene # | Start Frame |  Start Time  |  End Frame  |   End Time   |
-----------------------------------------------------------------------
 |      1  |           0 | 00:00:00.000 |          68 | 00:00:02.836 |
 |      2  |          68 | 00:00:02.836 |         173 | 00:00:07.216 |
 |      3  |         173 | 00:00:07.216 |         193 | 00:00:08.050 |
 |      4  |         193 | 00:00:08.050 |         224 | 00:00:09.343 |
 |      5  |         224 | 00:00:09.343 |         253 | 00:00:10.552 |
-----------------------------------------------------------------------

Also this is a perfect scene for testing a new feature I plan on rolling out in #35 to suppress flashes that cause false positives - I was looking for something like this for a long time (where flashlights "strobe" across the frame). I'll do my due diligence and obtain a copy legally on my end so I can extract this short segment and possibly use it as a test case for that feature (the short clip you provided me will only be used by me to triaging this specific issue, and will never be distributed as agreed). Love this movie, btw :)

[tslater]

Yeah the timing is off in the clip I sent you. I actually cut it down from a longer clip that was ~15 minutes because my upload speed is horrendous and the original clip is ~500MB or so...but the timing is accurate on that one. That is interesting that it is working for you. I am actually using docker (for consistency), so if you are able to use that, it should be easy to replicate. Here is my Dockerfile:

FROM python:3.8-slim

RUN apt-get update && apt-get install -y \
  libglib2.0-0 \
  libsm6 \
  libxext6 \
  libxrender-dev \
  libgl1-mesa-glx \
  && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir scenedetect[opencv,progress_bar]==0.5.3

[Breakthrough]

Just to confirm, you're experiencing the same issue with that cut sample you sent me? (Smaller is better for issues like this, most definitely - just hoping it's not due to the way you cut the file, so want to confirm that first)

I'm not really familiar with docker so might need to have a few colleagues get me up to speed with that. In the meantime I'll try to replicate this under a Linux environment. I was under the impression that the operating system Docker was running on is important - do you have that information by any chance? (If not, no worries - I still have a lot of triaging to do with this one on my end still, just trying to grasp for any additional details before getting into too deep of a rabbit hole haha).

[tslater]

Yeah, the issue exists in the parent clip and the one I sent you. I confirmed before I sent it. It might be something unique to the docker setup. I'm using Docker on Ubuntu 18.04, but that's really the point of docker is that the underlying operating system should make little to no difference. At this point, maybe it would be easier for me to start getting into PySceneDetect source code and do my own debugging than make you learn docker though...Unless you're interested.

[Breakthrough]

That definitely helps! Will setup an Ubuntu VM on my end and let you know if I can replicate. The issue here (from what I can see in the debug logs you provided at least) is that the underlying OpenCV/ffmpeg implementation doesn't seem to give any errors for PySceneDetect to report - it seems to fail silently, with 0 frames processed. I definitely need to add some better handling of that situation so I can report that something went wrong, but am uncertain if I can tell the user why specifically (other than just that the video is incompatible, and to try re-encoding/re-muxing/transcoding it).

If you're up for it, you can try to just use the OpenCV library from Python directly to open the video file using the VideoCapture class and try to read() a few frames from it. As mentioned though, I'll setup a VM to try and replicate this on my end, so no rush (but anything else you find in the meantime would definitely be helpful!). Just working on a few other issues for the v0.5.4 release, after which this will be the highest priority issue.

Even if I can't find a root cause for why this is happening, better error handling is most definitely required on my end here so I can report a better error message when this situation occurs (which has been occasionally reported in the past). Hopefully I can sneak that in for v0.5.4 as well. All the best, will keep you posted - and thank you again for providing a test case & the required info!

[tslater]

Thanks so much. If I have time later next week, and I haven't heard back, I might do some poking with OpenCV

[Breakthrough]

Hey @tslater;

Should have some improved messaging for this error now in v0.5.4, so this should be in the next release. Long term would require switching libraries to handle video I/O which isn't really feasible right now unfortunately. I did update the FAQ with two methods to get around this issue using ffmpeg or mkvmerge via the following commands:

ffmpeg -i input.mp4 -c copy -an output.mp4

mkvmerge -o output.mkv input.mp4

Thanks again for the report!

[Breakthrough]

This may actually be possible to fix once multiple video backends are supported as per #213 - that is still a way's away, but just wanted to state that for completeness.

Edit: Have re-opening this and moving to the v1.0 release, as I expect supporting PyAV should resolve the issue.

[Breakthrough]

Interestingly enough, I'm now unable to reproduce this issue with the latest version of Python/OpenCV. If anyone runs into this issue, try updating to the latest version and see if that resolves the issue - and if not, please post an update here!

[Breakthrough]

Some users are still running into issues with this - this should be resolved in v0.6 if you install PyAV (#213). An error message will also be displayed when processing a video fails indicating to attempt using the PyAV backend as an alternative.

The v0.6 beta is out now which has this fix in place.