Open mmokrejs opened 6 months ago
If you are able, can you test running the video with another related project as suggested in the linked issue? It entails running PySceneDetect on the failing video with the following commands:
scenedetect -i video.mp4 --verbosity debug --backend opencv detect-content
scenedetect -i video.mp4 --verbosity debug --backend pyav detect-content
I would like to know if using a different library to handle video IO would resolve the situation, or at least be able to return an error that can be acted on. Will also try this once I have a sample, sent you an email just now. I might be missing some debug logging from PySceneDetect so will look into making sure that gets logged in DVR-Scan.
With OpenCV we try to read at least 5 more frames after a failure to decode (see https://github.com/Breakthrough/PySceneDetect/blob/main/scenedetect/backends/opencv.py#L282). Unfortunately there is no error that gets returned from OpenCV in the case of a decode failure.
Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.
As long as you are not seeking in the input, you should be able to tell how many frames were processed from the log output:
[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).
Thank you for the detailed report!
Here I ran the command on just the problematic 51MB file (unlike merged HEVC file for whole day):
$ scenedetect -i input.3gp --verbosity debug --backend opencv detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
DEBUG: __init__.open_video(): Opening video with opencv...
DEBUG: context._open_video_stream(): Video opened using backend VideoStreamCv2
DEBUG: context.handle_options(): Initializing SceneManager.
DEBUG: __init__.detect_content_command(): Adding detector: ContentDetector({'weights': Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), 'kernel_size': None, 'luma_only': False, 'min_scene_len': 10, 'threshold': 27.0})
DEBUG: controller.run_scenedetect(): Running controller.
INFO: scene_manager.detect_scenes(): Downscale factor set to 10, effective resolution: 256 x 192
INFO: scene_manager.detect_scenes(): Detecting scenes...
Detected: 0 | Progress: 37%|█████████████████████████████████████████████████████▍ | 295/801 [00:02<00:04, 111.83frames/s][hevc @ 0x5641b3d58d80] Duplicate POC in a sequence: 13.
DEBUG: opencv.read(): Frame failed to decode.
Detected: 0 | Progress: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 800/801 [00:07<00:00, 103.77frames/s]
INFO: controller._detect(): Processed 800 frames in 7.7 seconds (average 103.76 FPS).
INFO: controller.run_scenedetect(): Detected 1 scenes, average shot length 46.9 seconds.
$ scenedetect -i input.3gp --verbosity debug --backend pyav detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
Usage: scenedetect [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Try 'scenedetect -h' for help.
Error: Invalid value for -b/--backend: Specified backend pyav is not available on this system!
$
Hmm, I have dvr-scan
and scenedetect from pip
. I have opencv on Gentoo system-wide.
OK, I installed https://pypi.org/project/av/ into same environemnt like dvr-scan and scenedetect.
$ pip install av
Collecting av
Downloading av-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.6 kB)
Downloading av-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.7/34.7 MB 5.9 MB/s eta 0:00:00
Installing collected packages: av
Successfully installed av-12.0.0
$
Now I have:
$ scenedetect -i input.3gp --verbosity debug --backend pyav detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
DEBUG: __init__.open_video(): Opening video with pyav...
DEBUG: pyav.__init__(): Restoring default ffmpeg log callbacks.
DEBUG: pyav.__init__(): Threading mode set: AUTO
DEBUG: context._open_video_stream(): Video opened using backend VideoStreamAv
DEBUG: context.handle_options(): Initializing SceneManager.
DEBUG: __init__.detect_content_command(): Adding detector: ContentDetector({'weights': Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), 'kernel_size': None, 'luma_only': False, 'min_scene_len': 15, 'threshold': 27.0})
DEBUG: controller.run_scenedetect(): Running controller.
INFO: scene_manager.detect_scenes(): Downscale factor set to 10, effective resolution: 256 x 192
INFO: scene_manager.detect_scenes(): Detecting scenes...
Detected: 0 | Progress: 0%| | 0/801 [00:00<?, ?frames/s][swscaler @ 0x7f631cd76f80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
Detected: 0 | Progress: 0%|▏ | 1/801 [00:00<01:45, 7.56frames/s][swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdd2840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
Detected: 0 | Progress: 1%|██▌ | 12/801 [00:00<00:13, 59.30frames/s][swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
Detected: 0 | Progress: 2%|███▉ | 19/801 [00:00<00:13, 59.98frames/s][swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
...
[swscaler @ 0x7f631ce4b380] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce93940] deprecated pixel format used, make sure you did set range correctly
Detected: 0 | Progress: 37%|████████████████████████████████████████████████████████████▋ | 293/801 [00:02<00:04, 116.16frames/s][swscaler @ 0x7f631cdbd080] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cef3040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce76ac0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cea1980] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce84b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ced4600] deprecated pixel format used, make sure you did set range correctly
[hevc @ 0x7f636ced2480] Duplicate POC in a sequence: 13.
[hevc @ 0x7f636ced2480] Error parsing NAL unit #0.
[swscaler @ 0x7f631ce84b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cee3b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce0c700] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce68a80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce1a1c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
Detected: 0 | Progress: 38%|███████████████████████████████████████████████████████████████▍ | 306/801 [00:02<00:04, 113.30frames/s][swscaler @ 0x7f631cebff00] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce935c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdede40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
CRITICAL: scene_manager._decode_thread(): Fatal error: Exception raised in decode thread.
Detected: 0 | Progress: 39%|█████████████████████████████████████████████████████████████████ | 314/801 [00:02<00:04, 105.20frames/s]
Traceback (most recent call last):
File "/home/mmokrejs/virtualenv/dvr-scan/bin/scenedetect", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/__main__.py", line 47, in main
run_scenedetect(cli_ctx)
File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/_cli/controller.py", line 58, in run_scenedetect
scene_list, cut_list = _detect(context)
^^^^^^^^^^^^^^^^
File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/_cli/controller.py", line 101, in _detect
num_frames = context.scene_manager.detect_scenes(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/scene_manager.py", line 895, in detect_scenes
raise self._exception_info[1].with_traceback(self._exception_info[2])
File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/scene_manager.py", line 918, in _decode_thread
frame_im = video.read()
^^^^^^^^^^^^
File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/backends/pyav.py", line 279, in read
self._frame = next(self._container.decode(video=0))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "av/container/input.pyx", line 208, in decode
File "av/packet.pyx", line 80, in av.packet.Packet.decode
File "av/video/stream.pyx", line 35, in av.video.stream.VideoStream.decode
File "av/video/stream.pyx", line 44, in av.video.stream.VideoStream.decode
File "av/codec/context.pyx", line 507, in av.codec.context.CodecContext.decode
File "av/codec/context.pyx", line 404, in av.codec.context.CodecContext._send_packet_and_recv
File "av/error.pyx", line 328, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input
$
Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.
As long as you are not seeking in the input, you should be able to tell how many frames were processed from the log output:
[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).
So how many total frames were in the input file, I mean how many ffmpeg would find otherwise regardless of eventual failure)?
Or show column with percentages of the total input?
So how many total frames were in the input file, I mean how many ffmpeg would find otherwise regardless of eventual failure)?
Or show column with percentages of the total input?
There's no easy way to show the expected total, some containers don't report it accurately and it is often incorrect. Using other libraries like PyAV allows ensuring that you have exhausted the full stream, but using OpenCV like this library currently does, it's not possible to do things like that. I think PyAV allows capturing those warnings, although I recall running into some deadlocks when doing that... I was recently made aware of some progress on the issue at PyAV-Org/PyAV#751 though so it probably is worth looking into to see if it's possible.
In the meantime this is similar enough to #118 and has been a huge help in the debugging process. I'll mark this as a duplicate of that, but feel free to continue the discussion there.
Thank you for your efforts so far. I tend to believe the [ffmpeg/video] hevc: Error parsing NAL unit #0
is not always criticial and maybe the testcase cannt be processed due to the Error while decoding frame (hardware decoding)!
instead. I see multiple [ffmpeg/video] hevc: Error parsing NAL unit #0
messages in a row while playng the movies.
I have several more testcases of mpv
unplayable HEVC files (and causing dvr-scan
to skip the remainder), but with different errors:
attempt 1
[ffmpeg] Failed to reallocate parser buffer to -2147482773
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] NULL: missing picture in access unit with size 579234816
(Buffering) V: 00:00:38 / unknown (71%) Dropped: 262
attempt 2
[ffmpeg] Failed to reallocate parser buffer to -2147483292
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] NULL: missing picture in access unit with size 1418189824
(Buffering) V: 00:00:24 / unknown (71%) Dropped: 129
https://trac.ffmpeg.org/ticket/8732 https://github.com/mpv-player/mpv/issues/5893 https://github.com/mpv-player/mpv/issues/5755
My CPU is Intel i-13900K
chip so I use Intel QSV for hardware speedup (i915 kernel module).
My assumption is that all of this happens due to flakey ONVIF cameras and a switch dropping packets. Because of that, sometimes h264
video playing slows down or even stops, possibly while scanning for next full frame. Probably some combination of long series of events triggers a bug somewhere. It seems I will really need to re-encode the original 3gp
files into different stream (h265
) to ensure the incomplete frames are discarded. This also means the FPS (originally 15 or 25 fp) will appear variable frame rate (VFR
) instead of fixed because many of the frames were randomly lost along the way to NVR. But then I will run into a different issue with dvr-scan as it does not support variable frame rate.
I think the design could be completely changed. A recording should be analyzed, regions of repeated movement (trees blowing in the wind) detected as glossy regions and their median color (a car may show up in front of them later so do not ask this region completely). In another sweep dvr-scan should look for motions in other places, even in a single frame. Further, dvr-scan detects some kernel size during it run. Provided I merge 3gp
per 24-hrs into a single HEVC
container (no re-encoding so far) all the videos start in night, in b/w color. I do not know what this kernel size is but certainly dvr-scan should split the video into ranges. This would be the very first pass of reading the file. Notably, the recordings at about the switch of IR cur filter are more grainy with distorted color and should be treated as a separate. Comparing individual/adjacent frames with each other does not make sense. The scene is largely static, sources of errors (small windblow movements can be learned) and then the real scene changes be detected more easily, even if recorded in a single frame.
Manual does not describe expectations so this is how I evolved my scripts to handle thousands of files per day and what I think is meaningful. Merging all videos per day into a single file is asking for a trouble but technically is the only way for me not to get lost in the zillions of files. It already sucks enough that ffmpeg -i list-of-files.txt ... -o merged.hevc
dies due to the moov atom not found
issue which means I have to edit the list-of-files.txt
file and drop from it the problematic file and re-run ffmpeg
.
It seems I will really need to re-encode the original 3gp files into different stream (h265) to ensure the incomplete frames are discarded. This also means the FPS (originally 15 or 25 fp) will appear variable frame rate (VFR) instead of fixed because many of the frames were randomly lost along the way to NVR. But then I will run into a different issue with dvr-scan as it does not support variable frame rate.
The video output should be correct, as long as you are using the default OpenCV output mode. VFR video is only an issue for the accuracy of timestamps, which is why the ffmpeg ouptut mode won't work with it currently. That being said, there should be a filter for ffmpeg that will allow you to convert the VFR video into a fixed framerate one - you may want to consider that as a workaround.
A recording should be analyzed, regions of repeated movement (trees blowing in the wind) detected as glossy regions and their median color (a car may show up in front of them later so do not ask this region completely). In another sweep dvr-scan should look for motions in other places, even in a single frame
This sounds possible and like a good iea, but am lacking a lot of test videos which makes it difficult to develop these kinds of features. That being said, you may have some luck experimenting with the kernel size setting, which should help reduce small movements due to trees for example, and using the region editor to mask out problematic areas that aren't of interest.
@Breakthrough Did you get my emails with links to the testcases causing dvr-scan
to stop processing input files?
@mmokrejs sorry for the delay, thank you for the samples I did receive them. I'll leave this open for now and re-word it as an investigation into improved logging on frame corruption. I haven't had any time to experiment with the videos you sent yet, but will let you know if I can come up with any improvements. In the meantime happy to accept any PRs if folks have similar fixes.
Many thanks for the test material.
Bug/Issue Description:
In bug https://github.com/Breakthrough/DVR-Scan/issues/118 somebody else reported that dvr-scan silently stops processing its input. I have several HEVC containers manifestig same issue with
dvr-scan
v1.6
frompip
. The HEVC file was created from many short3gp
files byffmpeg
.I ran the same command now with
--verbosity debug
, there is little bit more info.I tracked it back to the original
3gp
file (51MB).mpv
stalls in the problematic position with:Maybe similar underlying issue is in https://github.com/iina/iina/issues/3448 (no testcase)?
Meanwhile you can drop me an email so I can share some example files with you.
Would be nice if the
List of motion events:
table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.Expected Behavior: A clear error message if dvr-scan stopped reading the input, ideally reporting
ffmpeg
command to extract some data around the problematic position in the input stream (for copy&pasting).Computing Environment:
but also with: