scottlamb / moonfire-nvr

Moonfire NVR, a security camera network video recorder
Other
1.25k stars 137 forks source link

poor behavior when camera has audio enabled #36

Closed scottlamb closed 5 years ago

scottlamb commented 6 years ago

There's an ffmpeg bug when a camera has audio enabled. I'll quote the troubleshooting guide below. Either we need this ffmpeg bug fixed (and the fixed version bundled with Moonfire NVR) or we need to switch RTSP libraries.

Error: pts not monotonically increasing; got 26615520 then 26539470

If your streams cut out with an error message like this one, there are a couple possibilities.

One is that your camera outputs B frames. If you believe this is the case, file a feature request; Moonfire NVR currently doesn't support B frames. You may be able to configure your camera to disable B frames in the meantime.

A more subtle problem occurs in cameras such as the Dahua Starlight series when the following is true:

Moonfire NVR currently uses the ffmpeg library to talk to the cameras. ffmpeg doesn't properly support this situation. It uses the NTP time to adjust the PTS and DTS, and thus experiences jumps forward and backward. The forward jumps cause one frame to be artificially lengthened. The backward jumps create an impossible situation which causes Moonfire NVR to abort the session and retry.

In the long term, Moonfire NVR will likely implement its own RTSP support.

In the short term, you can use either of two workarounds:

scottlamb commented 5 years ago

Here's the offending ffmpeg code, in libavformat/rtpdec.c:

    if (s->last_rtcp_ntp_time != AV_NOPTS_VALUE && s->ic->nb_streams > 1) {
        int64_t addend;
        int delta_timestamp;

        /* compute pts from timestamp with received ntp_time */
        delta_timestamp = timestamp - s->last_rtcp_timestamp;
        /* convert to the PTS timebase */
        addend = av_rescale(s->last_rtcp_ntp_time - s->first_rtcp_ntp_time,
                            s->st->time_base.den,
                            (uint64_t) s->st->time_base.num << 32);
        pkt->pts = s->range_start_offset + s->rtcp_ts_offset + addend +
                   delta_timestamp;
        return;
    }
jlpoolen commented 4 years ago

For others reading this issue, commit 091217b addressed the issue by isolating the video stream only thereby ignoring the stream marked "audio":

open_options.set(c_str!("allowed_media_types"), c_str!("video")).unwrap();

Further references:

FFmpeg Trac Ticket 5018 incorrect PTS/DTS on first frame of RTSP stream opened 11/20/2015 (as of 6/8/2020) Analyzed by developer: no

See also: FFmpeg Trac Ticket 6415 Strange DTS of first packet in RTSP stream

scottlamb commented 4 years ago

Note the ffmpeg ticket 5018 is a separate problem. Since it affects only the first frame, my workaround (in src/stream.rs) is to just wait for the second key frame. It's annoying that it takes an extra second or two to open a stream but it's not a big problem.

        if discard_first {
            info!("Discarding the first packet to work around https://trac.ffmpeg.org/ticket/5018");
            stream.get_next()?;
        }

The discontinuity on SNTP adjustment is a bigger problem, not yet reported to ffmpeg folks; it's a deal-breaker for audio.

jlpoolen commented 4 years ago

I'll review both issues and see if I can propose to you, Scott Lamb, a draft for submission to FFmpeg's Trac system. Can't do until this weekend.

jlpoolen commented 4 years ago

It may be with the current high watermark of FFmpeg that the problem in FFmpeg Trac Ticket 5018 is less egregious. See my update: Comment #4

I decided it is appropriate to create a reproducible test with the least few variables and therefore sought a solution where: 1) an MP4 file is served by a locally controlled RTSP server. After spending a lot of time evaluating various RTSP server alternatives, I decided that project at https://github.com/aler9/rtsp-simple-server works well; they have downloadable binaries so you do not have to install a Go compiler. Here's what I established as a testing pattern.

Select a publicly available MP4 so everyone can use the same file.

3 Consoles.

Console A: launch the rts-simple-server

 ./rtsp-simple-server

Console B: stream with ffmpeg the file to the rtsp-simple-server

  ffmpeg -re -stream_loop -1 -i row.mp4 -c copy -f rtsp rtsp://localhost:8554/mystream

Console C: pull the stream and run ffmpeg in debug mode. Preserve the color mode using "script"

Create a log file name for this test and launch script:

   export TEST_LOG="ffmpeg_test_$(date +"%Y_%m_%d_%I_%M_%p").log"
   script $TEST_LOG

then in the script shell:

  ffmpeg  -loglevel debug -rtsp_transport tcp -rtsp_flags prefer_tcp -t 240 -i rtsp://localhost:8554/mystream -map 0:v -filter:v showinfo -frames:v 3200   -f null /dev/nul
  exit

then review the color-preserved log in the main shell:

   cat -n $TEST_LOG | less
   cat -n $TEST_LOG | grep error

The concern I have with the proposed test in Ffmpeg Trac Ticket 5018 (https://trac.ffmpeg.org/ticket/5018) is that the rtsp server at wowzaec2demo.streamlock.net is an unknown quantity and may be a source of error.
The URL used in Ticket 5018 is an rtsp stream served up from the file BigBuckBunny_115k.mov. It looks like the file "BigBuckBunny_115k.mov" is an alias created on the wowzaec2demo.streamlock.net.
The closest I could find of the video files is at http://bbb3d.renderfarming.net/download.html, specifically http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_30fps_normal.mp4 which I created a link to as "bunny.mp4". When I tried using FFmpeg to send bunny.mp4 to my new rtsp server, FFmpeg errored out:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bunny.mp4': Metadata: major_brand : isom minor_version : 1 compatible_brands: isomavc1 creation_time : 2013-12-16T17:44:39.000000Z title : Big Buck Bunny, Sunflower version artist : Blender Foundation 2008, Janus Bager Kristensen 2013 comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net genre : Animation composer : Sacha Goedegebure Duration: 00:10:34.53, start: 0.000000, bitrate: 3481 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 2998 kb/s, 30 fps, 30 tbr, 30k tbn, 60 tbc (default) Metadata: creation_time : 2013-12-16T17:44:39.000000Z handler_name : GPAC ISO Video Handler Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default) Metadata: creation_time : 2013-12-16T17:44:42.000000Z handler_name : GPAC ISO Audio Handler Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default) Metadata: creation_time : 2013-12-16T17:44:42.000000Z handler_name : GPAC ISO Audio Handler Side data: audio service type: main [rtp @ 0x561e2e20c6c0] Unsupported codec ac3 Could not write header for output file #0 (incorrect codec parameters ?): Operation not permitted Stream mapping: Stream #0:0 -> #0:0 (copy) Stream #0:2 -> #0:1 (copy) Last message repeated 1 times

The ac3 codec was a proprietary codec developed by Dolby for Surroundsound.
Wikipedia tells us the patents finally expired 2017, but I suspect Gentoo carefully excluded it from its build of FFmpeg in order respect the patent. Any rate, when I got into this ac3 rabbit hole, I decided to step back and see if the initial test Scott submitted to FFmpeg 5 years ago still was a problem. It doesn't look like it is; Scott may want to try again with a newer version of FFmpeg to see if he can duplicate the results. Scott has recently indicated that this problem may no longer be a showstopper for audio, so I'll let my contribution above be my last to this Issue. I do like my testing methodology that preserves the colorized output.

jlpoolen commented 4 years ago

I also should add, I tried using FFmpeg against my Reolink cameras several weeks ago and duplicated the "monotonic" errors. Seeing the errors motivated me to help work on this issue. My version of FFmpeg has been upgraded since then, upgrades are a common thing when running Gentoo Linux, and today's attempt (7/3/2020) did not produce any "monotonic" errors.

jlpoole@taurus /tmp/test $ ffmpeg -loglevel debug -rtsp_transport tcp -rtsp_flags prefer_tcp -t 20 -i rtsp://admin:[REDACTED]@192.168.1.48:554/h264Preview_01_main -map 0:v -filter:v showinfo -frames:v 1200 -f null /dev/null ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 8.3.0 (Gentoo 8.3.0-r1 p1.1) configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.3/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --nm=x86_64-pc-linux-gnu-nm --ranlib=x86_64-pc-linux-gnu-ranlib --optflags='-march=native -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-optimizations --disable-libcelt --disable-indev=alsa --disable-indev=oss --disable-indev=jack --disable-outdev=alsa --disable-outdev=oss --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --enable-gnutls --disable-gmp --enable-gpl --disable-hardcoded-tables --enable-iconv --disable-libtls --disable-libxml2 --disable-lzma --enable-network --disable-opencl --disable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl2 --disable-vaapi --disable-vdpau --disable-vulkan --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --disable-opengl --disable-libv4l2 --enable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --enable-libdav1d --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-libaribb24 --disable-mmal --disable-libmodplug --enable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --disable-librsvg --disable-ffnvcodec --disable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-cuda-llvm --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-libtesseract --disable-lv2 --disable-libfreetype --disable-libvidstab --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --disable-libmp3lame --disable-libkvazaar --enable-libaom --enable-libopenh264 --disable-libsnappy --disable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --enable-libx264 --enable-libx265 --disable-libxvid --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-vfpv3 --disable-armv8 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-vsx --disable-power8 --disable-amd3dnow --disable-amd3dnowext --disable-fma4 --disable-xop --cpu=host --disable-doc --disable-htmlpages --enable-manpages libavutil 56. 51.100 / 56. 51.100 libavcodec 58. 91.100 / 58. 91.100 jlpoole@taurus /tmp/test $ cat -n $TEST_LOG|grep mono 121 Stream #0:1, 18, 1/16000: Audio: aac (LC), 16000 Hz, mono, fltp jlpoole@taurus /tmp/test $ cat -n $TEST_LOG|grep error 2013 603 frames successfully decoded, 0 decoding errors jlpoole@taurus /tmp/test $