search

scottlamb / moonfire-nvr

Moonfire NVR, a security camera network video recorder
Other
1.22k stars 137 forks source link

One of Four Cameras Is Failing #114

Closed jlpoolen closed 3 years ago

jlpoolen commented 3 years ago

One of four cameras seems to go into a repeated loop stopping and being unable to accept streams and record same. See screenshot below of the web UI where garage_west has an abnormal number of small files

Running moonfire-nvr updated 3/10/2021. Have run previous version, including circa 6/1/2020 where I had several months of operation without this kind of problem.

Desired result: equal recording of camera "garage_west" as three other cameras and web interface offering files of similar size as other cameras

moonfire-nvr@raspberrypi:/usr/local/src/moonfire-nvr/server/target/release $ git describe v0.6.0-40-ge66a88a moonfire-nvr@raspberrypi:/usr/local/src/moonfire-nvr/server/target/release $ ./moonfire-nvr --version moonfire-nvr 0.6.1 moonfire-nvr@raspberrypi:/usr/local/src/moonfire-nvr/server/target/release $

Ran with RUST_BACKTRACE=1

log ( 254MB; feel free to select snippets and post herein as I'll remove this large file after 1 week) for approximately 4 hour session this afternoon at:

https://drive.google.com/file/d/1Ir0SZ3jlsiLO5IiyMELQnX4YkJp_Akxk/view?usp=sharing

Screenshot showing the Web interface and the garage_west files constantly restarting Screenshot_2021-03-10_1728_ Moonfire_NVR

As a check, I have the Reolink client software running and capturing events from garage_west and the three other cameras and there has been no noticeable difference for garage_west's camera. I therefore think something on the receiving side, e.g. moonfire-nvr on my Raspberry Pi4 is causing the problem.

scottlamb commented 3 years ago

Well, looks like the panic in #113 is no longer happening, so that's an improvement at least.

Do you still have any logs from the older, working installation? I'd like to see the lines after Initialized ffmpeg. Versions: if so.

It's not happy with garage_west or peck_west from the beginning. Is there anything different between these cameras and the others? Different model, firmware, etc?

I see several kinds of errors here:

W20210310 13:06:38.690 s-garage_west-main moonfire_ffmpeg::rtsp] 0xaf300950: CSeq 8 expected, 0 received.

W20210310 13:17:09.060 s-garage_west-main moonfire_ffmpeg::rtsp] 0xaf300950: RTP H.264 NAL unit type 29
W20210310 13:17:09.060 s-garage_west-main moonfire_ffmpeg::rtsp] 0xaf300950:  is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

W20210310 14:00:48.748 s-garage_west-main moonfire_nvr::streamer] garage_west-main: sleeping for Duration { secs: 1, nanos: 0 } after error: pts not monotonically increasing; got 26140140 then 26120517

E20210310 14:22:37.305 s-garage_west-main moonfire_ffmpeg::rtsp] 0xac5ac8d0: RTP: PT=60: bad cseq f304 expected=3882

E20210310 14:35:12.457 s-garage_west-main moonfire_ffmpeg::h264] 0xac6598f0: Invalid NAL unit 8, skipping.

E20210310 14:22:37.361 s-garage_west-main moonfire_ffmpeg::h264] 0xac518460: error while decoding MB 22 47, bytestream -9
I20210310 14:22:37.373 s-garage_west-main moonfire_ffmpeg::h264] 0xac518460: concealing 6907 DC, 6907 AC, 6907 MV errors in I frame

The pts not monotonically increasing is the direct cause of the disconnects, but I wonder if the other errors are due to the same root cause.

I wonder if you can reproduce with the ffmpeg commandline utility, without involving Moonfire NVR at all. Knowing that would narrow the problem down anyway. Maybe via a command like this:

$ ffmpeg -report -loglevel debug -fflags nobuffer -rtsp_transport tcp -allowed_media_types video -i 'rtsp://admin:YOUR_PASSWORD_HERE@192.168.1.48:554/h264Preview_01_main' -c copy -f null -

That command will just open a stream and discard the received bytes until it gets dropped or you hit q. Note you'll have to manually redact your password from the logfile.

If that shows problems, it'd be interesting to see if dropping the -fflags nobuffer argument fixes it. Moonfire NVR started setting this option recently, in 75dce88.

We could also try getting a packet capture at the same time as a log; it'd be relatively easy then to see if ffmpeg is right about the CSeq being incorrect. But it'd be even more data to deal with and harder to redact the password in so maybe we'll hold onto that idea for later.

scottlamb commented 3 years ago

Most of these complaints happen to a lesser extent on the other cameras too.

$ fgrep 'CSeq' logs.txt | cut -d' ' -f3 | sort | uniq -c
   1016 s-garage_east-main
  18265 s-garage_west-main
    716 s-peck_east-main
    243 s-peck_west-main

$ fgrep 'H.264 NAL unit' logs.txt | cut -d' ' -f3 | sort | uniq -c
      2 s-garage_west-main

$ fgrep 'sleeping for Duration' logs.txt | cut -d' ' -f3 | sort | uniq -c
      1 s-garage_east-main
    164 s-garage_west-main
     10 s-peck_east-main
      3 s-peck_west-main

$ fgrep 'bad cseq' logs.txt | cut -d' ' -f3 | sort | uniq -c
1986452 s-garage_west-main
   2895 s-peck_east-main

$ fgrep 'Invalid NAL' logs.txt | cut -d' ' -f3 | sort | uniq -c
     17 s-garage_west-main

$ fgrep 'error while decoding MB' logs.txt | cut -d' ' -f3 | sort | uniq -c
   1795 s-garage_west-main
     58 s-peck_east-main

$ fgrep 'concealing' logs.txt | cut -d' ' -f3 | sort | uniq -c
   5070 s-garage_west-main
    125 s-peck_east-main
scottlamb commented 3 years ago

Further breakdown of the sleeping for Duration lines:

$ fgrep 'pts not monotonically increasing' sleeps.txt | cut -d' ' -f3 | sort | uniq -c
      1 s-garage_east-main
    110 s-garage_west-main
      4 s-peck_east-main
      3 s-peck_west-main
$ fgrep 'excessive pts jump' sleeps.txt | cut -d' ' -f3 | sort | uniq -c
      8 s-garage_west-main
      4 s-peck_east-main
$ fgrep 'exceeds maximum' sleeps.txt | cut -d' ' -f3 | sort | uniq -c
     28 s-garage_west-main
      2 s-peck_east-main
$ fgrep 'Connection refused' sleeps.txt | cut -d' ' -f3 | sort | uniq -c
     17 s-garage_west-main
$ fgrep 'Connection timed out' sleeps.txt | cut -d' ' -f3 | sort | uniq -c
      1 s-garage_west-main
jlpoolen commented 3 years ago

I'll see if I have some old logs... I doubt it, though. Also, I did upgrade the Raspberry Pi system, so I would think that I'm using a later version of ffmpeg. It looks like an upgrade of ffmpeg may be the culprit here. Nonetheless, I try to behave on the Raspberry Pi 4 server as if I were a consumer because that is the prospective user you may have for this project (if you are going to continue to suggest Raspberry Pi as a viable platform) and if they run into trouble doing an upgrade on the system, then it's good to flush the issue out with me. It's going to take some time which I'll try to devote over the weekend.

scottlamb commented 3 years ago

Good luck!

fwiw, I think a "consumer" would be more likely to follow the recommended path of using the Docker image, which comes with a fixed version of the ffmpeg libraries. But I don't know for sure yet if that would avoid this problem, or if it's a problem with ffmpeg at all. It's probably not the whole story; I have a pi4 with those same versions and that ffmpeg command doesn't seem to be having the same problems with my cameras (in the first few minutes at least).

scottlamb commented 3 years ago

Hmm, in issue #84 you pasted a log including this:

I0710 190739.619 main moonfire_ffmpeg] Initialized ffmpeg. Versions:
avutil: running=56.22.100 compiled=56.22.100
avcodec: running=58.35.100 compiled=58.35.100
avformat: running=58.20.100 compiled=58.20.100

which is the same as the one from this bug:

I20210310 13:04:59.019 main moonfire_ffmpeg] Initialized ffmpeg. Versions:
avutil: running=56.22.100 compiled=56.22.100
avcodec: running=58.35.100 compiled=58.35.100
avformat: running=58.20.100 compiled=58.20.100

There could be some subtle difference not captured in the version number but it seems unlikely.

jlpoolen commented 3 years ago

I found a log file that has: I0527 050128.310 main moonfire_nvr::cmds::run] Database is loaded. I0527 050128.373 main moonfire_nvr::cmds::run] Directories are opened. I0527 050128.373 main moonfire_nvr::cmds::run] Resolved timezone: America/Los_Angeles

W0527 050128.373 main moonfire_nvr::web] UI directory file "jquery-ui.bundle.js.gz" has unknown extension; skipping
W0527 050128.373 main moonfire_nvr::web] UI directory file "vendor.bundle.js.gz" has unknown extension; skipping
W0527 050128.373 main moonfire_nvr::web] UI directory file "jquery.bundle.js.gz" has unknown extension; skipping
W0527 050128.373 main moonfire_nvr::web] UI directory file "nvr.bundle.js.gz" has unknown extension; skipping
I0527 050128.374 main moonfire_ffmpeg] Initialized ffmpeg. Versions:
avutil: running=56.31.100 compiled=56.31.100
avcodec: running=58.54.100 compiled=58.54.100
avformat: running=58.29.100 compiled=58.29.100

Moreover, I cannot say for certain that in June or July I was using the same ffmpeg, but here's what is showing my Rasbertty Pi 4 now. I am not familiar with the Raspberry Pi deployment of binaries to know whether the July 8th date is the date the binary was installed on my machine, or an artificial date create by a some package manager.

jlpoole@raspberrypi:~ $ which ffmpeg
/usr/bin/ffmpeg
jlpoole@raspberrypi:~ $ ls -la /usr/bin/ffmpeg
-rwxr-xr-x 1 root root 243304 Jul  8  2020 /usr/bin/ffmpeg
jlpoole@raspberrypi:~ $ffmpeg -version
ffmpeg version 4.1.6-1~deb10u1+rpt1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 8 (Raspbian 8.3.0-6+rpi1)
configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --libdir=/usr/lib/arm-linux-gnueabihf --cpu=arm1176jzf-s --arch=arm
libavutil      56. 22.100 / 56. 22.100
libavcodec     58. 35.100 / 58. 35.100
libavformat    58. 20.100 / 58. 20.100
libavdevice    58.  5.100 / 58.  5.100
libavfilter     7. 40.101 /  7. 40.101
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  3.100 /  5.  3.100
libswresample   3.  3.100 /  3.  3.100
libpostproc    55.  3.100 / 55.  3.100
jlpoole@raspberrypi:~ $

I think what I need to do is just isolate the ffmpeg command for camera garage_west in its own session and see if I can recreate the error reports.

jlpoolen commented 3 years ago

So I tried the following:

A) 4 consoles on Rapsberry Pi

I connect to the Rapsberry Pi and with screen attempted to create four windows, each being a session using the following formula:

 ffmpeg -loglevel debug -fflags nobuffer -rtsp_transport tcp -i 'rtsp://admin:[redacted]@192.168.1.49:554/h264Preview_01_main' -c copy -f null -

As can happen with screen, I became lost in its maze after trying to create a four panel window and unsuccessfully trying to give each a title. (Will still try with tmux another time.)

So I connected to the Raspberry Pi with four ssh consoles via Putty on Windows and ran ffmpeg against each camera. After about an hour, one then a second camera started to go into an infinite loop state with messages such as:

 frame=442457 fps= 12 q=-1.0 size=N/A time=10:01:36.15 bitrate=N/A speed=   1x

Again, I cannot say if my ffmpeg on the Raspberry Pi 4 was altered when I performed a system upgrade in the last few months. But I suspected my disparity of results, e.g. June - January vs. recent few weeks, was being caused at the ffmpeg level.

B) Parallel Console On Different Servers & Different ffmpeg

I then wanted to compare my Gentoo VM version of ffmpeg with the Rapsberry Pi high watermark. I let the two consoles run overnight and 8 hours later found the Gentoo one in the endless loop "frame=...." above; the Raspberry Pi only had a handful of errors.

Gentoo ffmpeg

ares /home/jlpoole # ffmpeg -version
ffmpeg version 4.3.2 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 9.3.0 (Gentoo 9.3.0-r2 p4)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.3.2/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --nm=x86_64-pc-linux-gnu-nm --ranlib=x86_64-pc-linux-gnu-ranlib --pkg-config=x86_64-pc-linux-gnu-pkg-config --optflags='-march=native -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-optimizations --disable-libcelt --enable-version3 --disable-indev=alsa --disable-indev=oss --disable-indev=jack --disable-indev=sndio --disable-outdev=alsa --disable-outdev=oss --disable-outdev=sndio --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --enable-gnutls --disable-gmp --enable-gpl --disable-hardcoded-tables --enable-iconv --disable-libtls --disable-libxml2 --disable-lzma --enable-network --disable-opencl --disable-openssl --enable-postproc --enable-libsmbclient --disable-ffplay --disable-sdl2 --disable-vaapi --disable-vdpau --disable-vulkan --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --disable-libv4l2 --enable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --enable-libdav1d --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-libaribb24 --disable-mmal --disable-libmodplug --disable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-cuda-llvm --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-libtesseract --disable-lv2 --enable-libfreetype --disable-libvidstab --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --disable-libaom --disable-libopenh264 --disable-librav1e --disable-libsnappy --disable-libtheora --disable-libtwolame --disable-libwebp --enable-libx264 --disable-libx265 --disable-libxvid --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-vfpv3 --disable-armv8 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-vsx --disable-power8 --disable-amd3dnow --disable-amd3dnowext --disable-aesni --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-sse3 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-xop --cpu=host --disable-doc --disable-htmlpages --enable-manpages
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100
ares /home/jlpoole #

Raspberry Pi4 ffmpeg 

jlpoole@raspberrypi:~ $ ffmpeg -version
ffmpeg version 4.1.6-1~deb10u1+rpt1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 8 (Raspbian 8.3.0-6+rpi1)
configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --libdir=/usr/lib/arm-linux-gnueabihf --cpu=arm1176jzf-s --arch=arm
libavutil      56. 22.100 / 56. 22.100
libavcodec     58. 35.100 / 58. 35.100
libavformat    58. 20.100 / 58. 20.100
libavdevice    58.  5.100 / 58.  5.100
libavfilter     7. 40.101 /  7. 40.101
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  3.100 /  5.  3.100
libswresample   3.  3.100 /  3.  3.100
libpostproc    55.  3.100 / 55.  3.100
jlpoole@raspberrypi:~ $

Note, that the Gentoo [server: ares] version does not have the library warning -- one of the benefits of compiling everything in Gentoo, methinks. It is unclear from Google searches what the risk of the library warning presents.

My next step is to "unmask" a more recent version of ffmpeg on my Gentoo VM and see what happens with that.

This issue is moving away from Moonfire-nvr and looking to be an ffmpeg issue.

Further thoughts: I suppose I should try to save what comes in via ffmeg's rtsp call to learn if the repeated message of "frame=442457 fps= 12 q=-1.0 size..." means nothing and affects what is processed, or whether it is white noise. I'm going to have to learn more about ffmpeg which is a monstrous tome, nay operating system guised as a program.

scottlamb commented 3 years ago

the Raspberry Pi only had a handful of errors.

What were the errors? I don't see them here. I'm particularly interested to see if it says similar errors to as ones you saw when running through Moonfire NVR:

I also don't know what you mean by "library warning". The ffmpeg -version output from both looks normal to me.

The repeated frame=442457 fps= 12 q=-1.0 size... message is normal; that's its way of showing progress. It shouldn't take an hour to start seeing this, though; I'd expect it to happen immediately.

jlpoolen commented 3 years ago

Here's a copy from the Raspberry Pi console. See on line 5 "WARNING: library configuration mismatch".

jlpoole@raspberrypi:~ $ ffmpeg -loglevel debug -fflags nobuffer -rtsp_transport tcp -i 'rtsp://admin:testtest@192.168.1.49:554/h264Preview_01_main' -c copy -f null -
ffmpeg version 4.1.6-1~deb10u1+rpt1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --libdir=/usr/lib/arm-linux-gnueabihf --cpu=arm1176jzf-s --arch=arm
  WARNING: library configuration mismatch
  avutil      configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  avcodec     configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  avformat    configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  avdevice    configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  avfilter    configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  avresample  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  swscale     configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  swresample  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  postproc    configuration: --prefix=/usr --extra-version='1~deb10u1+rpt1' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --cpu=cortex-a7 --arch=armv6t2 --disable-thumb --enable-shared --disable-doc --disable-programs
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-fflags' ... matched as AVOption 'fflags' with argument 'nobuffer'.
Reading option '-rtsp_transport' ... matched as AVOption 'rtsp_transport' with argument 'tcp'.
Reading option '-i' ... matched as input url with argument 'rtsp://admin:testtest@192.168.1.49:554/h264Preview_01_main'.
Reading option '-c' ... matched as option 'c' (codec name) with argument 'copy'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'null'.
Reading option '-' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url rtsp://admin:testtest@192.168.1.49:554/h264Preview_01_main.
Successfully parsed a group of options.
Opening an input file: rtsp://admin:testtest@192.168.1.49:554/h264Preview_01_main.
[tcp @ 0x13331d0] No default whitelist set
[tcp @ 0x13331d0] Original list of addresses:
[tcp @ 0x13331d0] Address 192.168.1.49 port 554
[tcp @ 0x13331d0] Interleaved list of addresses:
[tcp @ 0x13331d0] Address 192.168.1.49 port 554
[tcp @ 0x13331d0] Starting connection attempt to 192.168.1.49 port 554
[tcp @ 0x13331d0] Successfully connected to 192.168.1.49 port 554
[rtsp @ 0x1331230] SDP:
v=0
o=- 1615111245371232 1 IN IP4 192.168.1.49
s=Session streamed by "preview"
i=h264Preview_01_main
t=0 0
a=tool:LIVE555 Streaming Media v2013.04.08
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session streamed by "preview"
a=x-qt-text-inf:h264Preview_01_main
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:500
a=rtpmap:96 H264/90000
a=range:npt=0-
a=fmtp:96 packetization-mode=1;profile-level-id=640033;sprop-parameter-sets=Z2QAM6zoAoALWQ==,aO48MA==
a=control:trackID=1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
b=AS:256
a=rtpmap:97 MPEG4-GENERIC/16000
a=fmtp:97 streamtype=5;profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408; profile=1;
a=control:trackID=2

Failed to parse interval end specification ''
[rtsp @ 0x1331230] video codec set to: h264
Failed to parse interval end specification ''
[rtsp @ 0x1331230] RTP Packetization Mode: 1
[rtsp @ 0x1331230] RTP Profile IDC: 64 Profile IOP: 0 Level: 33
[rtsp @ 0x1331230] Extradata set to 0x1334b58 (size: 22)
[rtsp @ 0x1331230] audio codec set to: aac
[rtsp @ 0x1331230] audio samplerate set to: 16000
[rtsp @ 0x1331230] audio channels set to: 1
[rtsp @ 0x1331230] setting jitter buffer size to 0
    Last message repeated 1 times
[rtsp @ 0x1331230] hello state=0
Failed to parse interval end specification ''
[h264 @ 0x13346e0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x13346e0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x13346e0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x13346e0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x13346e0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x13346e0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x13346e0] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x13346e0] Format yuv420p chosen by get_format().
[h264 @ 0x13346e0] Reinit context to 2560x1440, pix_fmt: yuv420p
[h264 @ 0x13346e0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 2
    Last message repeated 5 times
[rtsp @ 0x1331230] Non-increasing DTS in stream 1: packet 17 with DTS 17408, packet 18 with DTS 16679
[rtsp @ 0x1331230] All info found
[rtsp @ 0x1331230] rfps: 11.750000 0.015265
[rtsp @ 0x1331230] rfps: 11.833333 0.006981
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 11.916667 0.001900
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 12.000000 0.000023
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 12.083333 0.001348
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 12.166667 0.005877
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 12.250000 0.013609
[rtsp @ 0x1331230] rfps: 23.750000 0.016162
[rtsp @ 0x1331230] rfps: 23.833333 0.007602
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 23.916667 0.002245
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 24.000000 0.000091
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 24.083333 0.001141
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 24.166667 0.005393
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 24.250000 0.012849
[rtsp @ 0x1331230] rfps: 36.000000 0.000206
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 48.000000 0.000365
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 60.000000 0.000571
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 120.000000 0.002284
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 240.000000 0.009136
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 23.976024 0.000383
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 59.940060 0.002392
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 11.988012 0.000096
    Last message repeated 1 times
[rtsp @ 0x1331230] rfps: 47.952048 0.001531
    Last message repeated 1 times
Input #0, rtsp, from 'rtsp://admin:testtest@192.168.1.49:554/h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0, 28, 1/90000: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 2560x1440, 0/1, 12 tbr, 90k tbn, 180k tbc
    Stream #0:1, 39, 1/16000: Audio: aac (LC), 16000 Hz, mono, fltp
Successfully opened the file.
Parsing a group of options: output url -.
Applying option c (codec name) with argument copy.
Applying option f (force format) with argument null.
Successfully parsed a group of options.
Opening an output file: -.
Successfully opened the file.
Output #0, null, to 'pipe:':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
    encoder         : Lavf58.20.100
    Stream #0:0, 0, 1/90000: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 2560x1440 (0x0), 0/1, q=2-31, 12 tbr, 90k tbn, 90k tbc
    Stream #0:1, 0, 1/16000: Audio: aac (LC), 16000 Hz, mono, fltp
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
    Last message repeated 17 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
    Last message repeated 13 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
    Last message repeated 13 times
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
    Last message repeated 4 times
[rtsp @ 0x1331230] CSeq 19 expected, 0 received.4.25 bitrate=N/A speed=1.01x
    Last message repeated 1 times
[rtsp @ 0x1331230] CSeq 19 expected, 0 received.4.83 bitrate=N/A speed=   1x
    Last message repeated 4 times
[rtsp @ 0x1331230] CSeq 44 expected, 0 received.9.97 bitrate=N/A speed=   1x
    Last message repeated 9 times
[rtsp @ 0x1331230] CSeq 44 expected, 0 received.1.69 bitrate=N/A speed=   1x
    Last message repeated 4 times
[rtsp @ 0x1331230] CSeq 59 expected, 0 received.6.01 bitrate=N/A speed=   1x
[rtsp @ 0x1331230] CSeq 59 expected, 0 received.6.78 bitrate=N/A speed=   1x
    Last message repeated 4 times
[rtsp @ 0x1331230] CSeq 59 expected, 0 received.8.89 bitrate=N/A speed=   1x
    Last message repeated 11 times
[rtsp @ 0x1331230] CSeq 84 expected, 0 received.4.54 bitrate=N/A speed=   1x
[rtsp @ 0x1331230] CSeq 99 expected, 0 received.2.11 bitrate=N/A speed=   1x
    Last message repeated 5 times
[rtsp @ 0x1331230] CSeq 156 expected, 0 received..87 bitrate=N/A speed=   1x
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87171947
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87172971
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87173995
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87175019
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490344011
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87176043
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490351481
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490358951
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490366511
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87177067
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87178091
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490373981
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87179115
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87180139
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490381451
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490389011
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87181163
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87182187
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490396481
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490403951
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87183211
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87184235
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490411511
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87185259
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87186283
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490418981
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87187307
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 490433759 >= 490426451
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 87188775 >= 87188331
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202388695
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138437240
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202389719
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138444800
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202390743
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202391767
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138452270
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202392791
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138459740
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202393815
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202394839
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138467300
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202395863
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138474770
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202396887
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202397911
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138482240
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202398935
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138489710
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138497270
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202399959
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202400983
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138504740
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202402007
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138512210
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 202423971 >= 202403031
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1138635316 >= 1138519770
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317636742
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317637766
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317638790
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317639814
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786710615
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317640838
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786718085
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317641862
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786725645
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786733115
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317642886
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786740585
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317643910
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317644934
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786748145
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786755615
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317645958
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317646982
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317648006
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786763085
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786770555
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 317657815 >= 317649030
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1786825419 >= 1786778115
[rtsp @ 0x1331230] CSeq 773 expected, 0 received.9.67 bitrate=N/A speed=   1x
    Last message repeated 3 times
[rtsp @ 0x1331230] CSeq 857 expected, 0 received.1.76 bitrate=N/A speed=   1x
    Last message repeated 2 times
[rtsp @ 0x1331230] CSeq 895 expected, 0 received.6.78 bitrate=N/A speed=   1x
    Last message repeated 2 times
[rtsp @ 0x1331230] CSeq 895 expected, 0 received.7.55 bitrate=N/A speed=   1x
    Last message repeated 4 times
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432868026
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434883816
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432869050
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432870074
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434891286
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432871098
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432872122
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434898846
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432873146
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434906316
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432874170
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434913786
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 432892991 >= 432875194
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434921346
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2435021087 >= 2434928816
[rtsp @ 0x1331230] CSeq 917 expected, 0 received.2.71 bitrate=N/A speed=   1x
[rtsp @ 0x1331230] CSeq 1000 expected, 0 received..41 bitrate=N/A speed=   1x
    Last message repeated 6 times
[rtsp @ 0x1331230] CSeq 1015 expected, 0 received..33 bitrate=N/A speed=   1x
    Last message repeated 10 times
[rtsp @ 0x1331230] CSeq 1015 expected, 0 received..34 bitrate=N/A speed=   1x
    Last message repeated 14 times
[rtsp @ 0x1331230] CSeq 1024 expected, 0 received..67 bitrate=N/A speed=   1x
[rtsp @ 0x1331230] CSeq 1068 expected, 0 received..72 bitrate=N/A speed=   1x
    Last message repeated 2 times
[rtsp @ 0x1331230] CSeq 1081 expected, 0 received..18 bitrate=N/A speed=   1x
    Last message repeated 3 times
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 548087929 >= 548051960
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 548087929 >= 548052984
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 548087929 >= 548084754
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 548087929 >= 548085778
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3082991914 >= 3082800412
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 548087929 >= 548086802
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 548087929 >= 548087826
[null @ 0x13823d0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 3082991914 >= 3082985670
[rtsp @ 0x1331230] CSeq 1144 expected, 0 received..40 bitrate=N/A speed=   1x
[rtsp @ 0x1331230] CSeq 1144 expected, 0 received..62 bitrate=N/A speed=   1x
    Last message repeated 6 times
[rtsp @ 0x1331230] CSeq 1144 expected, 0 received..92 bitrate=N/A speed=   1x
    Last message repeated 1 times
[rtsp @ 0x1331230] CSeq 1144 expected, 0 received..52 bitrate=N/A speed=   1x
    Last message repeated 7 times
[rtsp @ 0x1331230] CSeq 1158 expected, 0 received..37 bitrate=N/A speed=   1x
    Last message repeated 3 times
[rtsp @ 0x1331230] CSeq 1194 expected, 0 received..78 bitrate=N/A speed=   1x
    Last message repeated 3 times
[rtsp @ 0x1331230] CSeq 1194 expected, 0 received..89 bitrate=N/A speed=   1x
frame=447480 fps= 12 q=-1.0 Lsize=N/A time=10:04:10.61 bitrate=N/A speed=   1x
video:35977814kB audio:137645kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (rtsp://admin:testtest@192.168.1.49:554/h264Preview_01_main):
  Input stream #0:0 (video): 447500 packets read (36842824068 bytes);
  Input stream #0:1 (audio): 566060 packets read (140948879 bytes);
  Total: 1013560 packets (36983772947 bytes) demuxed
Output file #0 (pipe:):
  Output stream #0:0 (video): 447480 packets muxed (36841282003 bytes);
  Output stream #0:1 (audio): 566060 packets muxed (140948879 bytes);
  Total: 1013540 packets (36982230882 bytes) muxed
0 frames successfully decoded, 0 decoding errors
Exiting normally, received signal 2.
jlpoole@raspberrypi:~
scottlamb commented 3 years ago

Okay, those Application provided invalid, non monotonically increasing dts errors look like the same problems that caused Moonfire NVR to close the stream.

Do they happen on the Gentoo server? Do they happen if you remove the -fflags nobuffer?

The library warning seems to be saying that it compiled the binary and the libraries with this difference in options:

$ diff -u binary avutil
--- binary  2021-03-12 08:57:21.179545106 -0800
+++ avutil  2021-03-12 08:57:21.179363592 -0800
@@ -64,7 +64,10 @@
 --enable-chromaprint
 --enable-frei0r
 --enable-libx264
+--libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp
+--cpu=cortex-a7
+--arch=armv6t2
+--disable-thumb
 --enable-shared
---libdir=/usr/lib/arm-linux-gnueabihf
---cpu=arm1176jzf-s
---arch=arm
+--disable-doc
+--disable-programs

I don't think it's causing any problems.

jlpoolen commented 3 years ago

So the "frame=..." is a bottom line in a console streen that is continually updated to show progress. I'm assuming this is an ncurses kind of feature to avoid scrolling caused by repetitive output. putty_2021-03-12_09-39-30

So in my comment earlier above, it looks like something happened within the console (or ncurses?) that caused the update line to move down the screen rather than replace itself so the console looks like it has a constantly updated bottom line. What I thought was an error with ffmpeg looks to have been something caused in my console where the refresh line no longer replaced, but moved down a line in the screen buffer causing the screen to scroll with each line of output.

I now have three consoles running against the same camera, each with a different version of ffmpeg. I have on my taurus laptop the latest & greatest version of ffmpeg, release 4.3.2. I am doing this to see if there are consistent errors in all three session give a certain point of time. So far, I'm seeing different results. I'm now thinking network issues may be in play. The garage cameras come from the garage through a Ubiquiti wireless link, the Peck comes via a long Cat6 running to the house next door, and the laptop has my current house's wireless. So, a variety of network connections.

I eventually found all screens populated with red error messages and then hung. No more updates. I entered Control-C and the ffmpeg program terminated and I was returned to a standard prompt. This proves that my ssh session was still alive and the hang up occurring on ffmpeg's side.

I'm starting to think that moonfire-nvr is faced with handling problems from ffmpeg and being able to recover gracefully when ffmpeg hangs as it appears to have done in several versions on several machines. I'm going to do further testing just using various ffmpegs rtsp sessions, my approach has been haphazard and not very methodical and at this point we need to have some methodically produced results.

scottlamb commented 3 years ago

The frame= line is continuously updated just by sending a carriage return, sending the cursor back to the start of the line. If more lines are printed in the meantime(like that CSeq log line), the display gets a little off as in your screenshot.

There's another option Moonfire NVR sets when it opens streams with ffmpeg, stimeout 10000000 (socket timeout). If a RTSP connection hits that timeout (in microseconds, so 10,000,000 µs = 10 seconds), the connection's dropped, and then Moonfire NVR retries after a second. You can match that with this variation of the command:

ffmpeg -report -loglevel debug -fflags nobuffer -stimeout 10000000 -rtsp_transport tcp -allowed_media_types video -i 'rtsp://...' -c copy -f null -

If your network connection is unreliable, that might just be the best that can be done, with ffmpeg or any other RTSP client. Maybe what changed since you had a working setup on the Pi wasn't ffmpeg or Moonfire but the wireless network conditions.

scottlamb commented 3 years ago

I've been playing a bit with a Reolink camera (a RLC-410, hardware version IPC_3816M, firmware v2.0.0.1441_19032101). I think it's actually sending interleaved RTP data over a RTSP connection which must be intended for a different RTSP connection:

I imagine these problems are worse if you have multiple NVRs running, record both the main and sub streams, and/or have frequent connection drops for other reasons (eg poor wifi).

Once I move to a non-ffmpeg RTSP implementation, I might be able to work around this problem (by discarding data which doesn't have the expected ssrc) but wow, that's a bug that just shouldn't happen. I wonder if newer models are better (this one's apparently end-of-lifed) and if there's any way to get a bug report to an engineer who would actually fix it.

I also get warnings in the logs about long times to get the next frame if there's a dramatic change to the image. I have the thing in a cabinet with a door right now; if I open or close the door, I can see it in my logs every time. That really shouldn't happen either.

Finally I see cases where the RTP timestamp goes slightly backwards; Moonfire NVR drops the connection in this case. (iirc this also happens with my Hikvision and/or Dahua cameras sometimes.) This seems to happen when they do a time step with SNTP—it must be calculating offsets with CLOCK_REALTIME rather than CLOCK_MONOTONIC or some such. If the system clock goes backwards, so do the RTP timestamps. Notably, I currently have the Reolink camera to do a SNTP step every 60 minutes, and the timestamp problems happen almost exactly every two hours. I guess it goes forward half the time and back half the time. I also see a huge mismatch when I first set the clock, so I'm pretty sure this is the problem. I could probably work around this by say assuming an increment of the frame rate if the timestamp delta is unreasonable.

scottlamb commented 3 years ago

It may not be all the camera's fault, though. The bad cseq message from ffmpeg is bogus in at least some cases (#126), and we can add support for RTP H.264 NAL unit type 29.

jlpoolen commented 3 years ago

I think I've found something that might bear on the problems I encountered. One of the facts that were different after I started having problems was that I had upgraded my RaspberryPi4 system. This bug gets into the nitty gritty of kernels and sleep modes for USB attached hard drives, and some of the people posting noted their problem went away when the downgraded to a previous kernel. I just wanted to add this URL because it may have bearing on what I experienced: https://github.com/raspberrypi/linux/issues/3981 No need for comments at this time. I'm also grappling with the issue of whether I should replace my SD HC to remove any question about media problems contributing to some dropped videos. I've not been keeping diligent records, but what I am noticing is that my Windows Reolink client is capturing events, triggered by motion settings, at times when moonfire-nvr is not capturing. I basically can use the Reolink software as a control: if it saves files and I cannot find them through moonfire-nvr, then I know something has happened on the RapsberryPi end. I believe my problems are hardware and/or system related and not related to moonfire-nvr.

I am kind of regretting having gone with RaspberryPi since it depends on an SD HC which has an admitted limited life. Even Scott's suggestions tries to limit the cycles performed against the diskette. I noticed some of the of the systems Scott recommended are out-of-stock at the linked vendor. I'm thinking for 24/7 video, something not operating on an SD HC is preferable. Just too may write cycles. And the kernel issue in the bug of the URL herein is troubling. I've also complicated things by using LVM. I used to use LVM for my Xen hypervisor and then I discovered some shortcoming and well as LVM developer malaise that made me decide to go back to the tried-and-true partition approach.

What tangled webs we weave. But having video surveillance feeds coming to a generic browser is so worth it, I just wish I had more time.

scottlamb commented 3 years ago

I am kind of regretting having gone with RaspberryPi since it depends on an SD HC which has an admitted limited life.

From a previous email, I recall you have a 5-bay USB SATA enclosure. Have you considered buying a SATA SSD and sticking in there? The Pi4 supports USB booting, and a real SATA SSD would be much more reliable.

jlpoolen commented 3 years ago

Good suggestion, but I'm shy about SSDs. Any reason not to use a Western Digital Blue, at $20, it seems like a no-brainer? I don't need the performance an SSD offers; the stability is more important to me. I have two empty slots in my USB array. Thank you.

Edit: changed the URL to points to 3.5" form, original link was to notebook which is a form factor I'm not sure is supported by my array.

scottlamb commented 3 years ago

I don't think you need to feel shy about SSDs; they're in a different reliability class than SD cards, and they actually report SMART statistics so you can see how much wear they have on them. I'd expect them to be more reliable as a root volume than a refurbished ("renewed") HDD like that one.

But to answer your question: putting the root drive and SQLite database on a HDD is probably fine. Database accesses will be noticeably slower but probably bad enough so that Moonfire NVR will actually break.

scottlamb commented 3 years ago

Edit: changed the URL to points to 3.5" form, original link was to notebook which is a form factor I'm not sure is supported by my array.

I still see the 2.5" link. Another thing to check is if the 3.5" drive (or whatever you select) is using SMR; I try to avoid those drives.

Usually arrays have trays that let you put 2.5" drives in 3.5" slots. SSDs all come in 2.5" form so you'd need this to take my suggestion of putting in a SSD.

jlpoolen commented 3 years ago

I did more reading about SSD and decided Scott's suggestion was well taken. So I ordered the following:

Installed new Raspberry Pi minimal operating system on PNY. Reconfigured my Pi to boot from USB first (now SD HC is removed). I will be installing (have to buy a CR 1222 battery locally):

The goal here is to isolate potential failures to more reliable systems than an SD HC card that has been in use for over 1 year. Yesterday, on the SD HC-based system, something happened and I could not even reboot from a console, the system locked up. /var/log/messages had some ext4 related errors which causes me to think there were disk problems.

Aug 17 13:07:59 raspberrypi kernel: [55663.494808] Exception stack(0xc923dfa8 to 0xc923dff0)
Aug 17 13:07:59 raspberrypi kernel: [55663.494828] dfa0:                   00000020 ad2f50b0 00000020 00000002 00000000 00000000
Aug 17 13:07:59 raspberrypi kernel: [55663.494847] dfc0: 00000020 ad2f50b0 00000000 00000076 ad2f4598 ad2f4574 00000000 ad2f4568
Aug 17 13:07:59 raspberrypi kernel: [55663.494865] dfe0: 00000076 ad2f43a0 b59e1805 b59e34d6
Aug 17 13:07:59 raspberrypi kernel: [55663.494931] task:s-GarageEast-ma state:D stack:    0 pid: 1556 ppid:  1500 flags:0x00000000
Aug 17 13:07:59 raspberrypi kernel: [55663.494962] Backtrace:
Aug 17 13:07:59 raspberrypi kernel: [55663.494996] [<c0b8c820>] (__schedule) from [<c0b8d1f8>] (schedule+0x68/0xe4)
Aug 17 13:07:59 raspberrypi kernel: [55663.495016]  r10:00000001 r9:00000000 r8:00000000 r7:005e4859 r6:c33cc050 r5:c2ce3e00
Aug 17 13:07:59 raspberrypi kernel: [55663.495032]  r4:ffffe000
Aug 17 13:07:59 raspberrypi kernel: [55663.495053] [<c0b8d190>] (schedule) from [<c057eb60>] (wait_transaction_locked+0xc0/0xf8)
Aug 17 13:07:59 raspberrypi kernel: [55663.495070]  r5:c1205048 r4:c33cc000
Aug 17 13:07:59 raspberrypi kernel: [55663.495092] [<c057eaa0>] (wait_transaction_locked) from [<c057f04c>] (start_this_handle+0x4b4/0x970)
Aug 17 13:07:59 raspberrypi kernel: [55663.495110]  r7:c3a08360 r6:c0e51834 r5:c33cc028 r4:c33cc000
Aug 17 13:07:59 raspberrypi kernel: [55663.495133] [<c057eb98>] (start_this_handle) from [<c057f610>] (jbd2__journal_start+0x108/0x1fc)
Aug 17 13:07:59 raspberrypi kernel: [55663.495153]  r10:00000001 r9:c13fb098 r8:00000000 r7:00000008 r6:00000002 r5:c33cc000
Aug 17 13:07:59 raspberrypi kernel: [55663.495168]  r4:c3a08360
Aug 17 13:07:59 raspberrypi kernel: [55663.495194] [<c057f508>] (jbd2__journal_start) from [<c050d568>] (__ext4_journal_start_sb+0x84/0x100)
Aug 17 13:07:59 raspberrypi kernel: [55663.495214]  r10:c0c15df4 r9:c0529bd0 r8:c3ae24b0 r7:00000bb6 r6:00000002 r5:00000001
Aug 17 13:07:59 raspberrypi kernel: [55663.495229]  r4:c5958400
Aug 17 13:07:59 raspberrypi kernel: [55663.495255] [<c050d4e4>] (__ext4_journal_start_sb) from [<c0532614>] (ext4_da_write_begin+0x1cc/0x498)
Aug 17 13:07:59 raspberrypi kernel: [55663.495273]  r7:00000000 r6:01f40000 r5:c3ae23a0 r4:d99538f0
Aug 17 13:07:59 raspberrypi kernel: [55663.495297] [<c0532448>] (ext4_da_write_begin) from [<c03af0c8>] (generic_perform_write+0x108/0x1dc)
Aug 17 13:07:59 raspberrypi kernel: [55663.495317]  r10:c0532448 r9:c0c15c24 r8:00000dcd r7:00000233 r6:c3ae24b0 r5:c90adee0
Aug 17 13:07:59 raspberrypi kernel: [55663.495333]  r4:00000233
Aug 17 13:07:59 raspberrypi kernel: [55663.495355] [<c03aefc0>] (generic_perform_write) from [<c051b1f8>] (ext4_buffered_write_iter+0xb4/0x184)
Aug 17 13:07:59 raspberrypi kernel: [55663.495375]  r10:c90adee0 r9:c1205048 r8:c90adee0 r7:c3ae2428 r6:c3ae23a0 r5:c2ce3e00
Aug 17 13:07:59 raspberrypi kernel: [55663.495390]  r4:c90adef8
Aug 17 13:07:59 raspberrypi kernel: [55663.495412] [<c051b144>] (ext4_buffered_write_iter) from [<c051b7a0>] (ext4_file_write_iter+0x120/0x988)
Aug 17 13:07:59 raspberrypi kernel: [55663.495431]  r8:c90adf58 r7:00000000 r6:c90adef8 r5:c3ae23a0 r4:00000000
Aug 17 13:07:59 raspberrypi kernel: [55663.495453] [<c051b680>] (ext4_file_write_iter) from [<c043dd64>] (vfs_write+0x234/0x464)
Aug 17 13:07:59 raspberrypi kernel: [55663.495473]  r10:00000000 r9:00000000 r8:c90adf58 r7:00000000 r6:c1205048 r5:00033ec2
Aug 17 13:07:59 raspberrypi kernel: [55663.495488]  r4:cc96e9c0
Aug 17 13:07:59 raspberrypi kernel: [55663.495509] [<c043db30>] (vfs_write) from [<c043e0f8>] (ksys_write+0x68/0xe0)
Aug 17 13:07:59 raspberrypi kernel: [55663.495529]  r10:00033ec2 r9:aafaade8 r8:cc96e9c0 r7:00000000 r6:01f3fdcd r5:cc96e9c3
Aug 17 13:07:59 raspberrypi kernel: [55663.495544]  r4:c1205048
Aug 17 13:07:59 raspberrypi kernel: [55663.495564] [<c043e090>] (ksys_write) from [<c043e188>] (sys_write+0x18/0x1c)
Aug 17 13:07:59 raspberrypi kernel: [55663.495583]  r10:00000004 r9:c90ac000 r8:c0200204 r7:00000004 r6:accf20b0 r5:aafaade8
Aug 17 13:07:59 raspberrypi kernel: [55663.495599]  r4:00033ec2
Aug 17 13:07:59 raspberrypi kernel: [55663.495620] [<c043e170>] (sys_write) from [<c0200040>] (ret_fast_syscall+0x0/0x28)
Aug 17 13:07:59 raspberrypi kernel: [55663.495636] Exception stack(0xc90adfa8 to 0xc90adff0)
Aug 17 13:07:59 raspberrypi kernel: [55663.495655] dfa0:                   00033ec2 aafaade8 0000002a aafaade8 00033ec2 00000000
Aug 17 13:07:59 raspberrypi kernel: [55663.495675] dfc0: 00033ec2 aafaade8 accf20b0 00000004 00ba77d0 00033ec2 accf0bd8 aafaade8
Aug 17 13:07:59 raspberrypi kernel: [55663.495693] dfe0: 00000004 accefa50 b59e13ad b59e34d6
Aug 17 13:07:59 raspberrypi kernel: [55663.495757] task:s-PeckAlleyWest state:D stack:    0 pid: 1557 ppid:  1500 flags:0x00000000
scottlamb commented 3 years ago

As this is now understood as an RTSP-level interop problem with these buggy cameras, let's continue discussion at scottlamb/retina#17 as necessary. Current best solution is to use the new --rtsp-transport=udp argument.