Problems streaming from Hikvision NVR

[MindrustUK]

Thanks for the application, it looks good so far but I'm having trouble streaming from a Hikvision NVR, console output follows:

[rtsp @ 0x7fe254005320] max delay reached. need to consume packet [rtsp @ 0x7fe254005320] RTP: missed 84 packets [h264 @ 0x7fe254009360] error while decoding MB 25 30, bytestream -5 [h264 @ 0x7fe254009360] concealing 2424 DC, 2424 AC, 2424 MV errors in I frame

This works in VLC without a problem, quickly looking around it appears to be related to UDP vs TCP transport mechanisms. Could you add an option to specify a transport option? This will hopefully sort the issue.

[iEvgeny]

Hi! The last commit has now added the ability to set AVFormat options for a stream in cmd line format. Try set -rtsp_transport tcp option in text field for each viewport, if necessary.

IMPORTANT! Currently, the "-analyzeduration 0 -probesize 500000" options are implicitly set to speed up the start of the stream. Try increasing these parameters to their default values: -analyzeduration 5000000 -probesize 5000000

P.S. All AVFormat options are be available.

[MindrustUK]

Thanks for the reply. Setting the flag seems to have cleared the errors, but the video feed is still not working due to what looks like an authentication problem. Do you have any suggestions? I'm going to investigate further when I have some time.

[iEvgeny]

The link for Hikvision cameras usually looks like this: rtsp://<user>:<password>@<address>:554 Just replace the values in the <> fields with the required ones.

[MindrustUK]

Thanks, this is a Hikvision NVR which has the format rtsp://<user>:<pass>@<address>:554/Streaming/Channels/102 (Where 1 is the camera, and 2 is the substream). This doesn't seem to work, I'm getting no errors just silent failure.

[iEvgeny]

Now I checked it. Hikvision DS-2CD2022WD-I supports both types of links. Also try this: rtsp://<user>:<pass>@<address>:554/Streaming/Channels/101

[iEvgeny]

I also recommend using ffmpeg instead of VLC for your tests. This will give you a more complete diagnostic output. For example: ffplay -rtsp_transport tcp -i rtsp://<user>:<password>@<address>:554

[MindrustUK]

ffplay works perfectly as suggested (ffplay -rtsp_transport tcp -i rtsp://<user>:<pass>@<address>:554/Streaming/Channels/101). Still no joy in cctv-viewer for some reason.

[iEvgeny]

Does this demo stream work for you in CCTV Viewer? rtmp://live.a71.ru/demo/0

[MindrustUK]

Yup the demo stream works just fine in CCTV Viewer.

[iEvgeny]

Hm... Then try to run CCTV Viewer from the console. Usually the output is quite informative in terms of problems.

[MindrustUK]

From the console I get the following output:

Qt: Session management error: None of the authentication protocols specified are supported /usr/share/libdrm/amdgpu.ids: No such file or directory

The demo stream and a stream from a Unifi camera works fine.

[iEvgeny]

These are program execution errors in an isolated snap environment. They are not related to RTSP stream playback problems. I assume that your stream contains an unsupported pixel format or something like that. Support for some specific pixel formats is not yet implemented... Could you give here the ffprobe output for your stream?

[MindrustUK]

Output as requested:

ffprobe version 4.2.4-1ubuntu0.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libaribb24 --enable-liblensfun --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtsp @ 0x55bfffcddf00] max delay reached. need to consume packet
[rtsp @ 0x55bfffcddf00] RTP: missed 64 packets
[h264 @ 0x55bfffce1ec0] error while decoding MB 22 35, bytestream -19
[h264 @ 0x55bfffce1ec0] concealing 7835 DC, 7835 AC, 7835 MV errors in I frame
[rtsp @ 0x55bfffcddf00] max delay reached. need to consume packet
[rtsp @ 0x55bfffcddf00] RTP: missed 64 packets
[h264 @ 0x55bfffce1ec0] error while decoding MB 7 35, bytestream -9
[h264 @ 0x55bfffce1ec0] concealing 7850 DC, 7850 AC, 7850 MV errors in I frame
Input #0, rtsp, from 'rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/101':
  Metadata:
    title           : HIK Media Server V3.4.98
    comment         : HIK Media Server Session Description : standard
  Duration: N/A, start: 0.449000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 2048x1536, 20 fps, 50 tbr, 90k tbn, 40 tbc

[iEvgeny]

yuv420p is fully supported. If your stream does not contain any confidential information, please run this command and send me the output file to e-mail for codec analysis, etc.: ffmpeg -rtsp_transport tcp -i rtsp://<user>:<password>@<address>:554/Streaming/Channels/101 -c copy -t 30 stream-sample.mkv

P.S. In the future it may be necessary to analyze the stream transport, but let's hope this will not be necessary.

[MindrustUK]

I've mailed you a sample of the camera footage to have a look at. Happy to run any debugging as needed.

[iEvgeny]

Well, the file can be played locally in CCTV Viewer. It's good! Just in case, please check with you: file:///<path>/stream-sample.mkv But this indicates transport layer problems or other errors in CCTV Viewer. First, please try to install CCTV Viewer from my PPA repository: sudo snap remove cctv-viewer && sudo add-apt-repository ppa:ievgeny/cctv-viewer && sudo apt install cctv-viewer This will make sure that the problem is not in snap restrictions or outdated ffmpeg libraries (Snap packages are currently building on Ubuntu 18.04, but you have Ubuntu 20.04 as I noticed).

[MindrustUK]

I've tested local playback from file and that works fine.

I've replaced the snap with the ppa as above. Still the same results playing remotely.

Yes, confirmed Ubuntu 20.04.

I guess this is transport layer related then?

[iEvgeny]

Yes. It looks like a transport level problem. If, when trying to play a stream with the -rtsp_transport tcp option, there is no meaningful output in the console or there are no other messages in the current viewport ("Invalid media!", "Stalled" and etc.), so I have no ideas how to debug it without access to the stream.

[iEvgeny]

However, I think it's worth adding a debug mode to the application. I will think about it in the near future.

[MindrustUK]

A debug option would be good. Thanks. Feel free to close off the issue as needed.

[iEvgeny]

I have made some improvements. If the problem is still relevant, then after the update, try launching CCTV Viewer as follows: export VERBOSE_LEVEL=3 && cctv-viewer The resulting output may help solve your problem. Please, be careful. The information may contain sensitive data.

[MindrustUK]

Thanks for adding debugging, and thanks for the warning I've scrubbed anything sensitive from the following output:

Qt: Session management error: None of the authentication protocols specified are supported
[55e10919b090] QmlAVDemuxer::QmlAVDemuxer()
[55e10919b090, ] QmlAVPlayer::stateMachine() => 2:0
Input #0, rtsp, from 'rtsp://<user>:<pass>@<address>/Streaming/Channels/401':
  Metadata:
    title           : HIK Media Server V3.4.98
    comment         : HIK Media Server Session Description : standard
  Duration: N/A, start: 0.400000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1280x960, 25 fps, 25 tbr, 90k tbn, 50 tbc
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] Media loaded successfully!
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVPlayer::stateMachine() => 3:0
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVDemuxer::run(1)
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVPlayer::stateMachine() => 3:0
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVPlayer::stateMachine() => 3:1
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVPlayer::stateMachine() => 6:1
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVDemuxer::requestInterruption()
[55e10919b090, rtsp://<user>:<pass>@<address>/Streaming/Channels/401] QmlAVDemuxer::requestInterruption()
[55e10919b090] QmlAVDemuxer::~QmlAVDemuxer()
[55e10919b090] QmlAVDemuxer::run(2)

[iEvgeny]

I don't see any anomalies yet. I added additional diagnostics. Please run the CCTV Viewer again with the environment variable specified above.

[MindrustUK]

Repeated as requested:

[7fb0d00032a0] QmlAVDemuxer::QmlAVDemuxer()
[7fb0d00032a0] Added AVFormat option: -analyzeduration 0
[7fb0d00032a0] Added AVFormat option: -probesize 500000
[7fb0d00032a0] QmlAVPlayer::stateMachine() : { m_status=2; m_playbackState=0 }
[rtsp @ 0x7fb07c0032c0] max delay reached. need to consume packet
[rtsp @ 0x7fb07c0032c0] RTP: missed 38 packets
[h264 @ 0x7fb07c008340] error while decoding MB 78 38, bytestream -21
[h264 @ 0x7fb07c008340] concealing 1731 DC, 1731 AC, 1731 MV errors in I frame
Input #0, rtsp, from 'rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401':
  Metadata:
    title           : HIK Media Server V3.4.98
    comment         : HIK Media Server Session Description : standard
  Duration: N/A, start: 0.320000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1280x960, 25 fps, 25 tbr, 90k tbn, 50 tbc
QString::arg: Argument missing: "Found 1 video and 1 audio streams" , 0
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] Found 1 video and 1 audio streams
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] Media loaded successfully!
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVPlayer::stateMachine() : { m_status=3; m_playbackState=0 }
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVDemuxer::run() : { startTime=1599826389430298 }
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVPlayer::stateMachine() : { m_status=3; m_playbackState=0 }
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVPlayer::stateMachine() : { m_status=3; m_playbackState=1 }
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVPlayer::stateMachine() : { m_status=6; m_playbackState=1 }
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVPlayer::setHasVideo(1)
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVPlayer::setHasVideo(0)
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVDemuxer::requestInterruption()
[7fb0d00032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVDemuxer::requestInterruption()
[7fb0d00032a0] QmlAVDemuxer::~QmlAVDemuxer()
[7fb0d00032a0] QmlAVDemuxer::run() : { m_videoDecoder.clock()->1599826389430298 }

[iEvgeny]

I extended the logging once again as I still do not find the clue for this problem. Now we will be able to see the information for each frame. This is not optimal, so please do not run multiple streams and do not run the test too long. Listing I think will be better placed on GitHub Gist: https://gist.github.com/

[MindrustUK]

Output here: https://gist.github.com/MindrustUK/f2ea1ca1e33404290d7b6eba36bb486d

I've truncated the output as the line [7f3a080032a0, rtsp://<USER>:<PASS>@<ADDRESS>:554/Streaming/Channels/401] QmlAVDemuxer::run() : { Loop of message queue depth: m_handledTime=0; m_lastTime=1600094878110922 }

repeats infinitely until existing the program.

[iEvgeny]

This is already much more interesting, although it does not explain the differences in the playback of the stream and the local file. In the near future I will make a small code refactoring, which should give real hope to understand the problem. Thank you for your help and please keep in touch!

[MindrustUK]

Thanks for investigating the issue, let me know when there is more ready for testing and I'll provide my feedback.

[iEvgeny]

I have made some improvements that are supposed to produce results. Please check the new version of CCTV Viewer.

[MindrustUK]

Glad to say whatever you've done is mostly working! Thanks!

I've got a bug on one camera to do with sub-streams, dump here: https://gist.github.com/MindrustUK/da2c6c1d6e2f5637bda143d297e989ae

If I use the sub-stream: 502, CCTV Viewer crashes and I have to remove the configuration from from the CCTV Viewer.conf file to start CCTV Viewer up again.

Another bug is graphical corruption; This relates to the main stream (/Streaming/Channels/nn1). These streams are the main high resolution stream type. The sub-streams are the streams ending /Streaming/Channels/nn2 appear to be 320x240. The Hikvision android app shows the low resolution streams when viewing multiple cameras and switches to the main stream (/Streaming/Channels/nn1) for full resolution when displaying a single camera. When using the high resolution streams, random corruption occurs (Picture mailed to you). Changing the URL clears this corruption after a few switches back and forth between the substream. I've seen this before in VLC and FFMpeg but it normally clears quickly after the stream has started. Any thoughts?

[MindrustUK]

I've also managed to capture the corruption detailed above in the output logs as per the following gist: Link

[iEvgeny]

On the first bug, unfortunately, I could not reproduce it, and the output does not contain the necessary information. I need a so-called stack-trace. Ideally I would ask you to set ffmpeg dev-libraries, Qt 5.9.5, Qt Creator and build and run the program in debug mode. It is not difficult and I could prepare instructions for you. As a result we will get a pointer to an exception in the source code, which will save a lot of time.

As for the graphical corruption, it is related to the loss of UDP packets. As you correctly noticed, this happens in both VLC and ffmpeg, but in CCTV Viewer these artifacts are more common. This is due to the fact that for ease of implementation in CCTV Viewer, frame decoding takes place in the same tread as demultiplexing. Right now you can radically solve this problem by using the -rtsp_transport tcp option. Proper implementation of decoding in a separate thread will take too much time.

[MindrustUK]

If you can give me some instructions on building and debugging, I'll be glad to give it a go.

Second issue, setting the TCP flag strangely doesn't seem to fix the issue reliably. Are there additional setting I could try on this one? I totally understand that a re-implementation is too big a job to be realistic in a reasonable amount of time.

[MindrustUK]

Re: second issue, setting; -analyzeduration 0 -probesize 500000 -rtsp_transport tcp in the "AVFormat Options" resolved this. I don't know why cctv-viewer -rtsp_transport tcp didn't have the same effect? Is there a local override?

[iEvgeny]

CCTV Viewer does not take ffmpeg options at the command line. The collection of ffmpeg options is very big and difficult and it is not possible to implement a full-fledged parser. However, this is not necessary. You can edit a set of options that will be applied by default to all viewports in the CCTV Viewer config file. As example: defaultAVFormatOptions="{\"analyzeduration\":0,\"probesize\":500000,\"rtsp_transport\":\"tcp\"}"

P.S. I also thought that implementing decoding in a separate thread could further reduce playback latency, which was the main goal of the project. I suppose the necessary implementation will appear within a few days.

P.P.S I will also prepare debugging instructions a little later.

[iEvgeny]

Currently, the version with decoding, implemented in a separate thread, in the process of automatic assembly. In my case, the problem of frame corruption with RTSP over UDP has been solved. Memory usage is now slightly increased. This is how it should be. As a regression, there are problems with local files playback. This will be fixed in the near future. Please also check the program crash problem. Major changes have been made to the codebase. Before debugging we need to make sure that the problem is relevant.

[MindrustUK]

Just updated from your PPA, it looks like the RTSP over UDP corruption is still happening. The crash on camera 5 also continues.

[iEvgeny]

Are you using the latest version of CCTV Viewer? The build has just finished...

[MindrustUK]

Just updated and retrieved: 0.1.7+git202009181213.1bc99ea+build202009181221 UDP recovery is really quick, seems to have solved that bug. The camera 5 crash remains an issue.

[iEvgeny]

Well, so far I see behavior completely similar to ffplay... Let's tackle the 5 camera crash problem for now. At this stage, we will use noninteractive debugging with GDB. Please run this command in terminal and send me the output after crash: gdb -batch -ex "run" -ex "bt" cctv-viewer 2>&1 | grep -v ^"No stack."$

[MindrustUK]

Output from the crash dump can be found here: Link

[iEvgeny]

This is a very good hint! I have to think about a solution to this problem. Thanks!

[iEvgeny]

I found some peculiarities of ffmpeg memory alignment. Please try it now.

[MindrustUK]

Still crashing I'm afraid dump here: Link

[iEvgeny]

The problematic function is localized. Somewhere in it, there is an overflow of memory. A debugger in non-interactive mode is useless in this case. I have expanded the debug output. Please run CCTV Viewer as follows: export VERBOSE_LEVEL=3 && cctv-viewer The resulting output should give us a hint about the size of the allocated memory for the video frame buffer.

[MindrustUK]

As requested, dump here: Link

[iEvgeny]

Yes. The problem is clearly visible here. FFmpeg still provides undocumented alignment for non-standard resolutions... Let's try to fix it now.

[iEvgeny]

I made some changes. I think now the solution should be around the corner. Please try it again.

[MindrustUK]

Looks like that's done the job! Many thanks for investigating and fixing!