search

Anonymousdog / displaycameras

System for displaying RTSP feeds from IP cameras on the Raspberry Pi
Apache License 2.0
568 stars 114 forks source link

Screen Images Freezes after 30ish minutes #39

Closed Nicarlo closed 2 years ago

Nicarlo commented 4 years ago

Describe the bug After about 30-45 minutes of runtime the images freeze up. I checked htop and there is no resource exhaustion happening however I do see the omxplayer.bin rtsp running in the process list. If I check the network traffic via something like iftop or vnstat you can see that the traffic has dropped dramatically to the point where i feel comfortable saying that the traffic received is related to the typicaly idle noise you'd see.

Some things that I've tried to do to troubleshoot:

  1. Attempted to restart the NVR to see if it was related to the NVR
  2. Ran a single RTSP feed from the debug omxplayer mode as found in the READM.md
  3. Changed the power supply of the PI to ensure that it wasnt related to power.

To Reproduce Steps to reproduce the behavior:

  1. Install per README.md
  2. Configure my feeds in layout.conf.default
  3. Run 'sudo systemctl restart displaycameras.service'
  4. Wait about 30-45 minutes until the images freeze
  5. Run 'sudo systemctl restart displaycameras.service' again to restart the sessions.

Expected behavior The images should not freeze up

Raspberry Pi (please complete the following information):

**Display Configuration

Camera information (please complete the following information):

Features enabled (please complete the following information):

NVR information (please complete the following information):

Anonymousdog commented 4 years ago

Please, post your layout.conf.default file (redacted as needed).

Please, recreate the symptoms, then run the following in an SSH client (e.g., putty), 'sudo /usr/bin/displaycameras status'. That yields interesting information from the various omxplayer instances (via DBUS).

Thanks, Andy

On Sun, Feb 2, 2020 at 2:03 PM Nicarlo notifications@github.com wrote:

Describe the bug After about 30-45 minutes of runtime the images freeze up. I checked htop and there is no resource exhaustion happening however I do see the omxplayer.bin rtsp running in the process list. If I check the network traffic via something like iftop or vnstat you can see that the traffic has dropped dramatically to the point where i feel comfortable saying that the traffic received is related to the typicaly idle noise you'd see.

Some things that I've tried to do to troubleshoot:

  1. Attempted to restart the NVR to see if it was related to the NVR
  2. Ran a single RTSP feed from the debug omxplayer mode as found in the READM.md
  3. Changed the power supply of the PI to ensure that it wasnt related to power.

To Reproduce Steps to reproduce the behavior:

  1. Install per README.md
  2. Configure my feeds in layout.conf.default
  3. Run 'sudo systemctl restart displaycameras.service'
  4. Wait about 30-45 minutes until the images freeze
  5. Run 'sudo systemctl restart displaycameras.service' again to restart the sessions.

Expected behavior The images should not freeze up

Raspberry Pi (please complete the following information):

  • Hardware model: Raspberry Pi 3 Model B
  • OS: Rasbian Buster Lite
  • Console: Text Console
  • gpu memory split: 256MB
  • displaycameras Version 0.8.3.3

**Display Configuration

  • Display/Monitor resolution: 1440 x 900
  • Window matrix: 4x4
  • Feed resolution: 704 x 480, Compression H.264H, 15 FPS, 256/Kb/S Bit Rate, Bit Rate Type CBR

Camera information (please complete the following information):

  • Number of cameras: 4
  • Camera models (if not Ubiquiti): Dalhua 2 x 4K cameras and 2 x 4MP Cameras
  • Managed

Features enabled (please complete the following information):

  • Rotation: no
  • Display Blanking: no
  • Display Detection: no

NVR information (please complete the following information):

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Anonymousdog/displaycameras/issues/39?email_source=notifications&email_token=AHIYIKL3Q7SFJDFR7PYORU3RA4KIXA5CNFSM4KO3NPZ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKNWUIA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIYIKOZEHYW5K23TRZPJVLRA4KIXANCNFSM4KO3NPZQ .

Nicarlo commented 4 years ago

Hey Andy,

thanks for reaching out

Here's the layout.conf.default

# This is the camera feed/windows layout configuration file for the
# displaycameras service.  It ONLY configures the layout and feeds for
# the cameras; the rest of the configuration is in displaycameras.conf.
# See the comments in that file for notes on configuring the below.

# This example defines seven 1/2-HD windows, three of which are off-screen to the right,
# through which the service rotates six camera feeds (it actually uses only six windows)
# on a full-HD monitor.  If this suites your needs, modify only the camera names to taste
# and feed URLs to what your cameras or NVR provides.

# Window names

# 2x2 screen with 3 off-screen windows
windows=(upper_left upper_right lower_left lower_right)
# Make sure to account for each window above in the list below.

# Windows positions

window_positions=(
#First Row
#upper_left
# 960x540
"0 0 719 449" \
#upper_right
# 960x540
"720 0 1439 449" \

#Second Row (missing all but the far right window because large_left is double size
#lower_left
# 960x540
"0 450 719 899" \
#lower_right
# 960x540
"720 450  1439 899" \

#off-screen
# 960x540 window just off-screen to the right
"1920 0 2879 539" \
# 960x540 window just below the other
"1920 540 2879 1079" \
# 960x540 window just off-screen to the left
"2880 0 3839 539" \
)

# Camera Names

camera_names=(NE SE South SW )
# Make sure to account for each camera above in the list of feeds below.

# Camera Feeds

camera_feeds=( \
# Mid-Res if your RPi can handle the load
# "rtsp://xxx.xxx.xxx.xxx/yyyyy_1" \
# Low-Res otherwise
# "rtsp://xxx.xxx.xxx.xxx/yyyyy_2" \
#Front Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1" \
#Driveway
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1" \
#Side Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1" \
#Street View
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1" \
#West
"<Camera stream URL>" \
#Vestibule
"<Camera stream URL>" \
)

# Are we rotating cameras through the window matrix? (default false if not set here)
# rotate="true"

output of the sudo /usr/bin/displaycameras status

pi@raspberrypi:~ $ sudo /usr/bin/displaycameras status
NE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1
at 15485sec
SE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1
at 15726sec
South is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1
at 15720sec
SW is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1
at 15713sec
chienlaid commented 4 years ago

I'm having the same problem, it freezes every 2 hours. My setup is exacly the same as Nicarlo. Using a LOREX NVR with same type of rtsp stream. If I do a restart everything is good for another couple hours. Stream is at low res H.264 352x240 512kbps....i've tried multiple resolutions with the same result.

Anonymousdog commented 4 years ago

That is the output of the sudo /usr/bin/displaycameras status command AFTER your display has frozen? If so, it appears that omxplayer still thinks it's playing your feeds. If you run that command multiple times in relatively quick succession (after things freeze), do the stream time positions increase?

The package really just wraps omxplayer in enough management to call is a service. Your setup just runs four instances of omxplayer concurrently and restarts them if they report as not playing. That's it.

The only "rule" you're breaking is your running downscaling. It doesn't matter that 480 is just a little bigger than 450, it still forces real-time downscaling in software. To test this hypothesis, you can hand code the omxplayer option changes in the main script (to run cropped videos) or use a 1080p display (and layout file).

Thanks, Andy

On Tue, Feb 4, 2020, 9:23 AM Nicarlo notifications@github.com wrote:

Hey Andy,

thanks for reaching out

Here's the layout.conf.default

This is the camera feed/windows layout configuration file for the

displaycameras service. It ONLY configures the layout and feeds for

the cameras; the rest of the configuration is in displaycameras.conf.

See the comments in that file for notes on configuring the below.

This example defines seven 1/2-HD windows, three of which are off-screen to the right,

through which the service rotates six camera feeds (it actually uses only six windows)

on a full-HD monitor. If this suites your needs, modify only the camera names to taste

and feed URLs to what your cameras or NVR provides.

Window names

2x2 screen with 3 off-screen windows

windows=(upper_left upper_right lower_left lower_right)

Make sure to account for each window above in the list below.

Windows positions

window_positions=(

First Row

upper_left

960x540

"0 0 719 449" \

upper_right

960x540

"720 0 1439 449" \

Second Row (missing all but the far right window because large_left is double size

lower_left

960x540

"0 450 719 899" \

lower_right

960x540

"720 450 1439 899" \

off-screen

960x540 window just off-screen to the right

"1920 0 2879 539" \

960x540 window just below the other

"1920 540 2879 1079" \

960x540 window just off-screen to the left

"2880 0 3839 539" \ )

Camera Names

camera_names=(NE SE South SW )

Make sure to account for each camera above in the list of feeds below.

Camera Feeds

camera_feeds=( \

Mid-Res if your RPi can handle the load

"rtsp://xxx.xxx.xxx.xxx/yyyyy_1" \

Low-Res otherwise

"rtsp://xxx.xxx.xxx.xxx/yyyyy_2" \

Front Door

"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1" \

Driveway

"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1" \

Side Door

"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1" \

Street View

"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1" \

West

"" \

Vestibule

"" \ )

Are we rotating cameras through the window matrix? (default false if not set here)

rotate="true"

output of the sudo /usr/bin/displaycameras status

pi@raspberrypi:~ $ sudo /usr/bin/displaycameras status NE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1 at 15485sec SE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1 at 15726sec South is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1 at 15720sec SW is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1 at 15713sec

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Anonymousdog/displaycameras/issues/39?email_source=notifications&email_token=AHIYIKKL7SH2J537LFBQ6EDRBF25TA5CNFSM4KO3NPZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKXZN3Y#issuecomment-581932783, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIYIKPV2LVR4ALTIGI3GOLRBF25TANCNFSM4KO3NPZQ .

Nicarlo commented 4 years ago

@Anonymousdog

the sudo /usr/bin/displaycameras status command returned results that were being incremented every time i would run the command.

After reading your post I started instead focusing my attention on omxplayer. After making changes to the resolution of the substream on the NVR, the feeds started playing again without me having to restart the session.

@chienlaid try playing around with the resolution on your lorex NVR and let us know if this also fixes your issue

Nicarlo commented 4 years ago

A little update on this.

I must have spoken to soon because the issue started happening again but this time after a longer time.

I did ran the following command directly to get a log output of the omxplayer instance for a single feed. sudo omxplayer --no-keys --fps 30 --refresh --video_queue 4 --avdict rtsp_transport:tcp "rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=2" --live -n -1 --timeout 0 -g

As @Anonymousdog mentioned his script is only a wrapper to make it easier to manage the omxplayer so I am convinced this is an issue with the omxplayer.

For those interested here is the last few lines of the log output before the feed freezes

23:22:23 T:18446744071756935099   DEBUG: Normal M:7352767473 (A:0 V:47459890844) P:0 A:-7352.77 V:40107.12/T:0.70 (0,0,1,1) A:0% V:99% (0.00,0.00)
23:22:23 T:18446744071756935183   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:0
23:22:23 T:18446744071756935226   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:1
23:22:23 T:18446744071756935609   DEBUG: Live: 40109.27 (40107.12) S:1.010 T:0.70
23:22:23 T:18446744071756956109   DEBUG: Normal M:7352788690 (A:0 V:47459890844) P:0 A:-7352.79 V:40107.10/T:0.70 (0,0,1,1) A:0% V:99% (0.00,0.00)
23:22:23 T:18446744071756956193   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:0
23:22:23 T:18446744071756956237   DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:1
23:22:23 T:18446744071756956617   DEBUG: Live: 40109.25 (40107.10) S:1.010 T:0.70
23:22:23 T:18446744071756966868   DEBUG: OMXClock::OMXStop
23:22:23 T:18446744071756978203   DEBUG: OMXThread::Run - Exited thread with  id -1389367616
23:22:23 T:18446744071756978746   DEBUG: OMXThread::StopThread - Thread stopped
23:22:23 T:18446744071756990474   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_scheduler handle 0xac900b40
23:22:23 T:18446744071757010371   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_decode handle 0xb4e860
23:22:23 T:18446744071757012491   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_render handle 0xac900688
23:22:23 T:18446744071757013160   ERROR: COMXPlayer::interrupt_cb - Timed out
23:22:23 T:18446744071757020050   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.clock handle 0xb5a090

Additionally I wanted to rule out that it wasnt an issue with my NVR so I initiated a stream from another computer using VLC and left it running overnight. When I returned the stream was still working properly which furthers my thinking that its an issue with omxplayer.

Lastly, I had a raspberry pi 4 4GB version laying around so I loaded that unit with raspbian lite and went through the installation of the displaycamera script and copied the old layout.default configuration to this unit. After some time, I still continue get a frozen feed.

some poking around on the omxplayer github issue page I've come across quite a few threads with people reporting similar issues.

I've listed some of them below but have not yet tried them all

https://github.com/popcornmix/omxplayer/issues/555

https://stackoverflow.com/questions/37053495/omxplayer-freezes-when-playing-video

I will continue troubleshooting this issue and will let you know what I do find.

Nicarlo commented 4 years ago

Alright another update on this.

The current maintainer of omxplayer is sayingthat omxplayer will be depreciated over vlc see https://github.com/popcornmix/omxplayer/issues/752#issuecomment-558788119

Now I did try running the same rtsp stream over vlc and I had no issues with it. @Anonymousdog , any plans to do the same as you did for omxplayer with vlc?

Anonymousdog commented 4 years ago

Not saying I won't do it, but I have no plans.

On Fri, Feb 7, 2020 at 8:42 PM Nicarlo notifications@github.com wrote:

Alright another update on this.

The current maintainer of omxplayer is sayingthat omxplayer will be depreciated over vlc see popcornmix/omxplayer#752 (comment) https://github.com/popcornmix/omxplayer/issues/752#issuecomment-558788119

Now I did try running the same rtsp stream over vlc and I had no issues with it. @Anonymousdog https://github.com/Anonymousdog , any plans to do the same as you did for omxplayer with vlc?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Anonymousdog/displaycameras/issues/39?email_source=notifications&email_token=AHIYIKJFGOFZBZOA6JNJJ33RBYEZPA5CNFSM4KO3NPZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFFSJA#issuecomment-583686436, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIYIKP2ZWTGB4I2EPLGJH3RBYEZPANCNFSM4KO3NPZQ .

AnonWSU86 commented 4 years ago

@Nicarlo - Have you had any luck since tinkering with this back in Feb? I am experiencing the same issue you described above. The image freezes and it takes a restart of the service to get it back online. sudo systemctl restart displaycameras.service

I have yet to try tinkering with any of the timing settings, could those make a difference?

I currently have a cron job that just restarts the service every 30 minutes. But I still seem to catch the stream frozen from time to time.

@Anonymousdog - Thank you for taking the time to put this together. It is a very neat solution. I would be very interested if you do the same with VLC.

Nicarlo commented 4 years ago

@AnonWSU86 I ended up settling for a single stream on vlc. Vlc does have the ability to do something similar however their documentation has gaps in it and no one appears to know how to get it working.

As @Anonymousdog mentioned this isn’t an issue with displaycameras but instead an issue with the omxplayer that it uses. I did check out their gitlab and omxplayer will eventually get depreciated for vlc so there are no plans for them to put out a fix for it.