Closed Nicarlo closed 2 years ago
Please, post your layout.conf.default file (redacted as needed).
Please, recreate the symptoms, then run the following in an SSH client (e.g., putty), 'sudo /usr/bin/displaycameras status'. That yields interesting information from the various omxplayer instances (via DBUS).
Thanks, Andy
On Sun, Feb 2, 2020 at 2:03 PM Nicarlo notifications@github.com wrote:
Describe the bug After about 30-45 minutes of runtime the images freeze up. I checked htop and there is no resource exhaustion happening however I do see the omxplayer.bin rtsp running in the process list. If I check the network traffic via something like iftop or vnstat you can see that the traffic has dropped dramatically to the point where i feel comfortable saying that the traffic received is related to the typicaly idle noise you'd see.
Some things that I've tried to do to troubleshoot:
- Attempted to restart the NVR to see if it was related to the NVR
- Ran a single RTSP feed from the debug omxplayer mode as found in the READM.md
- Changed the power supply of the PI to ensure that it wasnt related to power.
To Reproduce Steps to reproduce the behavior:
- Install per README.md
- Configure my feeds in layout.conf.default
- Run 'sudo systemctl restart displaycameras.service'
- Wait about 30-45 minutes until the images freeze
- Run 'sudo systemctl restart displaycameras.service' again to restart the sessions.
Expected behavior The images should not freeze up
Raspberry Pi (please complete the following information):
- Hardware model: Raspberry Pi 3 Model B
- OS: Rasbian Buster Lite
- Console: Text Console
- gpu memory split: 256MB
- displaycameras Version 0.8.3.3
**Display Configuration
- Display/Monitor resolution: 1440 x 900
- Window matrix: 4x4
- Feed resolution: 704 x 480, Compression H.264H, 15 FPS, 256/Kb/S Bit Rate, Bit Rate Type CBR
Camera information (please complete the following information):
- Number of cameras: 4
- Camera models (if not Ubiquiti): Dalhua 2 x 4K cameras and 2 x 4MP Cameras
- Managed
Features enabled (please complete the following information):
- Rotation: no
- Display Blanking: no
- Display Detection: no
NVR information (please complete the following information):
- NVR hardware Dahua N41B2P3 ( see https://us.dahuasecurity.com/wp-content/uploads/2018/09/Datasheet_4MP_4K_8ch_4K_Network_Security_System_N488D63_v001_001_FINAL.pdf )
- NVR UniFi Video server version: System Version: 3.216.0000000.0, Build Date: 2018-06-05
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Anonymousdog/displaycameras/issues/39?email_source=notifications&email_token=AHIYIKL3Q7SFJDFR7PYORU3RA4KIXA5CNFSM4KO3NPZ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKNWUIA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIYIKOZEHYW5K23TRZPJVLRA4KIXANCNFSM4KO3NPZQ .
Hey Andy,
thanks for reaching out
Here's the layout.conf.default
# This is the camera feed/windows layout configuration file for the
# displaycameras service. It ONLY configures the layout and feeds for
# the cameras; the rest of the configuration is in displaycameras.conf.
# See the comments in that file for notes on configuring the below.
# This example defines seven 1/2-HD windows, three of which are off-screen to the right,
# through which the service rotates six camera feeds (it actually uses only six windows)
# on a full-HD monitor. If this suites your needs, modify only the camera names to taste
# and feed URLs to what your cameras or NVR provides.
# Window names
# 2x2 screen with 3 off-screen windows
windows=(upper_left upper_right lower_left lower_right)
# Make sure to account for each window above in the list below.
# Windows positions
window_positions=(
#First Row
#upper_left
# 960x540
"0 0 719 449" \
#upper_right
# 960x540
"720 0 1439 449" \
#Second Row (missing all but the far right window because large_left is double size
#lower_left
# 960x540
"0 450 719 899" \
#lower_right
# 960x540
"720 450 1439 899" \
#off-screen
# 960x540 window just off-screen to the right
"1920 0 2879 539" \
# 960x540 window just below the other
"1920 540 2879 1079" \
# 960x540 window just off-screen to the left
"2880 0 3839 539" \
)
# Camera Names
camera_names=(NE SE South SW )
# Make sure to account for each camera above in the list of feeds below.
# Camera Feeds
camera_feeds=( \
# Mid-Res if your RPi can handle the load
# "rtsp://xxx.xxx.xxx.xxx/yyyyy_1" \
# Low-Res otherwise
# "rtsp://xxx.xxx.xxx.xxx/yyyyy_2" \
#Front Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1" \
#Driveway
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1" \
#Side Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1" \
#Street View
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1" \
#West
"<Camera stream URL>" \
#Vestibule
"<Camera stream URL>" \
)
# Are we rotating cameras through the window matrix? (default false if not set here)
# rotate="true"
output of the sudo /usr/bin/displaycameras status
pi@raspberrypi:~ $ sudo /usr/bin/displaycameras status
NE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1
at 15485sec
SE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1
at 15726sec
South is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1
at 15720sec
SW is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1
at 15713sec
I'm having the same problem, it freezes every 2 hours. My setup is exacly the same as Nicarlo. Using a LOREX NVR with same type of rtsp stream. If I do a restart everything is good for another couple hours. Stream is at low res H.264 352x240 512kbps....i've tried multiple resolutions with the same result.
That is the output of the sudo /usr/bin/displaycameras status command AFTER your display has frozen? If so, it appears that omxplayer still thinks it's playing your feeds. If you run that command multiple times in relatively quick succession (after things freeze), do the stream time positions increase?
The package really just wraps omxplayer in enough management to call is a service. Your setup just runs four instances of omxplayer concurrently and restarts them if they report as not playing. That's it.
The only "rule" you're breaking is your running downscaling. It doesn't matter that 480 is just a little bigger than 450, it still forces real-time downscaling in software. To test this hypothesis, you can hand code the omxplayer option changes in the main script (to run cropped videos) or use a 1080p display (and layout file).
Thanks, Andy
On Tue, Feb 4, 2020, 9:23 AM Nicarlo notifications@github.com wrote:
Hey Andy,
thanks for reaching out
Here's the layout.conf.default
This is the camera feed/windows layout configuration file for the
displaycameras service. It ONLY configures the layout and feeds for
the cameras; the rest of the configuration is in displaycameras.conf.
See the comments in that file for notes on configuring the below.
This example defines seven 1/2-HD windows, three of which are off-screen to the right,
through which the service rotates six camera feeds (it actually uses only six windows)
on a full-HD monitor. If this suites your needs, modify only the camera names to taste
and feed URLs to what your cameras or NVR provides.
Window names
2x2 screen with 3 off-screen windows
windows=(upper_left upper_right lower_left lower_right)
Make sure to account for each window above in the list below.
Windows positions
window_positions=(
First Row
upper_left
960x540
"0 0 719 449" \
upper_right
960x540
"720 0 1439 449" \
Second Row (missing all but the far right window because large_left is double size
lower_left
960x540
"0 450 719 899" \
lower_right
960x540
"720 450 1439 899" \
off-screen
960x540 window just off-screen to the right
"1920 0 2879 539" \
960x540 window just below the other
"1920 540 2879 1079" \
960x540 window just off-screen to the left
"2880 0 3839 539" \ )
Camera Names
camera_names=(NE SE South SW )
Make sure to account for each camera above in the list of feeds below.
Camera Feeds
camera_feeds=( \
Mid-Res if your RPi can handle the load
"rtsp://xxx.xxx.xxx.xxx/yyyyy_1" \
Low-Res otherwise
"rtsp://xxx.xxx.xxx.xxx/yyyyy_2" \
Front Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1" \
Driveway
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1" \
Side Door
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1" \
Street View
"rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1" \
West
"
" \ Vestibule
"
" \ ) Are we rotating cameras through the window matrix? (default false if not set here)
rotate="true"
output of the sudo /usr/bin/displaycameras status
pi@raspberrypi:~ $ sudo /usr/bin/displaycameras status NE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=1 at 15485sec SE is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=2&subtype=1 at 15726sec South is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=3&subtype=1 at 15720sec SW is Playing rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=6&subtype=1 at 15713sec
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Anonymousdog/displaycameras/issues/39?email_source=notifications&email_token=AHIYIKKL7SH2J537LFBQ6EDRBF25TA5CNFSM4KO3NPZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKXZN3Y#issuecomment-581932783, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIYIKPV2LVR4ALTIGI3GOLRBF25TANCNFSM4KO3NPZQ .
@Anonymousdog
the sudo /usr/bin/displaycameras status
command returned results that were being incremented every time i would run the command.
After reading your post I started instead focusing my attention on omxplayer. After making changes to the resolution of the substream on the NVR, the feeds started playing again without me having to restart the session.
@chienlaid try playing around with the resolution on your lorex NVR and let us know if this also fixes your issue
A little update on this.
I must have spoken to soon because the issue started happening again but this time after a longer time.
I did ran the following command directly to get a log output of the omxplayer instance for a single feed.
sudo omxplayer --no-keys --fps 30 --refresh --video_queue 4 --avdict rtsp_transport:tcp "rtsp://username:password@x.x.x.x:554/cam/realmonitor?channel=1&subtype=2" --live -n -1 --timeout 0 -g
As @Anonymousdog mentioned his script is only a wrapper to make it easier to manage the omxplayer so I am convinced this is an issue with the omxplayer.
For those interested here is the last few lines of the log output before the feed freezes
23:22:23 T:18446744071756935099 DEBUG: Normal M:7352767473 (A:0 V:47459890844) P:0 A:-7352.77 V:40107.12/T:0.70 (0,0,1,1) A:0% V:99% (0.00,0.00)
23:22:23 T:18446744071756935183 DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:0
23:22:23 T:18446744071756935226 DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:1
23:22:23 T:18446744071756935609 DEBUG: Live: 40109.27 (40107.12) S:1.010 T:0.70
23:22:23 T:18446744071756956109 DEBUG: Normal M:7352788690 (A:0 V:47459890844) P:0 A:-7352.79 V:40107.10/T:0.70 (0,0,1,1) A:0% V:99% (0.00,0.00)
23:22:23 T:18446744071756956193 DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:0
23:22:23 T:18446744071756956237 DEBUG: OMXClock::OMXSetSpeed(1.01) pause_resume:1
23:22:23 T:18446744071756956617 DEBUG: Live: 40109.25 (40107.10) S:1.010 T:0.70
23:22:23 T:18446744071756966868 DEBUG: OMXClock::OMXStop
23:22:23 T:18446744071756978203 DEBUG: OMXThread::Run - Exited thread with id -1389367616
23:22:23 T:18446744071756978746 DEBUG: OMXThread::StopThread - Thread stopped
23:22:23 T:18446744071756990474 DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_scheduler handle 0xac900b40
23:22:23 T:18446744071757010371 DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_decode handle 0xb4e860
23:22:23 T:18446744071757012491 DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.video_render handle 0xac900688
23:22:23 T:18446744071757013160 ERROR: COMXPlayer::interrupt_cb - Timed out
23:22:23 T:18446744071757020050 DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.clock handle 0xb5a090
Additionally I wanted to rule out that it wasnt an issue with my NVR so I initiated a stream from another computer using VLC and left it running overnight. When I returned the stream was still working properly which furthers my thinking that its an issue with omxplayer.
Lastly, I had a raspberry pi 4 4GB version laying around so I loaded that unit with raspbian lite and went through the installation of the displaycamera script and copied the old layout.default configuration to this unit. After some time, I still continue get a frozen feed.
some poking around on the omxplayer github issue page I've come across quite a few threads with people reporting similar issues.
I've listed some of them below but have not yet tried them all
https://github.com/popcornmix/omxplayer/issues/555
https://stackoverflow.com/questions/37053495/omxplayer-freezes-when-playing-video
I will continue troubleshooting this issue and will let you know what I do find.
Alright another update on this.
The current maintainer of omxplayer is sayingthat omxplayer will be depreciated over vlc see https://github.com/popcornmix/omxplayer/issues/752#issuecomment-558788119
Now I did try running the same rtsp stream over vlc and I had no issues with it. @Anonymousdog , any plans to do the same as you did for omxplayer with vlc?
Not saying I won't do it, but I have no plans.
On Fri, Feb 7, 2020 at 8:42 PM Nicarlo notifications@github.com wrote:
Alright another update on this.
The current maintainer of omxplayer is sayingthat omxplayer will be depreciated over vlc see popcornmix/omxplayer#752 (comment) https://github.com/popcornmix/omxplayer/issues/752#issuecomment-558788119
Now I did try running the same rtsp stream over vlc and I had no issues with it. @Anonymousdog https://github.com/Anonymousdog , any plans to do the same as you did for omxplayer with vlc?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Anonymousdog/displaycameras/issues/39?email_source=notifications&email_token=AHIYIKJFGOFZBZOA6JNJJ33RBYEZPA5CNFSM4KO3NPZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFFSJA#issuecomment-583686436, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIYIKP2ZWTGB4I2EPLGJH3RBYEZPANCNFSM4KO3NPZQ .
@Nicarlo - Have you had any luck since tinkering with this back in Feb? I am experiencing the same issue you described above. The image freezes and it takes a restart of the service to get it back online.
sudo systemctl restart displaycameras.service
If I run a system status, it says the stream is running and the # of seconds continue to increase if I keep running the status
If I look at network activity, it still looks the PI is pulling the stream
If I try to repair the status it does NOT resolve the is.
If I kill omxplayer, and then run the repair, this does solve this issue - Is this the same thing as restarting the service?
sudo killall omxplayer.bin
sudo /usr/bin/displaycameras repair
I have tried this with a single camera, full-screen stream on a 800x480 screen.
I have also tried this with a 4 Camera (2x2) stream, one the same 800x480 screen.
I have yet to try tinkering with any of the timing settings, could those make a difference?
I currently have a cron job that just restarts the service every 30 minutes. But I still seem to catch the stream frozen from time to time.
@Anonymousdog - Thank you for taking the time to put this together. It is a very neat solution. I would be very interested if you do the same with VLC.
@AnonWSU86 I ended up settling for a single stream on vlc. Vlc does have the ability to do something similar however their documentation has gaps in it and no one appears to know how to get it working.
As @Anonymousdog mentioned this isn’t an issue with displaycameras but instead an issue with the omxplayer that it uses. I did check out their gitlab and omxplayer will eventually get depreciated for vlc so there are no plans for them to put out a fix for it.
Describe the bug After about 30-45 minutes of runtime the images freeze up. I checked htop and there is no resource exhaustion happening however I do see the omxplayer.bin rtsp running in the process list. If I check the network traffic via something like iftop or vnstat you can see that the traffic has dropped dramatically to the point where i feel comfortable saying that the traffic received is related to the typicaly idle noise you'd see.
Some things that I've tried to do to troubleshoot:
To Reproduce Steps to reproduce the behavior:
Expected behavior The images should not freeze up
Raspberry Pi (please complete the following information):
**Display Configuration
Camera information (please complete the following information):
Features enabled (please complete the following information):
NVR information (please complete the following information):