3x3 Layout, last 2 cams take much longer to load?

[adutra22]

Using the 2.0 Beta 9: We're attempting to run multiple cams in 3x3 layouts (so it rotates 9 cams in every 30 seconds). We noticed that the first 7 cam's of the rotation load all within a second. The last 2 cams (row 3, columns 2 and 3) always take between 3-5 seconds to load, and sometimes one or both never load (sits on the "connecting..." screen. We have it going through several rotations, and several different cameras hit those two spots, and they all produce the behavior. So it's not the cameras.

We're using a Pi 3 B+ (wired LAN). GPU set to 640 (originally at 512, but upped it to see if it made a difference. Which it didnt based on what we could see). CPU Cores all sit around 10-40%. RAM bounces around 170-250. Duration set at 30 seconds (originally at 20, upped to 30 didnt seem to make a difference). Cams: Ubiquiti G3's, all hardwired. RTSP Resolution feed set to "Low"

Thoughts/suggestions? Thanks!!! Albert

[SvenVD]

Yes, this is most probably due to the short rotation period of 30 seconds and the large amount of streams for one screen(3x3=9 streams is maxing out the raspberrypi). Rpisurv 2.0 warms the next screen offscreen, so when the rotation happens, it happens quite fast. Now, when rpisurv is still warming the new screen offscreen and a rotation needs to happen when it is still in progress warming the screen, it will display the new screen onscreen nevertheless. The result of that is that it keeps on loading the camerastreams that weren't loaded yet, but this time the loading happens visible onscreen. So I suggest you to up your rotation to one minute or so OR lower the number of streams per screen, or a combination of both. And see if that changes anything.

[adutra22]

Thanks for the quick response.

Bumped the Timeout to 60, but it seemed worse. "most" of the time it never connected, rather than connecting most times, but only failing once a while. I changed it down to 20 seconds, and that actually seems better (i pretty consistently get 8 of 9 cam feeds, just that 8th screen takes an extra 3-5 seconds to display). Set each camera with a probe_timeout of 10, but no luck either. Resources on the Pie seem to have plenty leftover.

Perhaps we are hitting some sort of load limit on omxplayer? since we're attempting to pull in a total of 18 feeds (9 on screen, 9 off screen).

is there any log that we can see to perhaps tell if that 8th or 9th camera feed is erroring out? not connecting? etc?

[SvenVD]

the logs are located at /usr/local/bin/rpisurv/logs/. Can you send me the main.log? Did you try a reboot. Changing the rotation timeout should not have any impact on streams ability to connect.

[adutra22]

Gotcha. I think I did restart. So that was a false positive then.

I've attached the main.log file for review.

main.log

[SvenVD]

2 things already: the raspberry pi is reporting out of gpu memory frequently and secondly at around 14.48 there was a network problem.

But the problem you are seeing is caused by this:

2018/09/14 14:51:23 - l_default - DEBUG - DBusConnection: The bus: org.mpris.MediaPlayer2.screen1_cam_stream8 is NOT ready to accept videopos commands: org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying

Because of the raspberrypi overload connecting to dbus times out and thus also rpisurv ability to control this stream.

Can you test by increasing the timeout value in https://github.com/SvenVD/rpisurv/blob/v2.0_branch/surveillance/core/util/dbus_connection.py line 49 to something like "10". Anyways you are flirting with the hardware limits. Note that the offscreen cached screens barely use up resources.

[adutra22]

Thanks again for the reply/assistance. I understand we're pushing the limits, and hopefully can find a happy medium.

I bumpted the dbus_connection.py timeout to 10, and also bumped the GPU to 750Mb.

Attached is the latest log file. main_AD.log

[SvenVD]

And did the changes change something for the better, I still see dbus connection problems in your latest log? Please also do check if your destination 10.200.0.6 can handle 18 simultaneous streams. If it is not able to handle this, you might need to disable caching, see example config file:

By default rpisurv will cache the next screen in the rotation (if more than one screen is configured), to speed up rotation events.

However, this action takes up some resources of the raspberry pi.

If you do not mind a slower rotate event (slower building up of the screen during a rotate event) and you want to free up some resources, then set this to False

This will also speed up first start since no cached screen needs to be started at boot time.

cache_next_screen: False

Can you also send me the log of the separate cam_streams, maybe those show something interesting

[adutra22]

Sorry for the delayed response. Changing the dbus_Connection.py timeout to 10 and bumping the GPU to 750 didnt make a noticeable difference.

Our destination can certainly handle more than 18 streams (i've got several Pi's testing different variations of the screen layouts all working simultaneously, only the Pi with the 9 cam rotation is the one that has issues. No matter if its the only one running or if they are all running).

I did turn off the cache, and it loads them all, albeit slow (as expected). But that wont work for us.

How/where would I find the cam individual logs? It's not always the same cam's that are having the problem (Looking at it right now, cam 7 and 9 were off. It rotated an the next screen had 8 and 9 off. It rotated again and only 9 was off, Next rotation had all 9, next rotation had cam 3 off).

[SvenVD]

Hi, in your latest log I still saw the timeouts occurring. You can increase the timeout even further, but then it will start conflicting with your fast rotation. My fear is the raspberry can't cope with what you are requesting. Did you try with less streams per screen? Like 3 screens with 6 streams?

[adutra22]

OK. I'll play with the timeouts more. Looking at the logs today, I noticed the GPU had 500-600Mb free, According to 'htop', CPU's were just around 15-25% and RAM was around 180Mb out of 250Mb. So its strange if its a limitation of the Pie. I'm thinking more of a limitation in OMXPlayer?

I can run 8 streams per rotation just fine. Its once I had the 9th (on a 2 rotation (or more)). 3 rotation of 6 screens works fine too. We were trying to get 9 streams on the rotations to maximize visibility to as many camera's as possible.

[SvenVD]

Check the following: https://www.raspberrypi.org/forums/viewtopic.php?p=1061684&sid=471a6a2a7d8fd82e9d5643e2b325ff50#p1061684 It seems to be a limit of the hardware which could be "theoretically" circumvented by a modified omxplayer

Each instance of omxplayer initialises a new connection to the GPU. There is a limit at (IIRC) 8 connections, so trying to run the 9th will fail. Write a modified version that omxplayer that initialises one connection but has multiple decoders and renderers and you can happily have more decoded streams (up to the hardware limits). This is nicoske's issue.

[adutra22]

Interesting.... We can do 9 at once, so i wonder if that 8 connection limit is no longer valid... or maybe its simply doubled (16, since that seems to be the threshold we can't overcome). OR, its launching two instances of omxplayer with 8 feeds within each instance, and it can't do a 3rd instance.

[SvenVD]

Every stream has it's own watchdogged omxplayer instance. I am afraid I can't help you further. Maybe we should redirect this question to the omxplayer guys

[adutra22]

Thank you very much for your time and help in trying to figure this one out! I appreciate it.