Cameras lose connection for a second or 2

[ericjanvanputten]

Based on the recommendation to post an issue here (https://github.com/ccrisan/motioneye/issues/329)

I have a pi 3 with motioneyeOS 20161125 with motion 4 with 3 rtsp camera's on it running 1280x720p resolution. They seem to lose connection for a second or 2 frequently. When i reduce the camera's to 1 or 2, the connection issue reduces greatly, but still exists.

I'm unfamiliar with setting the debug mode so please tell me how and i will supply logs.

[tosiara]

Relevant errors from log:

Dec 25 11:07:33 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: negative number of zero coeffs at 43 23
Dec 25 11:07:33 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: error while decoding MB 43 23
Dec 25 14:36:52 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: out of range intra chroma pred mode at 40 24
Dec 25 14:36:52 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: error while decoding MB 40 24
Dec 25 14:36:52 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: mb_type 46 in P slice too large at 67 43
Dec 25 14:36:52 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: error while decoding MB 67 43
Dec 25 14:36:52 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: negative number of zero coeffs at 54 31
Dec 25 14:36:52 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: error while decoding MB 54 31
Dec 25 16:16:02 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: negative number of zero coeffs at 56 40
Dec 25 16:16:02 meye-97bcbe37 user.err motion: [0:av2] [ERR] [ENC] ffmpeg_avcodec_log: error while decoding MB 56 40
Dec 25 16:19:51 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [NET] netcam_handler_loop: Bad image.  Reconnecting with camera....
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused
Dec 25 16:19:53 meye-97bcbe37 user.err motion: [2:nc2] [ERR] [ENC] ffmpeg_avcodec_log: Connection to tcp://192.168.0.18:554?timeout=0 failed: Connection refused

Are you sure you don't have connectivity issues with your Rpi?

[ericjanvanputten]

2 of the cams are wired 1 of them is wifi only

so i can't commit to saying i don't have issues there, but they seem to work fine...

Just ran a ping on 2 cam's and although i saw them dissapear on the pi motion eye os view, the ping was rockstable.

[ericjanvanputten]

yesterday i did do some utp work on the switch so that 16:19 time with the wired cam (.18) might have had to do with that... not 100% sure though.

[tosiara]

Ok, later I see that motion gets into restart loop:

Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] conf_load: Processing thread 0 - config file /data/etc/motion.conf
Dec 26 18:44:05 meye-97bcbe37 user.warn motion: [0:motion] [WRN] [ALL] thread config option deprecated use camera
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-5.conf
Dec 26 18:44:05 meye-97bcbe37 user.warn motion: [0:motion] [WRN] [ALL] thread config option deprecated use camera
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-1.conf
Dec 26 18:44:05 meye-97bcbe37 user.warn motion: [0:motion] [WRN] [ALL] thread config option deprecated use camera
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-3.conf
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Motion 4.0.1+git37b3595 Started
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Logging to syslog
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Using default log type (ALL)
Dec 26 18:44:05 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Using log type (ALL) log level (WRN)
Dec 26 18:44:05 meye-97bcbe37 user.alert motion: [1:ml1] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rtsp://192.168.0.6:554/ch0.h264)
Dec 26 18:44:05 meye-97bcbe37 user.alert motion: [2:ml2] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rtsp://192.168.0.18/channel1)
Dec 26 18:44:05 meye-97bcbe37 user.alert motion: [3:ml3] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rtsp://192.168.0.21/channel1)
Dec 26 18:44:08 meye-97bcbe37 user.alert motion: [2:nc2] [ALR] [NET] netcam_handler_loop: Camera handler thread [5] started
Dec 26 18:44:08 meye-97bcbe37 user.alert motion: [3:nc3] [ALR] [NET] netcam_handler_loop: Camera handler thread [6] started
Dec 26 18:44:10 meye-97bcbe37 user.alert motion: [1:nc1] [ALR] [NET] netcam_handler_loop: Camera handler thread [7] started
Dec 26 18:49:08 meye-97bcbe37 user.alert motion: [3:nc3] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exiting
Dec 26 18:49:08 meye-97bcbe37 user.alert motion: [2:nc2] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exiting
Dec 26 18:49:08 meye-97bcbe37 user.alert motion: [1:nc1] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exiting
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] conf_load: Processing thread 0 - config file /data/etc/motion.conf
Dec 26 18:49:10 meye-97bcbe37 user.warn motion: [0:motion] [WRN] [ALL] thread config option deprecated use camera
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-5.conf
Dec 26 18:49:10 meye-97bcbe37 user.warn motion: [0:motion] [WRN] [ALL] thread config option deprecated use camera
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-1.conf
Dec 26 18:49:10 meye-97bcbe37 user.warn motion: [0:motion] [WRN] [ALL] thread config option deprecated use camera
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-3.conf
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Motion 4.0.1+git37b3595 Started
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Logging to syslog
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Using default log type (ALL)
Dec 26 18:49:10 meye-97bcbe37 user.notice motion: [0:motion] [NTC] [ALL] motion_startup: Using log type (ALL) log level (WRN)
Dec 26 18:49:10 meye-97bcbe37 user.alert motion: [1:ml1] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rtsp://192.168.0.6:554/ch0.h264)
Dec 26 18:49:10 meye-97bcbe37 user.alert motion: [2:ml2] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rtsp://192.168.0.18/channel1)
Dec 26 18:49:10 meye-97bcbe37 user.alert motion: [3:ml3] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rtsp://192.168.0.21/channel1)
Dec 26 18:49:12 meye-97bcbe37 user.alert motion: [3:nc3] [ALR] [NET] netcam_handler_loop: Camera handler thread [5] started
Dec 26 18:49:12 meye-97bcbe37 user.alert motion: [2:nc2] [ALR] [NET] netcam_handler_loop: Camera handler thread [6] started
Dec 26 18:49:14 meye-97bcbe37 user.alert motion: [1:nc1] [ALR] [NET] netcam_handler_loop: Camera handler thread [7] started
Dec 26 18:54:13 meye-97bcbe37 user.alert motion: [3:nc3] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exiting
Dec 26 18:54:13 meye-97bcbe37 user.alert motion: [2:nc2] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exiting
Dec 26 18:54:13 meye-97bcbe37 user.alert motion: [1:nc1] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exiting

Don't know what may be the reason as log level is not verbose enough

Could it be that motion simply fails to write a file? Are you saving on SMB share?

2016-12-26 18:44:03: [motioneye]    ERROR: failed to unmount smb share "//192.168.0.13/Download\134Security_recordings\134Tuin" from "/data/media/motioneye_192_168_0_13_download_134security_recordings_134tuin_admin"

[ccrisan]

@ericjanvanputten if you suspect network connectivity issues, please try using the local storage to save movies and see if the problem persists.

When motion can't write its output files it starts to "complain". motionEye, in turn, detects this as streaming issues between motion and motionEye. If these issues persist, it restarts the motion daemon. Hence the restart loop @tosiara was talking about.

On a side note, enabling debugging in motionEyeOS's UI will also make motion log more verbose.

[ericjanvanputten]

I'm using a smb windows server as a location to save the files. the 134tuin etc is a bit weird should not contain the 134.

I could try local saving, but interestingly enough the issue isnt happening often now i only have 2 of 3 cams active... could it have to do with nic not keeping up? its showing a 50-80% load most of the time...

[ccrisan]

@ericjanvanputten this is getting too complicated.

NIC not keeping up doesn't have (almost) anything to do with system load (or were you actually refering to the network load?).

The fact that you have extra text in your SMB share is also not a motion(Eye(OS)) issue. You have probably mistyped the path on your SMB share, but that shouldn't be a problem, as motionEye tries to mkdir -p the given path.

Again, please try saving locally and please do it in the same configuration you know results in errors (i.e. all 3 cameras). At least that's what I would do.

[ericjanvanputten]

Sorry, I have SSH'ed in and running 'top' and NIC says between 50-80% so the networkload on the pi 3. On the path, videos from motioneye are just making it fine over there so the 134 is a bit strange and i just checked.. seems to be correct in the motioneyeOS path.

I will do the following:

• first try to get masks and motion detection to get a bit more reliably to work :)
• then activate the 3 cameras
• see if that reproduces the video issues again (havent had it today with only 2 cameras active)
• change location of files to sd card for a day or so...

hope that sounds like a plan :)

[ccrisan]

That "nic" is the nice and has nothing to do with the network.

[ericjanvanputten]

crap shows my 'user'ness ;)

[ericjanvanputten]

hi guys, sorry had to go off for work a bit - but back now based on the log of a linksys cam, it seems the system drops connection every 5 minutes-ish

LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:48:08 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:48:04 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:47:53 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:47:51 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:47:40 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:47:37 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:42:45 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:42:43 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:37:39 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:37:37 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:32:33 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:32:31 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:27:28 2017 LOG_INFO-stream :Channel [1] stopped streaming to host '[192.168.0.14]', Tue Jan 10 15:27:25 2017 LOG_INFO-stream :Channel [1] started streaming to host '[192.168.0.14]', Tue Jan 10 15:22:21 2017

happy to run some additional test or something, please tell me how to log, and what to log.

[ericjanvanputten]

adding the normal log files they also seem to show that motion is 'rebooted' every 5 minutes - ish. motion5minutescamerabreak.zip

happy to run extended logging based on this thread -> https://github.com/Motion-Project/motion/issues/281#issuecomment-269614326

[tosiara]

@ericjanvanputten have you tried running motion standalone?

[ericjanvanputten]

@tosiara no not yet, but I'm to much of a noob/user to understand how to do that. happy to ssh in there and then copy paste anything you recommend :)

[ericjanvanputten]

would be great to see this 'resolved' happy to help, but need some help with it. not sure where the issue is (camera/other)

[tosiara]

Let me help you Can you try to open a terminal on your motion box and run the following commands:

/etc/init.d/S85motioneye stop
cd /data/etc
motion -c motion.conf -d 5 -n

What output do you get?

[ericjanvanputten]

I'm getting this:

[root@meye-97bcbe37 etc]# motion -c motion.conf -d 5 -n [0:motion] [NTC] [ALL] conf_load: Processing thread 0 - config file motion.conf [0:motion] [WRN] [ALL] thread config option deprecated use camera [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-3.con f [0:motion] [WRN] [ALL] thread config option deprecated use camera [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-1.con f [0:motion] [WRN] [ALL] thread config option deprecated use camera [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-5.con f [0:motion] [NTC] [ALL] motion_startup: Motion 4.0.1+git37b3595 Started [0:motion] [NTC] [ALL] motion_startup: Logging to syslog [0:motion] [NTC] [ALL] motion_startup: Using default log type (ALL) [0:motion] [NTC] [ALL] motion_startup: Using log type (ALL) log level (WRN) [2:ml2] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rts p://192.168.0.18/channel1) [3:ml3] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rts p://192.168.0.6:554/ch0.h264) [1:ml1] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rts p://192.168.0.21/channel1) [1:nc1] [ALR] [NET] netcam_handler_loop: Camera handler thread [5] started [2:nc2] [ALR] [NET] netcam_handler_loop: Camera handler thread [6] started [3:nc3] [ALR] [NET] netcam_handler_loop: Camera handler thread [7] started ^C[2:nc2] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, ex iting [3:nc3] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exit ing [1:nc1] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exit ing curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

hope it helps :)

[ericjanvanputten]

this seems cleaner

[root@meye-97bcbe37 etc]# motion -c motion.conf -d 5 -n [0:motion] [NTC] [ALL] conf_load: Processing thread 0 - config file motion.conf [0:motion] [WRN] [ALL] thread config option deprecated use camera [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-3.con f [0:motion] [WRN] [ALL] thread config option deprecated use camera [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-1.con f [0:motion] [WRN] [ALL] thread config option deprecated use camera [0:motion] [NTC] [ALL] config_camera: Processing camera config file thread-5.con f [0:motion] [NTC] [ALL] motion_startup: Motion 4.0.1+git37b3595 Started [0:motion] [NTC] [ALL] motion_startup: Logging to syslog [0:motion] [NTC] [ALL] motion_startup: Using default log type (ALL) [0:motion] [NTC] [ALL] motion_startup: Using log type (ALL) log level (WRN) [2:ml2] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rts p://192.168.0.18/channel1) [3:ml3] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rts p://192.168.0.6:554/ch0.h264) [1:ml1] [ALR] [NET] netcam_start: Network Camera thread starting... for url (rts p://192.168.0.21/channel1) [1:nc1] [ALR] [NET] netcam_handler_loop: Camera handler thread [5] started [2:nc2] [ALR] [NET] netcam_handler_loop: Camera handler thread [6] started [3:nc3] [ALR] [NET] netcam_handler_loop: Camera handler thread [7] started ^C[2:nc2] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, ex iting [3:nc3] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exit ing [1:nc1] [ALR] [NET] netcam_handler_loop: netcam camera handler: finish set, exit ing curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

[ericjanvanputten]

hmm seems to generate more.. let me run it for another 10-15 minutes and i will add a notepad

[tosiara]

Yeah, you can try different logging level, like -d 7 and leave it running for some time until you reproduce the issue. When reproduced - upload the log here we will investigate it

[ericjanvanputten]

see attached... i think its this that is the issue: netcam_interrupt_rtsp: Camera timed out for rtsp://192.168.0.6:554/ch0.h264

this is copy paste: netcam_interrupt_rtsp: Camera timed out for rtsp://192.168.0 .6:554/ch0.h264 (this is a yi wireless cam)

errors.txt

[tosiara]

So motion does not restart itself, its just a timeout issue with the wireless camera?

[ericjanvanputten]

I would love to tell you that... but i think you know that answer better then i do :) what i can do is monitor the rtsp stream on motion (pi) and on my local VLC... (i believe i did that before and the VLC doesnt drop, but the pi does. - but would need to recheck that)

[ericjanvanputten]

But even when i dont activate the wireless stream camera on motion, the other 2 wired cam's fail every 5ish minutes for a couple of seconds

[jogu]

What frame rate are the rtsp streams for the cameras set to? (This is a setting on the camera, not in motion.)

[ericjanvanputten]

camera 1: wired - 1280x720 - frame rate 15 camera 2: wired - 1280x720 - frame rate 10 camera 3: wireless - 1280x720 - frame rate no idea

[tosiara]

What I would also suggest is to capture network traffic and then inspect it on your desktop machine using Wireshark. Capture using tcpdump -i eth0 -w dump.pcap, then copy dump.pcap on your desktop, install Wireshark, open the file and set filter to tcp.analysis.flags. How many "red" packets you see (percents to total captured packets)?

[ericjanvanputten]

Not sure it helps, but here is a quick screen capture.

will work on the network capture

[jogu]

I suspect you're being a little optimistic expecting a raspberry pi to receive, decode and do motion detection on (and presumably also record / snapshot) 3 10fps+ video streams at the same time.

Though there are other explanations too; the way tosiara suggests is the way to narrow down the issue.

[ericjanvanputten]

sorry didnt work, attached now Motion.zip

[ericjanvanputten]

I'm considering a Odroid XU4 as i want to add 1 or 2 more cams :) and i suspected performance reasons being the issue as well.

But if you have tips to loadbalance it somehow, i'm open to it... ideally i would like 1 dashboard showing all cams... but maybe a second pi 3 can handle the motion and recording of 2 cams and a 3rd pi can handle the remaining 2 camps, and the 1st one just web dash? :)

[tosiara]

You can check CPU usage of your Pi while all 3 cameras are running by running: top. If the load is more than 4.0 (or cpu close to 400%) - then it is definitely a resource issue

[ericjanvanputten]

are we talking about the bild parts?

Mem: 742292K used, 10544K free, 136K shrd, 1772K buff, 89796K cached CPU: 50% usr 2% sys 0% nic 45% idle 0% io 0% irq 2% sirq Load average: 4.81 3.51 3.25 6/138 7679

Mem: 741852K used, 10984K free, 136K shrd, 1784K buff, 89096K cached CPU: 62% usr 2% sys 0% nic 32% idle 0% io 0% irq 1% sirq Load average: 4.67 3.52 3.26 7/138 7695

Mem: 742292K used, 10544K free, 136K shrd, 1772K buff, 89796K cached CPU: 50% usr 2% sys 0% nic 45% idle 0% io 0% irq 2% sirq Load average: 4.81 3.51 3.25 6/138 7679

Mem: 741892K used, 10944K free, 136K shrd, 1780K buff, 89096K cached CPU: 69% usr 3% sys 0% nic 24% idle 0% io 0% irq 2% sirq Load average: 4.90 3.55 3.27 4/138 7687

Mem: 741852K used, 10984K free, 136K shrd, 1784K buff, 89096K cached CPU: 62% usr 2% sys 0% nic 32% idle 0% io 0% irq 1% sirq Load average: 4.67 3.52 3.26 7/138 7695

CPU: 36% usr 1% sys 0% nic 61% idle 0% io 0% irq 1% sirq Load average: 4.11 3.44 3.24 3/138 7711

Mem: 741608K used, 11228K free, 136K shrd, 1784K buff, 89096K cached CPU: 35% usr 1% sys 0% nic 61% idle 0% io 0% irq 0% sirq Load average: 3.86 3.40 3.22 4/138 7719

Mem: 741936K used, 10900K free, 136K shrd, 1784K buff, 89096K cached CPU: 41% usr 1% sys 0% nic 55% idle 0% io 0% irq 1% sirq Load average: 3.55 3.34 3.21 4/138 7727

Mem: 742128K used, 10708K free, 136K shrd, 1792K buff, 89096K cached CPU: 39% usr 1% sys 0% nic 57% idle 0% io 0% irq 1% sirq Load average: 3.34 3.30 3.19 5/138 7735

Mem: 741936K used, 10900K free, 136K shrd, 1792K buff, 89096K cached CPU: 35% usr 1% sys 0% nic 61% idle 0% io 0% irq 1% sirq Load average: 3.24 3.28 3.19 4/138 7743

Mem: 379512K used, 373324K free, 136K shrd, 1792K buff, 89096K cached CPU: 12% usr 2% sys 0% nic 84% idle 0% io 0% irq 0% sirq Load average: 2.98 3.23 3.17 2/130 7818

Mem: 277024K used, 475812K free, 136K shrd, 1800K buff, 89060K cached CPU: 11% usr 2% sys 0% nic 86% idle 0% io 0% irq 0% sirq Load average: 2.74 3.17 3.15 14/135 7843

Mem: 533636K used, 219200K free, 136K shrd, 1808K buff, 89060K cached CPU: 42% usr 2% sys 0% nic 53% idle 0% io 0% irq 1% sirq Load average: 2.76 3.17 3.15 2/135 7851

Mem: 705372K used, 47464K free, 136K shrd, 1808K buff, 89060K cached CPU: 45% usr 2% sys 0% nic 50% idle 0% io 0% irq 1% sirq Load average: 2.70 3.15 3.15 5/141 7865

Mem: 705748K used, 47088K free, 136K shrd, 1816K buff, 89060K cached CPU: 37% usr 1% sys 0% nic 59% idle 0% io 0% irq 1% sirq Load average: 2.48 3.10 3.13 6/141 7873

Mem: 705920K used, 46916K free, 136K shrd, 1816K buff, 89060K cached CPU: 37% usr 0% sys 0% nic 60% idle 0% io 0% irq 1% sirq Load average: 2.36 3.06 3.12 7/141 7881

Mem: 705928K used, 46908K free, 136K shrd, 1816K buff, 89060K cached CPU: 36% usr 1% sys 0% nic 60% idle 0% io 0% irq 1% sirq Load average: 2.25 3.03 3.11 2/141 7889

Mem: 705816K used, 47020K free, 136K shrd, 1820K buff, 89060K cached CPU: 36% usr 0% sys 0% nic 60% idle 0% io 0% irq 2% sirq Load average: 2.23 3.01 3.10 2/141 7897

Mem: 705700K used, 47136K free, 136K shrd, 1820K buff, 89060K cached CPU: 35% usr 1% sys 0% nic 61% idle 0% io 0% irq 1% sirq Load average: 2.21 2.99 3.09 3/141 7905

[ericjanvanputten]

It seems to be lower when i don't have the web interface open (duh) :) so could it be because when i check live its running to much resources?

when not open interface it seems to stay well below <3, but when open its near 4... sometimes over if stuff is happening (motion)

[tosiara]

Yes, viewing live stream causes MJPEG encoding which is very expensive on Pi. If the load remains around 3.2 in idle, you will hit an issue as soon as motion tries to record a video. You are hitting the Pi's limit, really. And XU4 wouldn't help in this case as you would be still very close to the limit.

[ericjanvanputten]

Okay :) more powerrrr :)

so probably this setup would be smoother:

1x pi 3 for viewing only interface with 3-4 cams at 1280x720 1x pi 3 for Motion and Recording with 2 cams at 1280x720 1x pi 3 for Motion and Recording with remaining 2 cams at 1280x720

:) or even better, how would you set it up?

(thank you for the troubleshooting btw)

to make sure, im going to deactivate 2 out of 3 cams (leave wireless online) and see what happens :)

[tosiara]

Pi cluster for the win :) Moar Pi! You can also stick with something cheap, like OrangePi Zero and use one board per camera. I'm actually using it this way - one Zero per 720p@5fps

[ericjanvanputten]

Will look into that - interestingly enough, both with only 1 cam activated both the wireless as well as the wired seem to drop after 5ish min for a couple seconds, while load is around 0.5-1.5 it seems...

Now running the log thing again with 1 wired connection.

still have to sniff the traffic

[ericjanvanputten]

that orange pi zero indeed seems affordable and i can do 1 per cam... probably even more power available :)

[ericjanvanputten]

Still I'm not sure we found the problem.

With webinterface open the load is around 4, which isnt good. got that part. With webinterface NOT open the load seems to be around 2, which is good. With webinterface NOT open and my wife in front of the 2 cams the load is going up to 3.5, which is borderline...

But when I check the logs of the wired cams (linksys domes LCAD03VLNOD) it still starts and stops the camera streams to the Pi with motion a couple times an hour in most cases.

what i'm going to try:

• I'm now starting a stream to my wireless computer VLC and see if the wired cam logs drops there as well.
• keep an eye out on top on the pi to see if motion + recording maybe puts it over the top. (pun not intended :))

Otherwise I might order another pi 3 and/or orangepi zero's to see if helps.

what are your thoughts on this?

(btw if someone knows how to measure the load over longer time - maybe that can reveal the load issues in combination with the logs of the cam's)

[ericjanvanputten]

mini- update: I left the ssh connection on with top for an hour or so, in which the highest values on load was 3.7-3.8. But in that time the wired cams did lose connection for a moment (once only).

I have a rough grouping with the load of 1 minute (5 minute just used for count) So overall, it seems to manage okay on load. So i'm still a little bit unsure if load is the (only) issue here...

I'm now trying with only 1 wired cam, i have top running on ssh, and i will check in again on that in a couple hours or so.

And I ordered 2 Orange Pi Zero's :) hoping to take some of the load of the Pi

[ericjanvanputten]

Ran the 1 wired cam for couple hours, load never higher than 1. Camera never dropped according to the cam logs.

Trying with wired 2 cams now for couple of hours :) (3 hours in) - 80% of the time the load is below the 2.2 threshold. top value reached is 2.6 with 2 cams. Both camera's don't show dropped connects.

Going to give the wireless cam + 1 wired cam a go now. And we have a problem, the load is sub 2, but the wired cam is showing drops in streaming in the logs. So my new guess is that the wireless cam is doing something motion doesnt like... VLC the stream is solid-ish - with artifacts... maybe that is tripping motion?

Going to try wireless only with web interface on and VLC, see what happens. Seems that both VLC and MotionEye show dropped cams (grey screen), artifacts green and grey.. so maybe the RTSP stream from the wireless cam is wonky.

A full night with 2 wired cams = no drops in the cams logs.

Going to try 3 cams, with no motion or recording activated :) Still got failing cams with load of 2.2 max... I think its the YI wireless cam that triggers issues.

[ericjanvanputten]

Question: if i would put most of the load on secundairy systems (orange pi zero's, pi 3) can i still potentially get a stable web interface with multiple cam's just showing up?

just tried 3 rtsp streams with no motion, recording, etc. but the load was already above 4 when checking... and cam's were failing... just wondering if maybe using the forward streams from secundairy systems is more load effective?

I believe that @ccrisan said he had 9 cams working on the XU4 board. so maybe it is something to consider.

[tosiara]

9 cams on one XU4? Nine 720p RTSP streams? @ccrisan - can you provide the details about your setup please?

[ericjanvanputten]

not sure about the RTSP OR resolution, OR framerate :)

[ccrisan]

@tosiara my setup is quite simple: 9x Raspberry PI B+ boards running motionEyeOS in Fast Network Camera mode (i.e. simple netcams) added as MJPEG (not RTSP!) netcams to an OdroidXU4, also running motionEyeOS.

The OdroidXU4 board is the only board running motion (4.x) and it does all the heavy stuff (motion detection, video encoding, notifications, etc). All streams are configured at 1024x768@10fps but they rarely reach that rate while previewing from the browser (too many delays, too much traffic, and sometimes too much CPU usage for the Odroid).

The sysload on the Odroid varies between 4 and 10 (it has 8 cores), depending on motion detection and whether the UI is opened or not.

P.S. I have recently added the 10th RTSP camera to the same Odroid but it has serious stability issues (it's a cheap Chinese security camera).

[ericjanvanputten]

Pretty impressive :)

still considering one, but first will give the orange pi zeros and/or 2nd pi 3 a chance before getting the XU4.

[tosiara]

Lets close this issue for now. You can reopen if you experience the same problem with the new setup