Beaglebone freeze and conniction timeout

[BrettGoldbach]

When running tests through the server, occasionally the bot freezes(through both wifi and usb) and displays this error:

ctrl_server.py | call_method | 265 | WARNING | Exception: '[Errno 110] Connection timed out' --> {'msg': "Exception: '[Errno 110] Connection timed out'", 'type': 'error'} bot$ Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.008850] Internal error: : 1028 [#1] SMP THUMB2

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.089430] Process irq/30-4819c000 (pid: 27, stack limit = 0xdf274240)

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.096329] Stack: (0xdf275f08 to 0xdf276000)

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.100872] 5f00: c036767d df274000 df253d60 00000001 df253d40 df006840

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.109414] 5f20: c085537c c0074a97 00000000 c0074979 df071db0 00000000 df253d40 c00749f9

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.117955] 5f40: 00000000 00000000 00000000 c00454ab 00000000 00000000 00000000 df253d40

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.126488] 5f60: 00000000 00000000 dead4ead ffffffff ffffffff df275f74 df275f74 00000000

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.135013] 5f80: 00000000 dead4ead ffffffff ffffffff df275f90 df275f90 00000000 df071db0

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.143544] 5fa0: c0045441 00000000 00000000 c000c8fd 00000000 00000000 00000000 00000000

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.152080] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.160608] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000

Message from syslogd@ieee at Mar 28 14:29:59 ... kernel:[ 3619.195135] Code: 7849 fa01 f303 18d3 (8819) 6da0

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.017922] Internal error: Oops: 17 [#2] SMP THUMB2

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.096932] Process irq/30-4819c000 (pid: 27, stack limit = 0xdf274240)

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.103829] Stack: (0xdf275d30 to 0xdf276000)

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.108371] 5d20: c0074979 df24852c 00000000 c00435e5

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.116903] 5d40: 00000000 ed16ed16 00000000 df248140 00000000 df274000 c0859758 c0033677

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.125433] 5d60: df275de3 00000002 df274000 00000001 c070cf5c ed16ed16 00000000 c0889f34

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.133960] 5d80: df275ec0 c08d44d8 df275ec0 df274000 c0859758 0000000b df275de3 00000002

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.142491] 5da0: 20000193 c000f96d df274240 0000000b c03676aa 00000020 00000000 00000004

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.151020] 5dc0: 37000000 20393438 31306166 30336620 38312033 28203364 39313838 64362029

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.159557] 5de0: 00203061 00000000 00000000 c04d0c01 c06fee98 df275e14 c0d60fc0 00001028

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.168086] 5e00: fa19c02c df275ec0 c085a184 00000010 c085537c 00000000 00000000 c0008439

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.176614] 5e20: 00000000 c005763d 00000007 00000000 00000000 fa19c02c c0d60fc0 c0849fc0

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.185147] 5e40: c0855890 c00583b1 c0d60fc0 c0d60fc0 c0849fc0 020b020a df06e5c0 20000013

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.193673] 5e60: c0d60fc0 c04d6959 20000013 c04d5d93 df06e9d4 80000093 df06e9d4 c04d6959

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.202191] 5e80: 00000001 c00503ff df274000 c0074a4d c0849fc0 c0849fc0 e226f1b6 0000033f

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.210712] 5ea0: c0849fc0 c0849fc0 0000193a c03676a2 800000b3 ffffffff df275ef4 c04d6bb5

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.219231] 5ec0: 60000013 0000002c fa19c000 fa19c02c df258410 00000065 60000013 00004000

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.227759] 5ee0: 00000010 c085537c 00000000 00000000 00000000 df275f08 c04d6991 c03676a2

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.236279] 5f00: 800000b3 ffffffff c036767d df274000 df253d60 00000001 df253d40 df006840

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.244795] 5f20: c085537c c0074a97 00000000 c0074979 df071db0 00000000 df253d40 c00749f9

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.253319] 5f40: 00000000 00000000 00000000 c00454ab 00000000 00000000 00000000 df253d40

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.261842] 5f60: 00000000 00000000 dead4ead ffffffff ffffffff df275f74 df275f74 00000001

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.270363] 5f80: 00010001 dead4ead ffffffff ffffffff df275f90 df275f90 00000000 df071db0

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.278886] 5fa0: c0045441 00000000 00000000 c000c8fd 00000000 00000000 00000000 00000000

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.287406] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.295936] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000

Message from syslogd@ieee at Mar 28 14:30:00 ... kernel:[ 3620.417108] Code: 4770 bf00 f8d0 3274 (f853) 0c20

[BrettGoldbach]

It is possible that the error messages after the connection timeout are from my system. I need someone else to get the freeze error to see if they also get the other messages.

[AhmedSamara]

@dfarrell07 @PaladinEng @SeanKetring

So what do you guys think is causing this?

This seems to happen during I2C communications.

[AhmedSamara]

What exactly were you running when this came up?

[SeanKetring]

ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error
ctrl_server.py | listen | 172 | INFO | <class 'zmq.core.error.ZMQError'>
ctrl_server.py | listen | 171 | INFO | massive server error

[AhmedSamara]

This was in the log, may be relevant.

2016-03-28 14:26:02,631 | INFO | switch.py | __init__ | 19 | Running in non-test mode
2016-03-28 14:26:05,477 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:05,486 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:05,494 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:05,503 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:08,391 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:08,399 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:08,408 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:08,416 | CRITICAL | dmcc_motor.py | __init__ | 103 | Testing : False
2016-03-28 14:26:09,294 | INFO | ctrl_server.py | __init__ | 117 | Control server initialized
2016-03-28 14:26:09,296 | INFO | ctrl_server.py | listen | 154 | Control server: tcp://*:60000
2016-03-28 14:29:49,993 | WARNING | ctrl_server.py | call_method | 265 | Exception: '[Errno 110] Connection timed out'

[kvijay1995]

Yea I've never seen that WARNING flag before. That's probably our problem

[BrettGoldbach]

After testing the bot with the camera disabled I found that it didn't freeze. Because of that and the lights we were getting from the bone(after freezing it would have two solid lights and one slow blinking light) it pointed to a probably brownout issue. Once we powered the camera through the usb hub instead of the bone the freezing stopped and it is running smoothly.