Project 120 receiver issues

[zcrysler]

Project 120 seems to have issues at almost all the receivers in terms of shutting down for long periods of time. All receivers are on AC power so it's not a solar issue. There's a lot of lightening in the area so it could be power outages, but some of them seem to last quite a long time. It doesn't seem related to site visits.

On the most recent site visits to West Galveston SG-3216BBBK0757 and Quintana SG-5113BBBK2668, Susan Heath could connect to the live feed and see that the receivers were running, but neither were registering funcubes, a reboot solved the problem.

She's also noted that on the live page, funcubes will sometimes show up under "What I'm doing now" but not under "Devices", and sometimes they won't show up in either place.

Seems unlikely to be loose connections since it's happening at so many sites, I imagine it's a problem rebooting after a power outage, or possibly their outages are lasting that long and the gnomes are working fine. Can you take a look at the detections to see if there's cause to be concerned?

[jbrzusto]

One issue with running on AC power is that if the system hangs for some reason (e.g. a power spike or a software crash), it won't restart until a manual reboot. On a solar-powered system, one likely gets an occasional restart simply due to lack of sunshine.

Only the Scenic Galveston site seems to be working fully. I wonder what's different there?

One pattern shown by several receivers is that funcubes are all working or all not working, and the boundary between these periods corresponds to reboots. This suggests flaky USB cables between hubs and beaglebones, or perhaps the hub being hot-plugged into the beaglebone, or some other thing preventing the beaglebone from seeing the USB devices. Perhaps there are power glitches that end up causing the beaglebone to reboot, but leave the hub stuck.

Preliminary notes (by receiver)

SG-0816BBBK01E7 (Brazoria):

• all-or-nothing on antennas 1, 2, and this aligns with reboots. From a reboot on 30 April (i.e. boot session 39) to the end of boot session 42 on 9 June, the funcubes are not working.

SG-0816BBBK0678 (San Bernard):

• most files missing between 21 Jan and 4 April; only one file per day seems to have been uploaded. This does not appear to be a receiver issue. This gives the appearance of intermittent antenna activity, but it's impossible to tell without the remaining files.

• all-or-nothing on antennas 1-3 between 5 April and 14 June, aligned with reboots.

• very intermittent activity on antenna 4 between 5 April and 14 June, although totally off in same periods as antennas 1-3 are off.

• on 12 June, there is apparently a restart of the master process 3 minutes after rebooting, but there are two reboots in 4 minutes, so it's not clear what's going on here.

SG-2916BBBK0794 (Palacios):

• SG not working from 9 May to 24 June; no indication in system logs of why.
• antenna 2, 3 are working fine since 24 June

SG-3116BBBK0755 (Port O'Connor):

• antenna 2, 3, 4 showing all-or-nothing pattern aligned with reboots during January

• this unit has been online, but is failing to fully sync its data with the server, apparently.

SG-3216BBBK0757 (West Galveston):

• antenna stops reporting pulses 9 May until the reboot on 22 May

• apparent power outage from 4 June to 8 June

SG-3216BBBK0793 (Aransas):

• 30 March to 27 April, boot session goes from 14 to 17, but we have no files from boot sessions 15 or 16, and so no data in this period.

• antennas 1, 2 appear to be working fine since 27 April

SG-4216BBBK0315 (Rockport Bay):

two outages (periods with no files):

• 22 May to 14 June outage
• 11 Mar to 25 April outage

receiver seems to be working outside these periods.

SG-5113BBBK2668 (Quintana):

• all-or-nothing pattern on antennas 1, 2, 3 since mid 2016.

SG-7100BBBK0036 (Scenic Galveston):

• both antennas 1 and 2 working continuously and across 8 reboots since January.

SG-7100BBBK0271 (Trinity River):

• all-or-nothing pattern for antennas 1 and 2
• long periods of total outage in Feb, Mar, Apr,
• short period where SG running, but no funcubedongles
• files in .sqlite database have blank names - why???
• only 1144 out of 6631 files are present in file_repo folder even though their contents are in the DB. Why? Issues with running the tagfinder seem to have prevented files from being archived correctly. Uploads are still in the uploads/TrinityRiverNWR folder

SG-7100BBBK1493 (Bill Baker Tugbout)

• antenna1 shows all-or-nothing pattern aligned with (frequent) reboots. Seems as if antenna is often just not plugged in at boot time.

[zcrysler]

Thanks, I'll pass that onto Susan Heath - I'll suggest double checking all connections, and ensuring they never hot plug in the USB hub (or any other connection). If it's an issue with reboots and power, I guess the only thing they could do is try to visit sites after bad storms when power outages are likely to have occurred?

[jbrzusto]

I don't consider this closed. A couple of suggestions for now:

• plug the AC adapter for the SG into a surge-protected power bar.

• replace the short USB hub <-> beaglebone cables, ideally with longer ones. I have a pile of these in the lab I could send, but it might well be cheaper to just have them purchased locally. The problem is the short length leading to a tight bend, which can lead to problems either at the port (if it loosens) or inside the cable itself.

  These have led to similar problems elsewhere, and the damage is invisible to the eye. You can diagnose it by watching the web interface while connected to the SG, and then wiggling that cable. If devices come and go from the web interface, the cable (or one of the USB ports, e.g. on the beaglebone itself) is flaky.

[zcrysler]

Ok thanks, I passed those suggestions on, hopefully she can test out the USB <-> beaglebone cable to at least rule that in/out as an issue