Dropped Samples (with Ganglion)

[wliao229]

The Ganglion samples include a column of sample ID ranging between 0 - 200. I tracked this ID, and noticed some singular, roughly-periodically dropped samples when using the wifi board, sampling at 1600Hz.

Here is the IDs of the first 200 samples (saved by OpenBCI_GUI, using TCP, 10ms latency):

array([ 74,   0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  12,
        13,  14,  15,  16,  17,  18,  19,  20,  21,  23,  24,  25,  26,
        27,  28,  29,  30,  31,  32,  33,  35,  36,  37,  38,  39,  40,
        41,  42,  43,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,
        55,  57,  58,  59,  60,  61,  62,  64,  66,  67,  68,  69,  70,
        71,  72,  73,  74,  75,  76,  78,  79,  80,  81,  82,  83,  84,
        85,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
       100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 112, 113,
       114, 115, 116, 117, 118, 119, 120, 122, 123, 124, 125, 126, 127,
       128, 129, 130, 131, 132, 133, 135, 136, 137, 138, 139, 140, 141,
       142, 143, 144, 146, 147, 148, 149, 150, 151, 152, 153, 155, 156,
       157, 158, 159, 160, 161, 162, 163, 164, 165, 167, 168, 169, 170,
       171, 172, 173, 174, 175, 176, 177, 179, 180, 181, 182, 183, 184,
       185, 186, 187, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198,
       200,   0,   2,   3,   4,   5,   6,   7,   8,   9,  10,  12,  13,
        14,  15,  16,  17,  18])

To highlight the missing indices, I inserted "nan" as follows:

array([  74.,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,   nan,
         nan,    0.,    1.,    2.,    3.,    4.,    5.,    6.,    7.,
          8.,    9.,   10.,   nan,   12.,   13.,   14.,   15.,   16.,
         17.,   18.,   19.,   20.,   21.,   nan,   23.,   24.,   25.,
         26.,   27.,   28.,   29.,   30.,   31.,   32.,   33.,   nan,
         35.,   36.,   37.,   38.,   39.,   40.,   41.,   42.,   43.,
         nan,   45.,   46.,   47.,   48.,   49.,   50.,   51.,   52.,
         53.,   54.,   55.,   nan,   57.,   58.,   59.,   60.,   61.,
         62.,   nan,   64.,   nan,   66.,   67.,   68.,   69.,   70.,
         71.,   72.,   73.,   74.,   75.,   76.,   nan,   78.,   79.,
         80.,   81.,   82.,   83.,   84.,   85.,   nan,   87.,   88.,
         89.,   90.,   91.,   92.,   93.,   94.,   95.,   96.,   97.,
         98.,   nan,  100.,  101.,  102.,  103.,  104.,  105.,  106.,
        107.,  108.,  109.,  110.,   nan,  112.,  113.,  114.,  115.,
        116.,  117.,  118.,  119.,  120.,   nan,  122.,  123.,  124.,
        125.,  126.,  127.,  128.,  129.,  130.,  131.,  132.,  133.,
         nan,  135.,  136.,  137.,  138.,  139.,  140.,  141.,  142.,
        143.,  144.,   nan,  146.,  147.,  148.,  149.,  150.,  151.,
        152.,  153.,   nan,  155.,  156.,  157.,  158.,  159.,  160.,
        161.,  162.,  163.,  164.,  165.,   nan,  167.,  168.,  169.,
        170.,  171.,  172.,  173.,  174.,  175.,  176.,  177.,   nan,
        179.,  180.,  181.,  182.,  183.,  184.,  185.,  186.,  187.,
         nan,  189.,  190.,  191.,  192.,  193.,  194.,  195.,  196.,
        197.,  198.,   nan,  200.,    0.,   nan,    2.,    3.,    4.,
          5.,    6.,    7.,    8.,    9.,   10.,   nan,   12.,   13.,
         14.,   15.,   16.,   17.,   18.])

I plotted below 1 second of samples using different network protocols ---- I open this as a new issue because I feel this problem seems to be irrelevant to the protocols (TCP, UDP, or UDPx3, i.e., issue #45 #48). (I did not have enough time to try different level of latency, and the samples were collected using 10ms latency).

As shown in the chart, there are some consecutively dropped samples, probably due to dropped packets. However, there are also such singular, dropped samples as concerned here.

I have multiple wifi-boards and Ganglions, and this problem seems to be universal to all of them.

[andrewjaykeller]

I wonder if there is a sample byte calculation wrong in the driver, I’ll check this first in regards to the periodic sample drop. As a follow up to the intermittent dropped packets, are you doing WiFi over a router?

[wliao229]

For the intermittent dropped packets, yes, I am using Wifi over a router (Linksys WRT1900ACS). The dropped packets are less of a concern for me: Below is the distribution of sizes of consecutively missing samples, for a recording about 1.5 hours using TCP:

size, frequency
1.0      700467
2.0         169
200.0+       21
42.0          3
60.0          2
48.0          2
47.0          2
43.0          2
113.0         1
62.0          1
61.0          1
59.0          1
58.0          1
56.0          1
41.0          1
34.0          1
18.0          1
51.0          1

Overall percentage of dropped samples: 8.7640%

The singular, missing samples are the major source of missing data in my case. Also, such missing samples are not perfectly periodical.

[wliao229]

If this does not just happen to my setting, then I feel this is a quite critical issue. I would like to a least know the nature of such missing samples so that I can interpolate them correctly (e.g., assuming the IDs are equally spaced).

[andrewjaykeller]

For TCP: check out this issue #48 specifically this comment which dives into the problem with TCP and hangups, especially on a router.

The solution i think is altering the firmware is the only fix for TCP, where we use the Heap to store bytes to until the hangup finished. I'll look to fix the problem you're seeing with the 1 sample in the next GUI release at the end of the month.

[andrewjaykeller]

Okay I have some tasks I'm going to attempt tomorrow morning, switching to using the heap entirely instead of the stack and allocating ring buffer on stream start.

[jnaulty]

@wliao229 do you think you could try this with the wifi shield configured as an Access Point? Just to rule out the router as cause of interference?

[wliao229]

I experimented more and found the problem is probably due to Ganglion reading analog signals from the GPIO pins (see the plot below). I use Ganglion to read A3, A4, and A6, which are left unused by Wifi Shield. It seems like this may interfere the Wifi Shield and cause the drop of samples. When I turn off reading the analog signals (the 4th row in the plot), the problem disappeared (there are some dropped consecutive samples probably due to lost packets, but that's not an issue here).

This is not a problem when using Ganglion's own bluetooth (with wifi shield still attached and sensors went through the wifi shield's socket). See 1st and 2nd rows where the reading of analog signals was turned on and off, respectively; and there was no dropped samples.

[andrewjaykeller]

What code did you use to read the analog input pins? I don't think that firmware is part of the default library.

It seems like this may interfere the Wifi Shield and cause the drop of samples.

Depends which pins are being used! The WiFi Shield is only connected to a handful of the header pins, most of the ganglions are not connected

[wliao229]

@jnaulty I never managed to use the Wifi Direct mode––no data is read from the device. Here is a screenshot of the OpenBCI_GUI when using wifi shield via a router and via the "wifi direct," respectively:

The wifi shield's firmware is v2.0.3 (though, I used the latest release, DefaultWifiShield.2.0.4.bin), and the GUI was also the latest version.

[wliao229]

@aj-ptw I customized the Ganglion firmware a little bit to read the A3, A4, A5, A6 in place of channel 1-4. And I changed the functionality of turning on/off each channel to a switch between analog and regular signals. You may check my changes at https://github.com/wliao229/OpenBCI_Ganglion_Library

Is it because Ganglion (Simblee) cannot read GPIO pins at 1600Hz, so the sample is damaged before it is sent to the wifi shield? (There seems to be no problem of Ganglion sending the samples via bluetooth at 200Hz.)

[wliao229]

Okay. I think it is the problem of Ganglion (using customized firmware). When I use wifi-shield, TCP, and sample at 200Hz, the problem disappeared.

Either my customized firmware has some problems, or Ganglion cannot read GPIO pins at 1600Hz.

In this case, you may close this issue, since this seems to be not related to the wifi-shield. sorry about the false alarm.

[andrewjaykeller]

@wliao229 no problem! only a slight heart beat or two was skipped!

I would love to see this code added to the Ganglion library base

[wliao229]

@aj-ptw Thanks. I wish there is an official support of reading A3,4,5,6 on Ganglion----my customization was very blunt, and I don't know much about coding the firmware.

I will open a new issue regarding my issue of Wifi Direct. Hopefully, that is not a false alarm.