WiFi shield packet loss causes cyclical noise spikes

[conorrussomanno]

See attached screenshots. You can see there is a spike of noise once every second or so. I'm switching back and forth between Dongle and WiFi shield, and it only appears when using WiFi, so it doesn't seem to be an environmental effect.

I'm using 1000Hz and 10ms latency.

[conorrussomanno]

Now this is only happening every once and a while for me. I'm not sure what is the root cause. It could be my hardware. I'm interested to see if anyone else experiences something similar using the WiFi Shield.

[andrewjaykeller]

Are these regularly spaced? Could you try updating your firmware? Maybe something got botched the first time?

[biomurph]

I don't see this problem on my wifi

[andrewjaykeller]

could you try putting EXT PWR to OFF and use two batteries?

[andrewjaykeller]

i think that might help with the noise

[andytheengineerguy]

Hi guys, was there any progress on this?

I'm also experiencing cyclical noise spikes, which vary between 1-4hz, and only appear occasionally. The occurance of these cyclical spikes seems loosely correlated with resetting the system through the OpenBCI GUI.

I'm not using a WiFi shield, just the regular cyton EEG sensing board. It had been working well up until tonight.

I was playing around with the BIAS and SRB pins before I saw this behaviour. Any ideas?

[andrewjaykeller]

Hi! There has been a lot of progress on this re: Cyton with WiFi shield. However this is not an issue we normally see with the Cyton over serial. Could you upload a screen shot of your spikes here?

[andytheengineerguy]

Hi AJ, thanks for the quick reply!

I actually just left work so cant take any more screenshots until I get in tomorrow morning (different time zone here in Aus), however I did take this photo showing what happens after repeatedly connecting-disconnecting the system through the OpenBCI GUI. For some reason, the frequency of the pulses get faster each time I disconnect-reconnect, and after 6-7 iterations this is what i end up with.

Its a strange waveform thats distinctly different from random noise. While showing this waveform im unable to sense any EEG data.

Have you seen this before? Apologies if Ive gone too off track with this issue - this is really confusing me haha. Will upload pulse screenshots in the morning.

[andytheengineerguy]

[andrewjaykeller]

That looks almost like a test signal, hmm is this issue still happening?

[biomurph]

Not the test signal. Some environmental noise?

[retiutut]

I am experiencing this issue also when using the wifi shield with 8 channels and using two separate batteries (4xAA and 3.7v 500mah).

[andrewjaykeller]

What firmware are you running on the WiFi shield?

[retiutut]

The WiFi shield firmware is v2.0.5 . Updated the Cyton firmware this morning and still having same problem:

[andrewjaykeller]

Have you tried UDP?

[retiutut]

I've mostly been using TCP. I did swap out the WiFi shield for a Daisy header and was able to see proper EEG over BLE using the dongle. I'll try UDP and UDPx3 later today or tomorrow.

[andrewjaykeller]

No doubt in my mind this is 100% due to using WiFi! This can happen on some routers I’ve seen, it should magically disappear with UDP. Probably won’t happen with TCP over WiFi direct too.

[retiutut]

To document: I'm using a current generation Airport Extreme mounted high on the wall. The LOS distance from the Airport to the WiFi shield was about 5 feet.

Had to put my iPhone in airplane mode to have uninterrupted upload for the Cyton firmware via Arduino IDE. Left it in airplane mode during these tests.

[retiutut]

This issue is still happening over UDP, rendering the WiFi shield unusable. Was not able to find the shield using WiFi direct in the GUI.

[andrewjaykeller]

Did you erase the network credentials?

[retiutut]

I erased the network credentials again, got 10 blinks in 2 seconds on boot, and connected to the Wifi shield. The GUI still only finds this:

[andrewjaykeller]

Weird! Which GUI version are you using.

[retiutut]

Using OpenBCI_GUI 3.2.0 standalone on Mac

[retiutut]

@aj-ptw What settings (hardware and software, apart from up-to-date firmware) would you recommend to resolve this issue? I was looking forward to the increased resolution over WiFi, but I get worried every time I have to change the hardware just to see if the WiFi shield works.

[andrewjaykeller]

@retiutut could you try on a different router? If not, maybe we can do a call and I show you how WiFi direct works. Skype me at pushtheworld_aj

[retiutut]

I think I have to try and update the Cyton firmware again. I'm having serious issues over BLE also. Here we have a series of eye blinks followed by rest and then a spike across all channels.....

[andrewjaykeller]

Hey @retiutut ! I have just finished a new gui version that will allow you to use the wifi direct for sure.

[retiutut]

I have identified the cause of the problem using the Processing version. I see packet loss correlations with these visible spikes when using an Airport Extreme. I was able to finally connect via wifi direct using a newer Mac Mini and I still see regular spikes when streaming. Here is a screenshot that documents this issue:

[engsz]

Hi guys, was there any progress on this? I have same issue with wifi shield but not with USB dongle.

[jps2000]

I have same issue with wifi shield but not with USB dongle.>> So it is likely that you have electrical/ magnetical interference from the shield. You see also in your example that despite you have 60hz notch on there is huge 60Hz in the spectrum. This tells you what these signals are. Do you use AC adapter or battery. The latter is important! Also for safety reasons.

You can place a grounded copper foil between shield and cyton or play around with electrode cables location.

Another idea to figure out the cause is to short the inputs. Open inputs are very sensitive. For example I 3D printed a self-designed housing with ABS and noticed spikes appearing randomly in the time series. Then I noticed that just rubbing the outside of the box caused charging the plastic and hence lead to spikes. Counteraction was to paint inner side of housing with copper laquer used for EMI https://www.conrad.at/de/abschirmlack-crc-kontakt-chemie-emv-35-77509-aa-200-ml-813893.html and to connect the shield to ground. This also improved significantly mains interference.

NOTE: Do not paint the housing close by the antenna !!

[retiutut]

So it is likely that you have electrical/ magnetic interference from the shield.

The original post says this is likely not interference. Also, the spikes show up on frontal lobe sensors just as strong.

I'm switching back and forth between Dongle and WiFi shield, and it only appears when using WiFi, so it doesn't seem to be an environmental effect.

I still think this has something to do with packet loss.

[wjcroft]

@paquettejp thanks. I'd be interested in your thoughts on this recent forum post. We haven't verified this yet in the lab with an OpenBCI manufactured board. His oscilloscope traces were taken from a Wifi Shield, he built himself using OpenBCI board layouts.

http://openbci.com/forum/index.php?p=/discussion/2108/wifi-shield-power-supply-resonant-circuit-noise-ringing-also-audible

[wjcroft]

Possible workaround for Wifi Shield RF power output spikes (400 mA), that may be causing interference or lost packets

The ESP8266 has an optional internal command that can reduce the RF power output level of the transmitter. Right now I believe the shield runs at the default maximum power level. Which is probably overkill for most applications that are within 10 or 20 feet of the receiver. This high power BOTH has the potential to disrupt sensitive EEG signals, but also has caused measurable dips and resonance / ringing in the power supply circuitry. Oscilloscope traces show the supposedly regulated 3.3V shield power supply, dipping .6V below the 3.3V level. Not good. These dips are caused by the heavy current drain of the ESP8266 (up to 400 mA), when outputting beacon signals at max RF power. See @paquettejp (Jason Paquette) 's thread here on the forum:

"measured shield regulated 3.3V supply, has .6V dip at 1 second intervals, with resonant ringing" http://openbci.com/forum/index.php?p=/discussion/2108/measured-shield-regulated-3-3v-supply-has-6v-dip-at-1-second-intervals-with-resonant-ringing#latest

ESP8266 commands to set the power level are documented here; there are several ways to accomplish setting the RF power level:

https://github.com/beckdac/ESP8266-transparent-bridge/issues/53

Page 15 and 16 of this ESP document,

https://www.itead.cc/wiki/images/5/53/Esp8266_at_instruction_set_en_v1.5.4_0.pdf

I think this is worth a try. Not sure yet if this power level should be user settable, or if we could just drop back the power level to one that would work for most everyone. And substantially reduce the noise effects. Certainly there may be some noisy environments where high power is necessary, but in those situation there may be better workarounds such as just getting closer to the receiver.

Mentioning @produceconsumerobot @andrewjaykeller

[wjcroft]

From the Arduino API, the power is set using this function, ESP8266WiFiGenericClass::setOutputPower(float dBm), which calls the system_phy_set_max_tpw(val). See line 356 of:

https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp

The ESP 'AT' command mentioned previously is possibly invoked by the system_phy... function. This Arduino API call is obviously the right way to do it for the Wifi Shield.

---

Many more results on this search:

https://www.google.com/search?q=esp8266+rf+power+output

[retiutut]

[wjcroft]

Is this thread relevant?

http://openbci.com/forum/index.php?p=/discussion/2108/measured-shield-regulated-3-3v-supply-has-6v-dip-at-1-second-intervals-with-resonant-ringing

On that thread, Jason (alwayswearshats), shows that shield power supply glitches can result in spikes and packet loss. Adding a capacitor as discussed in the thread, improves power supply quality markedly.

William

[retiutut]

I rarely use Cyton and WiFi for all of the reasons on this thread. The noise spikes are one thing, but this packet loss is just out of hand even after all this time. I would like to see a new WiFi board!!!

Just to further document what happens, the noise spikes appear cyclically and when there is packet loss. ALSO, appears in SD card recordings after being properly converted and viewed in the GUI.

[wjcroft]

Here is a 680uF cap you could try. Since the cap is 8mm in diameter, and the inter-board spacing is just a bit more than that (say 11mm), would need to be positioned to avoid other components on the board below the shield.

https://www.amazon.com/pcs-Panasonic-FR-Capacitors-680uf/dp/B072SS6YBJ/

My suggestion would be to tape the cap to the bottom of the shield, then run leads to existing large ceramic cap on the bottom of the board. Do one side at a time, so you don't disconnect the old ceramic. Enclose leads of the 680uF with some type of tubing or insulation.

[wjcroft]

I just ordered the capacitor(s) at the Amazon link. Comes with 6. Could forward one to you if test is successful. Would like some advise on specific test situations where I can try before / after tests of the noise glitches or packet loss. In other words, scenario that creates MOST glitches.

Looks like a simple wire extension mod would also allow external antenna idea to be tested WITHOUT needing to build a case. https://www.google.com/search?q=external+solder+on+antenna+for+esp8266 And... just the capacitor alone may clean up the noise.

[retiutut]

@wjcroft That would be great! Cyton+WiFi "spikes" can occur without being connected to a user. If it can go 20+ seconds without a dropped packet while not connected or being moved, that would be the goal.

[wjcroft]

Richard, can you confirm, the packet loss and spikes have also been seen on Ganglion. I would assume. Is TCP more prone to this than UDP or UDPx3?

[retiutut]

As far as I know, the really pervasive spikes for little/no reason only happen with Cyton+WiFi or a Cyton with possible low battery.

To me, Ganglion+WiFi is really fun and stable. I've only been able to cause packet loss and spikes with Ganglion or Ganglion+WiFi by moving to another part of the house or even outside. Still if there is enough signal, a spike will show when a packet is dropped. A fix for these types of "hiccups" is in progress.

[wjcroft]

Had some exchanges with Jason on the thread discussing the power supply ringing. See the TI app note I posted there (and below). It exactly describes why the 100uF ceramic cap used on the bottom of the Ganglion, causes huge ringing and noise. Wish Joel or AJ had seen this before choosing that design. My guess is that the noise generated, is more disruptive to the Cyton than the Ganglion, due to the ADS1299 acting as kind of an antenna for high frequency noise pulses generated by the Shield. Components on the Ganglion are less complex and lower amplification, compared to Cyton.

http://openbci.com/forum/index.php?p=/discussion/2108/measured-shield-regulated-3-3v-supply-has-6v-dip-at-1-second-intervals-with-resonant-ringing#latest

TI app note: http://www.ti.com/lit/wp/snoa842/snoa842.pdf "Engineers note: Capacitors are key to voltage regulator design"

In summary, the LDO (low drop out) linear regulator as used on the Ganglion, plus the low ESR ceramic cap on the output, generate excessive ringing and noise due to the low ESR. Solution is to use a cap with both more microfarads, and more ESR. Electrolytic or tantalum looks promising. The previous 680uF electrolytic I happened to find on Amazon was an ultra low ESR; good for some applications, but not LDO linear regulator. Will reorder a more suitable cap for trial, using tips in the TI app note.

[jps2000]

I can not test it but it is important where tho place the cap. As it serves as a local battery it should be as close as possible to the load. In this case when the ESP is drawing current spikes the cap should be close to its VCC and GND pin. Ideally it is a tantal 100u or so having a low ESR. The cap close to the LDO regulator should be within data sheet recommendation ( 1-10u ). In this case I see that the layout is poor. I suggest to try a cap directly from pin 8 to the ground plane ( scratch away the solder resist) You can even try to solder it to the shield of the ESP Also remove the 100uf on the other side it can make it worse due to bigger current loops and resonances Archambeault1.pdf

[furmanlukasz]

Hi, having the same issue over UDP but I notice that the signal goes right after disconnecting electrodes , the spikes go off, and the signal is nearly railed, so it is not the packet loose. I saw that the 'gold cup' electrodes can bee the trouble, any time spike goes up the same happens to the impedance of electrode(just click the 'om' icon to see if it happens the same in yours setup). They just far away from the desired quality to make good measure. I will test it over Ag electrodes when they arrived so I hope It will solve the problem. Maybe somebody can test it earlier. Just a hint. But true silver electrodes cost 10 times the ' Chinese gold cup' and have a screen in cable that can be grounded also. The screen between this two boards are also the good idea , perfectly can be grounded from one side cyton ground and the other side wifi ground.

[retiutut]

I think it's time to close this issue as it exists here on the OpenBCI_GUI Repo.

Packet interpolation is live as of GUI 4.1.7-beta.0, and it resolves most packet loss spikes.

While this does improve the data received and displayed in the GUI, it doesn't fix the hardware problems that cause cyclical noise spikes when pairing the WiFi shield with the Cyton or Cyton+Daisy, regardless of power configuration or WiFi connection settings.

To reiterate, I have tried two WiFi shields with Cyton, Cyton+Daisy, and two Ganglions, and this problem only occurs with increased number of channels when using Cyton or Cyton+Daisy with the WiFi shield.

As previously indicated in the conversation above, it seems a hardware fix is required, either to existing WiFi shields or in the development of a new board/shield of some kind.

[wjcroft]

Richard, I can understand why you want to close this as related to the GUI. And especially now that you have the interpolation going. But this thread documents a serious hardware issue that has yet to be solved.

So it needs to be kept open SOMEWHERE. Perhaps by referring back to this thread from an open issue on the Wifi Shield hardware or software repo. The material in this thread is documenting some of the engineering fixes required.

This is the related thread in the other repo:

https://github.com/OpenBCI/OpenBCI_WIFI/issues/82 "WiFi Shield dropped packets"

I suggest changing the title on that thread from the original title, which was restricted to Direct mode. You might also consider changing the title on this issue thread to match.

William

[Joe-Westra]

Hi @retiutut and @wjcroft,

Thanks for providing this detailed overview of the cause of the noise spikes, they were maddening when unexplained. I am using a cyton+daisy+wifishield to transmit 16 channels at a minimum sample rate of 250hz. Is it correct that the only option to capture this stream reliably would therefore be to use the cyton+daisy+SD card and transfer the data via BLE at session termination?

Do you have any estimation of when a fix for the wifishield (or it's next iteration) might be available?

Thank you!

[wjcroft]

The shield power supply is helped, by running it on a separate AA battery pack. So TWO batteries, one on Cyton, one on Shield. Also UDPx3 is more robust than TCP, in general. Oddly some people have trouble with UDPx3, in which case fall back to TCP.

[Joe-Westra]

Thanks for the feedback @wjcroft, I'll let you know how successful we are when we get a chance to try the two separate battery packs.