Intermittent crackling static on I2S developed recently.

[tfabris]

Yesterday the I2S audio developed an occasional intermittent crackle sound. Like a digital version of dust on an LP record. This is after weeks of perfect operation.

The crackles are each a single digital "pop" sound that happens approximately once every 10 seconds, but it's very random, so sometimes 30 seconds or more will go by without a crackle, and sometimes you will get several crackles over the span of a few seconds.

The crackles happen in either the left channel or the right channel and it's random as to which channel they occur on. This tells me that it is a digital bitstream error that is very very small and localized, perhaps a single bit error which causes a given byte of audio data to have the wrong number in it, causing the pop.

Originally I had everything working fine on my Rev3 PCB for the BlueGigaEmpeg with the 7.5v Pololu regulator. Everything worked fine for weeks without crackles. Then I got the Rev6 boards in for BlueGigaEmpeg and soldered one together. It worked perfectly for a day with no crackles. Then I left it in the car overnight, and when I started the car up again the next morning there were crackles.

The crackles are mostly noticeable when the music is quiet, but, they are there all the time as long as there is digital music data to send. If you turn the volume on the Empeg front panel all the way down to -infinity (ie all sound is all zero bytes) then the crackles go away. But as soon as there's even the slightest amount of audio data to transmit, the crackles come back.

Best repro steps are:

• Turn volume way down on the empeg front panel but but not all the way down. Do not turn down to "-infinity DB". Turn down to, say, about-40db.
• Then on your bluetooth playback device, turn its volume up to a decent listening level so that you can hear the music faintly from the empeg.
• Play songs on the empeg, there must be songs playing.
• Wait for a minute or so and listen intently to the output. Listen for pops that sound like dust on an LP record. They will randomly switch back and forth between left and right channel.

I have checked all voltages and done a ton of diagnosis, including ruling out the Empeg, the wiring, the Arduino, and my new Rev 6 board. My current theory is that either the chip or my current Betz board is going bad.

This is troubling because I've fried chips before, and I thought I had solved all those problems. Another Betz board is on its way, and if that one doesn't crackle then I'll have to think about whether I need to look at power issues AGAIN.

[tfabris]

Tried all of these things, which did NOT fix the issue:

• Different Empegs (empeg Mk2 serial number 13 and a firesale Rio Car unit)
• Different Betz boards - three different ones total, including the one that worked fine for weeks prior. (Each Betz board contains its own WT32i chip, so trying different Betz boards gets different chips out of the deal.)
• Different Arduino (Tried the one that worked fine for weeks prior plus also a new out of box one.)
• Different Bluetooth output devices (Honda, Plantronics, Etekcity reciever, etc., all of them make the same sounds, it's definitely not the receiver.)
• Different wiring harness and sled: Same sound exists on THREE different sled+harness combos, including the one that worked fine for weeks prior.
• Fiddling with wiring proximity and twisting of I2S wires on wiring harness. Had a wiring harness completely peeled out with the wires far apart, no change.
• Different resistor values of voltage dividers (lowering voltage to below 2.5v by two different amounts)
• A whole bunch of re-doing and re-soldering the voltage dividers on the Rev6 PCB, new resistors, etc., It's not the resistors.
• Various changes to the I2S parameters in the BlueGiga code (removing keepalive, changing bit depth value, changing event parameter, etc.), it's not the code.
• Disconnecting serial port entirely. Not the serial port.
• Removing power from the MAX232 device entirely. Not the Max232.
• Removing the Pololu power supply and running the Arduino off of straight 12v from the empeg. Not the Pololu.
• Opening up an empeg and checking on internal I2S jumper wiring; so far the only commonality is that both empegs have had that internal wiring mod and the glue down of the wires. Fiddle with those wires and try to make them crackle and/or not crackle. Couldn't get either to happen with wire fiddling. And besides, both empegs worked for weeks prior and I didn't see any reason for them to have changed.
• Retouching the solder joints on the I2S pads on the inside of the empeg.
• Warm or cold. Doesn't seem to matter. Crackles the same on test bench in warm house, or in freezing cold in car trunk. Prior weeks-long successful run was the same way: When it was successful it didn't seem to matter whether it was cold or warm.
• Checking to see if the crackles coincided with any kind of messages on the serial debug console. There was no commonality. The crackles would come when there was nothing going on in terms of what the Bluetooth or Empeg was doing on its console. (Music has to be playing though, so the empeg is still putting out a timestamp once per second no matter what.)
• Using standoffs to separate the distance of the Arduino and the Betz board from the BlueGigaEmpeg PCB more, to see if it's interference causing digital bitstream errors. I think that maayyyyybe the frequency of the pops might have gone down when I did that but I am not sure.
• Different BlueGigaEmpeg PCB (going back to the rev3 with handmade voltage dividerss versus Rev6 with built-in voltage dividers). I only tested the old PCB for a few minutes, and the crackling is intermittent enough that it's possible that I just hadn't waited long enough to be sure it was gone with the old board. When I went back to the old rev3 board a second time, the crackling was back just the same as before.
• Going back to my old Rev3 PCB (with hand-built voltage dividers) and my oldest working Betz board (the second unit that I didn't fry) reduces the frequency of the pops significantly. However they are still there, they're just rarer. That likely explains why my first attempt at going back to the Rev3 PCB seemed to fix it on the first try.
• This go-back-to-Rev3-PCB thing that reduced the pop frequency was with no Pololu power supply at all, and instead was the Arduino running off 12v pure from the Empeg. However, removing the Pololu from the Rev6 PCB and running that at 12v pure did not fix the issue either, so that's not it.
• Which Betz board that I use in the process doesn't seem to matter with regard to the frequency of the crackling, though it's hard to tell. Perhaps maybe my oldest unfried board is maaaaaaaybe the least frequent of the crackling but I can't really tell for sure, it's hit or miss.
• Tried switching to 6v Pololu and also the 5v Pololu instead of 7.5v pololu. Still get crackling, though much more rare. Could be because I tried it on the old Rev3 board which was rarer to begin with. (Update: Yeah, it was the old board that made it more rare. 6v Pololu on the Rev6 board has more frequent crackling.)
• Note that the 6v Pololu seems to drive the Arduino power supply enough to get 4.8v out of its 5v output and that seems to be enough to drive the MAX232 chip.
• Using my oldest working non-fried Betz board with the Rev3 PCB and the 6v Pololu had rarest crackles. One of the next batch of betz boards (one of the the two "recovered" betz boards with the green checkmark) in the same configuration produced slightly more frequent crackles.
• Oldest working non-fried Betz board on the Rev6 PCB with 6v Pololu had just as frequent crackles as the newish "recovered" betz boards, so It's not related to which Betz board I'm using.

What does fix the crackling but does not help:

• Switching to analog audio. It confirms that the issue is in the I2S and not in the Bluetooth. But I want this thing to work with digital audio goddamn it.

TO DO:

• Try yet another different Betz board when you get in the new China shipment of 10. Not expecting much from this.
• Most promising but most disappointing: Try changing routing of I2S audio wires on the BlueGigaEmpeg board to look for possible proximity interference issues.
• See if grounding AGND on the Betz board helps. It should already be grounded through its own jumper, and also shouldn't make a difference since it's for analog audio data, but check it anyway.
• See if putting the 470uf cap at C16 back into position on the Betz board works? Now I'm questioning why I even removed it at all in the first place.

[mlord]

Could it be a grounding issue? Eg.. run a wire from GND on the empeg (eg, the chassis) to GND on the Betz board, and see if problem goes away.

[tfabris]

Thanks @mlord, I will definitely try that! I have high hopes for putting the C16 470uf cap back as well, I'm about to try that right now.

[tfabris]

Tried the following, unfortunately did not work:

• Adding additional grounding between bluetooth board and the empeg chassis as @mlord suggested.
• Adding additional grounding to the AGND line on the bluetooth board.
• Removing grounding to the AGND and cutting the AGND jumper on the bluetooth board.
• Adding back the 470uf cap at C16.

Tried the following which seemed to maaaaaaybe help:

• Used long standoffs to separate the Betz board from the BlueGigaEmpeg board by about an inch or two. This seemed to reduce the crackling but did not eliminate it.

TO DO:

• Working on the theory that this is some kind of proximity interference, try rerouting the I2S connections on the BlueGigaEmpeg board completely and see if their positioning affects the outcome.
• Working on the theory that adding back the C16 470uf cap was the right idea, but that all my Betz boards now have a damaged internal regulator inside the WT32i chip because I removed the cap, await the arrival of the new set of Betz boards from China and see if leaving the cap in place on a new board solves the problem.

[tfabris]

Works:

• Handmade test, running the I2S connections and the voltage divider resistors outside/away from the board assembly.

This tells me that the problem is some kind of interference, but what/how. Why did the R3 board have less of it than the R6 board?

Things to think about:

• R3 board had hand-made voltage dividers with slightly more separation between their lines than the R6 one has on its copper traces.
• R3 board had those voltage dividers positioned over two layers of ground plane, and that ground plane was between the vdivs and the Arduino CPU. Perhaps it's Arduino RF emissions tangling with the signal on the I2S lines?
• The I2S connections, the wires themselves, on either the R3 board or my handmade test, might be a bit thicker than the traces on the R6 board. Maybe thicker traces on the next R7 board?

[tfabris]

Moving the SCK wire closer to the Arduino produces some pops.

[tfabris]

Moving WS wire closer to the Arduino did not immediately produce any pops.

[tfabris]

Moving the SD wire closer to the Arduino did not immediately produce any pops.

Theory:

• Only the SCK wire is the issue, and we merely need to route it as far away from the Arduino as we can and make sure the voltage divider resistors are protected by ground planes.

[tfabris]

Tried:

• Routing the SCK wire differently and putting all voltage divider resistors on the opposite side of the PCB as the Arduino. Also completely cutting the original SCK trace at both ends so that it doesn't act as an antenna. This didn't fix the problem. My guess is that it's because in the Rev6 design there's not a lot of ground plane shielding between the voltage divider resistors and the Arduino.
• Fashioning a makeshift RF shield out of aluminum foil covered with tape and using it to cover the Arduino chip itself. Didn't really seem to do much, but that might be because I didn't ground it well or because it wasn't properly shaped.

Stumped, because the act of printing an entirely new PCB with a theoretical design is a bit of an outlay of time and expense and I'd hate to do all that and then not have it be fixed.

[tfabris]

Tried:

• Cutting traces for all three I2S lines across the entire BlueGigaEmpeg PCB and, while still leaving the voltage dividers in place, running wires up and over the Betz board. This significantly reduced the number of pops though it did not completely solve it. Promising though.

Consider:

• Possibly a complete redesign of the PCB which rotates the Betz board 180 degrees to place the I2S wires closer to the tuner connector so that I don't have to run I2S traces all the way along the BGE board. Shortest possible I2S runs. Maybe even reposition the RESET/PAIR switch/LED and other elements, to allow the Betz board to get close to the connectors since I haven't finalized the enclosure design either. This would be a large pain but it might be the only way to be sure.

[tfabris]

To do:

• Read the heck out of this: http://www.ti.com/lit/an/szza009/szza009.pdf

[mlord]

Good work. Google this: "isolation routing", and try applying it to the SCLK line (and others if you can). I think sn00p even posted some nice photos of such on the BBS (EDIT: found the post, but photos are gone..).

[tfabris]

Hey @mlord what do you think of this? This document here.... http://www.ti.com/lit/an/szza009/szza009.pdf ... says something interesting:

"In the case of an I/O pin in the input mode, the capacitance of the unused output transistors transfers noise from both power rails to the pin. The amount of noise is based on the impedance of whatever is connected to the pin. The higher the impedance, the more noise comes out of the microcomputer. That is why unused inputs should be tied to the lowest-impedance rail: ground, by direct short, if possible."

Do you think it would be worth it to try simply grounding all the unused analog and digital I/O pins on the Arduino? Right now I just leave them untouched.

I could also, in the code (i.e., no physical board changes), change all I/O pins from their default setting of "INPUT" to the alternative "INPUT_PULLUP", but I don't know if that would help. I was worried that changing them all to pullups would require more overall power draw for the Arduino board.

[tfabris]

I'm reading some stuff on the internet such as this... https://arduino.stackexchange.com/questions/574/should-all-unused-analog-input-ports-be-tied-to-ground-for-accurate-a2d-measurem https://forum.arduino.cc/index.php?topic=57067.0 http://forum.arduino.cc/index.php?topic=15495.0 ... which says that setting the pins to either "INPUT_PULLUP" or setting them to "OUTPUT" and "LOW" (essentially pulling them to ground) will reduce noise emitted by the Arduino. The Pullup uses a small amount of energy, Output/Low uses the least energy. Does that sound right to you, @mlord?

[tfabris]

To Do:

• Try grounding the RESET line and holding the Arduino in reset mode to see if that stops the interference. That will narrow it down to be the Arduino for sure. Suggested by Shonky on EmpegBBS. https://programmingelectronics.com/how-to-use-an-external-reset-button-with-arduino/ Pins go VIN GND GND 5V 3v3 RESET IOREF ???
• Try setting all unused pins to Output/Low or INPUT_PULLUP and see if that stops the interference.

[mlord]

You are not using the A2D (analog to digital sampling) pins, so there is no need to fuss much about those particular pins. But setting all of the unused ones to INPUT_PULLUP shouldn't hurt -- the internal pull-up is 10's of K-Ohms, so the power draw/waste ought to be pretty small.

[tfabris]

Tried the following, none fixed the issue:

• Setting all unused pins, both digital and analog, to OUTPUT/LOW.
• Setting all unused pins, both digital and analog, to INPUT_PULLUP
• Holding Arduino RESET line to ground.

I'm not sure if holding the Arduino reset line to ground truly sleeps the chip or not, so more investigation is needed there.

[tfabris]

Tried, did not fix the issue:

• Putting arduino chip to sleep with this code:

  set_sleep_mode(SLEEP_MODE_PWR_DOWN); sleep_enable(); sleep_cpu();

[tfabris]

Tried, based on @mlord suggestion:

• Changing 10k voltage dividers out for lower values to increase S/N ratio on the I2S lines at the expense of increased current draw from the empeg.

Result: With 2.33k of resistance (doubled up 4.7k's) there was a significant reduction in pops, though not completely fixed.

TO DO:

• Try 2.0k resistors and redesign the Betz board to rotate it 180 for shorter I2S runs.
• Check empeg interior for hot components in case the additional current draw is a problem.

[tfabris]

Tried:

• Going down to about 1.55k ohms on the voltage dividers by tripling up the 4.7k resistors (my PCB is a work of art, I tell ya).

Result:

• Pops are completely gone.
• I'm not noticing any unusual heat from the SA7705H or the WT32i.

To do:

• Order a passel of 1.5k resistors.
• Run it this way for a week or so to see if the empeg fries.
• Decide if that's enough, or if I still need a board redesign, too. Keeping in mind that the pops are not noticeable when the music is loud, they are only noticeable in the quiet passages and only if the headunit stereo volume is turned up to a decent amount.

[tfabris]

Closing issue. 1.55k resistors seems to work and I have 1.5k on the way. Happy dance!