DAC death

[sbourdeauducq]

Every ~15s my Zotino stops producing an output signal for a few seconds, and the blank period is followed by a burst of noise: This is with a kernel that produces a 1kHz tone:

import struct

from artiq.experiment import *
from artiq.coredevice.ad53xx import *

class Tone(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("zotino0")

    @kernel
    def run(self):
        self.core.reset()
        self.zotino0.init()
        delay(1*ms)
        while True:
            self.zotino0.write_dac(1, 0.1)
            self.zotino0.load()
            delay(0.5*ms)
            self.zotino0.write_dac(1, -0.1)
            self.zotino0.load()
            delay(0.5*ms)

Even with nothing running on the core device, the bursts of noise are still observed, which points to a hardware problem:

[dhslichter]

Looks like AD suggests powering up as -12V, +12V, +3.3V:

https://ez.analog.com/thread/47483

Is it possible we are damaging something (protection diodes?) by going in a different order?

[gkasprow]

I've noticed that. It is recommended pattern. We should fix it in next revision just in case. I applied other sequences to the devkit and nothing happened.

[hartytp]

@dhslichter I've seen that post before, but I don't think it applies to us. Importantly, Zotino should meet all the absolute maximum ratings by design. e.g. the digital sequencing shouldn't matter, since the inputs/outputs connect to ICs that run from the same regulator as Zotino.

Also, I've run eval boards for this DAC from bench top PSUs with no real care paid to power sequencing and never seen an issue like this.

[hartytp]

The reference can survive up to 40V on its input.

The data sheet says 5.5V max for VREF0 and VREF1 (not that I think this is our issue)

[hartytp]

hmm I really can't think of anything to do here apart from removing all components from a working board until we strip it down to the very most basic design. If that works, then add things one by one and try to see which causes DAC death.

[gkasprow]

Another observation - the board with issues, has them from very beginning. There are boards that work perfectly fine. I tried with temperature and various supply sequences on devkit and they didn't harm it. I will try the same with the board that works fine. But it's quite possible that we never find the problem in this way. I'll do yet another thing - buy a few DACs and will work on the same board that causes issues. I will replace the DAC on particular board and will see if the issue reappears. If not, I will play with voltage sequencing and logic levels during power up...

[gkasprow]

@hartytp I mean 40V for reference IC.

[dhslichter]

Also, I've run eval boards for this DAC from bench top PSUs with no real care paid to power sequencing and never seen an issue like this.

Same here, I merely mentioned it in case it might make a difference.

[sbourdeauducq]

Cooling the DAC chip works around the problem here as well.

[dnadlinger]

Just for the record, one of our boards (v1.2) has developed this issue now as well (N12V0A dropping to ~2.5V every few seconds). Adding fans pointed into the general direction of the DAC chip seems to work around the problem. (While the board was mounted with copious amounts of space to either side in the rack, and vertical airflow was unrestricted, no fans were used previously.)

[gkasprow]

I've ordered a bunch of DAC chips and will play with them to identify a moment where the DAC starts consuming excessive power on negative rail. Did not manage to do it with devkit.

[cjbe]

To add to @klickverbot's comment, the v1.2 board we saw this problem on was new, and showed the problem within a few hours of powering it on for the first time (i.e. we powered it on, and saw the problem as soon as we looked for it).

There has been a v1.1 Zotino sitting in the same airflow within 10cm of the problematic v1.2 - this v1.1 Zotino has been working for a month with no issues.

[jordens]

I enabled OT shutdown by default. This should at least keep the chips from dying in many cases. https://github.com/m-labs/artiq/commit/2c344686d9b0f8c3f8cc65db692468e7ac443310

[sbourdeauducq]

Here is a picture of the DAC on the faulty board, in case this is an issue with a certain batch of chips.

[hartytp]

@gkasprow any luck tracking this down. I'm going to buy a few of the DAC chips and have a look at the dead board here.

[hartytp]

@gkasprow did you see that you got a follow up question from ADI? https://ez.analog.com/message/350275-failing-ad5372-dacs

[hartytp]

@gkasprow Having said that, I don't see the BCPZ package stocked in any of the usual places on less than a 6 week lead time! Am I missing something?

[jordens]

That might be a production issue. Digikey says that delivery is past due.

[hartytp]

Yes, and the power detectors for booster are on 1 month lead time as well...

[gkasprow]

Lead time is serious problem that touches the industry. The economy in China boosted so much that they consume all resources. Look at the prices of 100nF capacitors - they increased over 300% last year. It's extremely hard to get i.e. 10uF X7R 0805 capacitors.

On 14 June 2018 at 16:42, hartytp notifications@github.com wrote:

Yes, and the power detectors for booster are on 1 month lead time as well...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sinara-hw/Zotino/issues/2#issuecomment-397320768, or mute the thread https://github.com/notifications/unsubscribe-auth/AEH-vrZrYTLnoqZXC4Um4C0K3AkK8kiHks5t8nZjgaJpZM4UGrU0 .

[hartytp]

Ouch!

Out of curiosity, did you order any more DACs to help debug this issue? If not, does that mean that we can't do any work to try to fix Zotino for another 6 weeks?

[gkasprow]

I did, tomorrow I will take them from Technosystem. Ordering things at the WUT is real nightmare.

On 14 June 2018 at 21:53, hartytp notifications@github.com wrote:

Ouch!

Out of curiosity, did you order any more DACs to help debug this issue? If not, does that mean that we can't do any work to try to fix Zotino for another 6 weeks?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sinara-hw/Zotino/issues/2#issuecomment-397418242, or mute the thread https://github.com/notifications/unsubscribe-auth/AEH-vl88fxzcvWnI63wM96-X2DeG5qxcks5t8r83gaJpZM4UGrU0 .

[hartytp]

:+1:

[dhslichter]

Or buy from Verical here, or directly from Analog Devices here. For reference, Octopart is also a pretty awesome tool for figuring out who has what in stock....

[gkasprow]

It's not a problem with buying things but letting WUT bureaucracy to pay for them. It takes months. Doing via private company is simpler :)

[hartytp]

directly from Analog Devices here

Only if you want to buy a tray of 260 DACs.

Or buy from Verical here,

even there, it's a MOQ of 8 and two weeks before shipping. Plus the fees/paperwork involved with shipping from the US.

I was just surprised that none of the usual suspects stock this atm, but I suspect it's because we're using the obsolete package.

[hartytp]

Switched the DAC for a new one, but the N12V0 is still at 0.7V.

Maybe the negative regulator is struggling to startup with the heavy load presented by all the OpAmp biases. I'll add an RC delay on the P12V0 and see if that helps...

[hartytp]

@gkasprow Two observations, which might help to explain all this:

1. The negative regulator is 200mA, while the positive regulator is 500mA. So, it should be expected that the positive rail will always switch on faster than the negative rail
2. The negative regulator goes high-Z when its output is reverse biased.

Given the relatively high forwards voltage of the Zener diodes installed on Zotino by default, it should thus be expected that the negative regulators will often go into lock up

[gkasprow]

did you add the schottky diode?

[hartytp]

Yes, but the one I added seems to have died...

[gkasprow]

I'm testing the board I have and power sequencing looks OK

[dnadlinger]

Hmm… The 1PS79SB30 that I believe @hartytp used supposedly take 300 mA peak forward current – if it died from that, where is this much current even being sourced from safely?

[gkasprow]

another scale

[hartytp]

hmm...something still doesn't add up here. I added a delay on the +12V LDO startup (120k+20k divider and 1uF capacitor). Despite the +13V being present, the negative LDO on this board does not even try to startup; the N12V0 line does nothing until the +12V line comes on, when it goes to one diode drop above ground. Will look carefully for dry joints etc...

[gkasprow]

are U sure there is -13V present ?

[gkasprow]

It could happen that Zotino starts with some load connected. Then the current drawn from negative source could be much higher causing the LDO startup to fail and triggering the latchup in DAC.

[hartytp]

are U sure there is -13V present ?

Yes, the +-13V rails start up nicely.

It could happen that Zotino starts with some load connected. Then the current drawn from negative source could be much higher causing the LDO startup to fail and triggering the latchup in DAC.

Yes, but right now I'm not even seeing a hundred mV negative at startup, which is surprising...

[gkasprow]

Maybe the negative DAC supply is shorted with some output?

[hartytp]

Maybe the negative DAC supply is shorted with some output?

What would that explain? My observation is that when the N13V0 supply comes on, the N12V0 does not produce any voltage (even -20mV or so). After 20ms or so, the P12V0 regulator switches on and the N12V0 rail goes to +0.7V (the Zener diode forward conductance). That suggests that the N12V0 regulator is dead. But, it should be very hard to kill this regulator, so I have no idea what happened...

[gkasprow]

this would explain too high current drown by negative DAC rail. Is the LDO getting hot?

[hartytp]

After disconnecting the positive LDO shutdown pin, the negative rail now comes on to -0.5V. +12V supply current draw is 100mA.

[hartytp]

What's odd is that the LDO output starts up very slowly (maybe 10s to reach -0.5V). But, the +12V supply current goes immediately to 100mA (even when the LDO output voltage is very low). The -12V rail discharges equally slowly, suggesting that it's not just shorted.

[hartytp]

Is the LDO getting hot?

Not noticeably

[gkasprow]

When I was playing with Sayma, i noticed that negative LDO can disable itself when started with positive voltage. But Schottky diode managed to prevent such case. You can send it to me for further examination.

[marmeladapk]

@gkasprow We have one Zotino which overheats, mounted leftmost in our 3U crate.

[gkasprow]

I'm just using it.

[hartytp]

@gkasprow The 100mA current draw I see on the 12V rail when the positive LDO is shutdown is consistent with the expected current draw from the LVDS interface. So, it seems that on this board the negative LDO is just dead.

Let me post this Zotino back to you for further examination. The symptoms it displays (no -12V supply) are different to all the other Zotinos we've looked at so far, which all have high current draw on the negative rail, so it's probably a different issue entirely.

[hartytp]

@gkasprow let's confirm once and for all that we understand what's going on here. How about this:

• take a working Zotino, which does not display this issue. Maybe try taking the damaged one and replace the DAC and check that it behaves normally, according to the data sheet
• replace the Zener diodes with Schottky diodes such as SS14
• use a VHDCI carrier card to connect Zotino to a remote programmable 12V supply
• connect the -12V supply to a digitally readable voltmeter
• do not fan cool it, so that any issues with the DAC are easy to spot (fan cooling should not be necessary to avoid damage)
• now run the following measurement 1000 times or so:
  • power Zotino on
  • wait a second or so for everything to start up
  • measure the negative supply rail a few times over 1 second to look for issues
  • power the board down
  • wait a few seconds
• if that works then replace the Schottky diodes with the Zener diodes and repeat the measurement

Hopefully, we will see that nothing breaks with the Schottky diodes, but it dies quickly with the Zener diodes. In that case, we know that we can resolve it by swapping the diodes on all boards as well as any dead DACs.

How does that sound? Can you do that measurement at some point in the next few days, please?

[gkasprow]

@hartytp Yes, I will.

[hartytp]

Amazing, thanks Greg.