3U DAC (Zotino) tests & rev 1 errata

[gkasprow]

Issues:

• [x] Use two output resistors in series/parallel to improve power handling (200mW short-circuit)
• [x] reverse C143, C145, C147, C84
• [x] mirrored SOT23 diode footprint due to Altium bug - part of component is on top layer, pads are on bottom layer.
• [ ] add Shottky diodes at the LDO outputs
• [ ] 3mm higher EMC shield on PSU
• [x] reverse IDC connector on BNC to IDC board
• [ ] Zero-index all channels
• [x] swap BNC connector GND terminals
• [ ] either add ~2mm plastic washers on BNC barrel behind front panel or move BNC connectors 2mm towards FP. This is to enable fixing the BNC connectors to the panel using the screws.
• [x] Please remove the annotation "This module connects to Kasli ..." and replace it with something like "EEM connector: IO are LVDS, I2C is 3V3 LVCMOS, P3V3_MP up to 20mA, P12V up to 1A."
• [ ] run PI and thermal simulations
• [ ] Can we run the digital interface from 2V5?
• Temperature controller issues:
  • [x] Decrease the thermal resistance between the thermal island and the main PCB/increase the thermal mass of the island by at least 1 order of magnitude -- without active stabilization, the DAC should only be 1-2C hotter than the main PCB. This should also improve the electrical ground connection between the main PCB and the thermal island.
  • [x] Remove current temperature controller and DAC over temperature shutdown circuitry.
  • [x] Add mounting holes/pads for peltier + heat sink controller by external circuit.
  • [x] Leave temperature probe on the thermal island, bring it out to an external connector, which should also provide +12V power, ground and I2C (maybe use I2C-controlled GPIO instead? This way we can do bit-banged SPI if we need to).
  • [ ] Assuming the peltier + heatsink mount on the top side of the PCB, do we want to add a stick-on insulation on the bottom side to screen the DAC from external air currents?
• [x] Silk + schematic + panel should be "v1.x" not "rev1.x"
• [x] Only have one set of +-12V regulators, not two!
• [x] Reduce SMPs outputs slightly, since we're only using +-12 regulators

Tests that can be done without an Artiq driver:

• [ ] Measure step response. This means measuring the 90%-10% rise time for a full-scale signal. If easily possible, also measure the 1% settling time.
• [ ] Measure noise PSD (0.1Hz to 10MHz) with full scale and mid scale output signal
• [ ] Glitches on LDAC, major carry, etc (trigger off the SPI)
• [ ] Confirm that temperature controller works. Measure temperature stability of DAC in lab conditions
• [ ] Measure drifts with temperature controller running. Set one channel to Vmax/0V, and look at drift over ~24 hours to confirm gain and offset drifts. Measure board temp co (e.g. use hot air gun to heat the board up while measuring output drifts).
• [ ] susceptibility to external EMI sources (i.e. neighbouring boards). I will test it with EMC equipment.

Tests that need an Artiq driver:

• [ ] Measure cross-talk (full-scale square wave at ~50kHz on one channel, another channel at 0V)
• [ ] Measure spurs due to SMPSs, digital-cross talk, pickup etc. Have one channel updating as fast as possible and measure noise on another channel
• [ ] Check for thermal issues (drifts) when outputs are slewing at max rate (e.g. testing for issues due to self-heating in the OpAmps). What’s the best way to test this?
• [ ] Measure harmonics with a full-scale sinusoid output at 50kHz.

Configuration for measurements is: DAC->3m SCSI cable->HD68 to IDC->IDC to BNC->scope/DVM/noise analyser. CMCs and a LPF on the IDC-BNC board as previously discussed. LPF chosen to avoid degrading the DAC noise or bandwidth too much.

[hartytp]

@hartytp well, it is purely historical. Initial idea assumed different supply levels for DAC and amplifiers.Then we decided to have 10V range instead of 12 due to the price of precise resistors so I only modified the LDO resistors. One can also make DC/DC voltage lower 13V would be enough.

Okay, in the next revision let's scrap 1 set of +-12V regulators and reduce the SMPS outputs. I've added a note to the top post.

[cjbe]

@gkasprow I am trying to bring up a Zotino. When I power it up and program it I get the correct voltages out of the DAC. However after a minute the DAC outputs start dropping out to 0V for ~500ms every 2s. This appears to be due to the DAC_VSS supply cutting out periodically. The input to IC33 (the DAC_VSS reg) does not drop out.

When the supply drops out the 'Lo' LED comes on, and when the supply is good the 'OK' LED comes on. The 'Htr' LED is permanently on. However the heater jumper (SW1) is not present, hence the heater should not be doing anything? Also the P12V0 supply for the heater circuit is steady, so this correlation is not due to a common power rail dropping out.

Any ideas?

[cjbe]

Aha - the hacked on over-temperature cutoff was activating. Even without the heater, the DAC temperature rose above 50 C in a warm lab with little airflow. Adding a fan keeps the DAC around 40C.

[jordens]

It would be interesting to benchmark the thermal simulations on Zotino. What's the prediction for the DAC temperature on the island at its idle power with no/little air flow?

[hartytp]

Even without the heater, the DAC temperature rose above 50 C in a warm lab with little airflow. Adding a fan keeps the DAC around 40C.

Ouch! That's not good.

We really need to fix this soon for the v1.1 boards (next revision), as the current design makes Zotino pretty hard to use.

My suggestion is this:

1. Increase the thermal conductivity between the thermal island and the main board by at least 1 order of magnitude. When idling without active temperature control, the DAC should only be a couple of degrees above the rest of the board (this is important, as these boards should be designed to function properly even without active temperature control). Once this is done, temperature controlling the DAC will also control the main PCB to a certain extent. AFAICT, that should be fine if we use a peltier.

2. Remove the temperature controller circuitry from the PCB -- this kind of simple temperature controller is unlikely to work given the wide range of ambient conditions that it has to operate in.

3. Also remove the DAC's thermal shutdown, this should not be necessary once we improve the design of the thermal island

4. Keep the temperature sensor on the thermal island, and bring it to a header which can connect to a separate temperature controller PCB. This header should also supply +12V power for such a board. Could also connect I2C to this header if we want to provide control interface for the temperature controller.

5. Add mounting holes near the thermal island that could be used to mount a separate temperature controller PCB + peltier.

6. At some point in the future (not a high priority at the moment) we can design a separate temperature controller PCB. The stackup is: DAC, peltier, temperature control PCB. This PCB has a heat sink, peltier driver and microprocessor. Are we okay mounting the peltier onto the DAC, or would it be better to mount it directly onto exposed copper on the thermal island?

@gkasprow @cjbe @jordens Are you happy with that plan?

[jordens]

Yes. Right now I'd maximize passive temperature stability by minimizing thermal impedances and maximizing thermal mass (opposite as of now). The thermal impedance between island and rest of the board could even be made a placement option during fabrication. Later, and with a peltier we can stabilize DAC temp around its average. There are already plenty of commercial temperature controller PCBs around. Add a mounting hole or two for the controller PCB and a header for sensor, I2C, and 12V.

[jbqubit]

Is the current front panel width enough to accommodate the added thickness from the separate temperature controller stack?

[hartytp]

Is the current front panel width enough to accommodate the added thickness from the separate temperature controller stack?

If not, one can easily buy a separate filler panel, so this isn't something we have to worry about for this design IMO.

[gkasprow]

For use cases that need special temperature control, we can install general purpose temperature controller in neighbouring slot. I mean one that will be developed as a part of planned hardware.

[gkasprow]

And for this purpose we may simply add some connector that would give access to the temperature sensor and Peltier solder pads. The connector can be located close to EEM and we could use 6 pin IDC. The same one would be placed on the temperature controller board

[hartytp]

And for this purpose we may simply add some connector that would give access to the temperature sensor and Peltier solder pads. The connector can be located close to EEM and we could use 6 pin IDC. The same one would be placed on the temperature controller board

For use cases that need special temperature control, we can install general purpose temperature controller in neighbouring slot. I mean one that will be developed as a part of planned hardware.

Sounds good!

[hartytp]

Top post updated with new plan.

[cjbe]

@gkasprow I am having a few issues with the Zotino prototype you sent me:

• DAC channel 7 has stopped working. It was working when I received the board, and the board has not been through any trauma. The failure appears to be in the DAC chip itself (output is ~12R to ground)
• There is some failure mode of the board that sometimes causes all of the DAC outputs to drop to zero briefly. Once this failure is triggered, the outputs cycle between 0 and correct once every few seconds. After a while, or after power cycling, this stops. I have not been able to reproduce this properly with instrumentation attached.

Also a few other minor niggles:

• The op-amp on channel 1 was dead on the board I received - I replaced it, and have not seen any more die.
• J5 (one of the DAC output IDCs) was populated the wrong way around.

[jordens]

@cjbe How are you handling the thermal issues now? Turning off the temp control, adding a heat sink on the DAC or with just the fan?

[gkasprow]

@cjbe The boards I modified could have some trauma before I discovered the mechanism that killed the DAC chips. So it's quite possible that your DAC was already slightly broken which explains why it self-triggered the thermal protection circuit which does not happen before on boards without heater enabled. So I'd suggest replacing the DAC chip - it's easy providing that you have hot air gun and the flux. If you have troubles with it, ship it to me and I will replace it.

[cjbe]

@jordens the Zotino is sitting next to an A/C vent in an open frame rack, and has a fan pointing at it. The thermal management on the Zotino is disabled, but the DAC internal thermal cutoff is enabled, but has not ever tripped (it trips at a junction temperature of ~120C!).

[cjbe]

@gkasprow I already have one on order - I will let you know if I see any funnies after replacing the DAC.

[hartytp]

@gkasprow (or @cjbe if you have time once your new DAC arrives): I think there are enough issues with this version of Zotino that it has become so hacked/fragile that further testing is hard. So, I propose that we do a couple of very quick tests to confirm basic performance and then move on to v1.2 fixing all known issues.

Please could you make the following measurements:

• 10% to 90% rise time for a full-scale signal
• Noise spectrum (nV/rtHz) for 0V and +-10V signals. Ideally, use an instrument with a very low noise floor for this (1nV/rtHz or so) and measure in the bandwidth of something like 0.1Hz to 1MHz.

If those measurements look okay then I'm happy moving on to v1.2

[hartytp]

@cjbe how did the new DAC work out? Any luck?

@gkasprow I know there's a lot on at the moment, but is there any chance you could do those two tests (rise-time and noise spectrum) next week? In it's current state, Zotino doesn't really work well enough for prototype testing/evaluation, so I'd really like to try to send v1.1 (the next revision) off for manufacture before Christmas if at all possible.

Thanks!

[gkasprow]

@hartytp some rise-time measurements were already done scroll the page up. I have a setup done so will do more measurements once I ship Urukul 9912.

[hartytp]

Thanks for the reminder @gkasprow

Looking at rise-time measurements in detail:

For the main slope: dV=17.6V, dt=25us which is broadly consistent with a slew rate of approx 0.7V/us. That's about what we expect: DAC slew rate is 1V/us, OpAmp is 20V/us.

This is a bit slow compared with @dhslichter's SPICE simulations (see plot on schematic). Presumably that is because the step response plot is only for the filter, and doesn't include the DAC's slew-rate limit? (@dhslichter can you confirm this?)

Primary overshoot looks to be about 120mV and settles out in a bit under 10us. Looks like our circuit is a little more damped than the situation (120mV versus something like 180mV in the sim).

Overall, those rise-time measurements look fine.

Point of the currents tests is just to make sure that this design more-or-less meets specification; we will make more careful measurements on v1.1 (with bug fixes). From that measurement, it looks like the board does behave properly from a rise-time perspective.

Once we check the noise performance, I'm happy to move onto v1.1

Edit: note that the numbers above are all read from a graph and so a bit rough. Looking for factors of 2 or more discrepancy between simulation and data, not worrying about 20% here or there!

[hartytp]

Answering my own questions: yes, AFAICT, the simulation does only include the filter (see https://github.com/m-labs/sinara/issues/185). This explains the slight differences between filter and measured performance. NB we should either remove this simulation or document what it means more clearly!

[dhslichter]

Yes, it's just the filter.

[cjbe]

@gkasprow I replaced the DAC on my Zotino. This fixed all of the output channels.

After an hour of operation the DAC chip remained cool (~30C). However after leaving the Zotino powered overnight the DAC chip was very hot (~90C). After turning the power off and letting it cool down, then reconnecting it, the DAC chips gets very hot in a few minutes.

This seems to suggest something very alarming is going on. @gkasprow : any suggestions?

[cjbe]

I made some noise measurements of the prototype Zotino. Note these are with a DAC chip that is behaving oddly (getting very hot), but I think all the interesting artifacts are not related to the DAC:

The noise floor is about 350 nV/rtHz. The pickup with fundamental 44.5 kHz is from the switching power supply. The amplitude of the fundamental is 70 uV rms.

I measured with the DAC output at 0V or 10V. The Zotino was connected to the test breakout board with 20cm of ribbon cable, then to a floating input of a Rhode & Schwarz UPV audio analyzer.

[gkasprow]

@cjbe did you disable my overtherm protection circuit? Heating was also disabled? Cooling was on all the time?

[gkasprow]

@cjbe The DC/DC convertes run at much higher frequency, so I'm not sure if this is a source.

[cjbe]

@gkasprow I believe it is the DC/DC, as I looked at the -Vout pin of IC1 and saw ~10mV of ripple at 44.5 kHz.

[cjbe]

@gkasprow I disabled your over-temperature circuit (it is still present, but I stopped it disabling the DAC supply). I have enabled the DAC internal over-temperature protection, which triggers at 120C, but so far this has never triggered.

I have the card sitting on my desk in a ~25C room - there is no active cooling. As it was happy for an hour, I assumed that there was no possibility of thermal runaway.

[gkasprow]

@cjbe it happened to my board that after some time, when the temperature or the air flow in the room changed, it got its thermal runaway.

[gkasprow]

And these 44.5kHz are surprisingly low . I'd expect at least 200kHz judging the converter size and its power. I'll look deeper to these ripples.

[a-shafir]

I have not found CC6-1212DF-E base freq but by the emi testing docs it looks like ~600KHz. TPS62175DQC is around 1MHz

44.5kHz and it's harmonics can be some interference from the D-class audio around (via ground loops etc). Or it can be noise from the digital circuits, the controller (KC705?) etc.

[gkasprow]

@cjbe could you connect just power supply to the DDS board, no interface and measure it one again? Or the best would be to operate it from 12V battery to make sure nothing gets via the supply lines. Shat wall wart do you use? Some of them have very poor EMC performance

[cjbe]

@gkasprow I am now using a nice linear power supply to power the Zotino. This plot shows both the voltage at the Zotino power supply input (Net P12V0), and directly at the output pin of the switchmode (-Vout):

So it looks like the ripple is not coming from the power supply to the zotino, but is generated on the Zotino. It looks like the switcher frequency is ~600 kHz, but there is ripple at ~41 kHz

[cjbe]

A significant part of this issue seems to be due to the DAC overheating:

After directly after the Zotino is turned on it draws ~290 mA at 12V, and the modulation on the output at 41 kHz is at ~-110 dBVrms. Over a few minutes the current creeps up by a few tens of mA, then fairly rapidly climbs to 500 mA, then wanders between 500 and 640 mA over tens of seconds. As this happens, the modulation peak increase to ~-80 dBVrms.

[gkasprow]

@cjbe your DAC is dead, negative LDO overheated and thermal shutdown gets enabled. This looks like the DC/DC converter gets unstable and its internal feedback loop causes oscillations at 41kHz

[hartytp]

@cjbe What's happening to the reference voltage during this period?

Thinking outloud here, but:

• Let's assume that when the DAC says its max operating temperature is 120C it really means that it can take this CW while still meeting specification. Moreover, let's assume that your tests do show that the DAC die never exceeds 120C. As a result, the DAC overheating isn't the problem.
• The reference is only rated to +70C. Given that it's also on the thermal island I can well believe that it can exceed this.
• The DAC can only take 5V5 reference input, so if the reference fails and applies DAC_VDD (+12V) to this pin, that will probably kill the DAC.

[a-shafir]

it is not impossible to have parasitic oscillation process between the switching regulators and the linerar regulators etc. There are some inductors in the power circuit etc. Also it can be interference between IC1/IC35 (including on the harmonics).

The regulators control loop b/w most likelly is not enough to suppress it fully.

The capacitors used on the board (22uF etc) are relatively small considering the load currents. Also the power filtering instructors can be close to saturation and increase the stability problem.

@cjbe can you try to add a big (over 100uF) tantalum capacitor parallel to C136, C130, C131, C141, C133 etc (can be one by one)?

If this is related to the oscillation like i suspect shall at lest change the ripples so it will indicate the problem.

[hartytp]

@a-shafir No, this is just the SMPS in overcurrent mode. It's only a 0.25A model, and once the DAC fails we draw a hell of a lot more than that.

Adding large caps probably will change the frequency of this oscillation, but that doesn't tell one much (it would be true for many different noise sources).

[hartytp]

@cjbe Thanks for the noise measurement! Pretty sure that's not actually the noise floor of the circuit when it's working properly! If the SMPS is in overcurrent mode then this data is all suspect. :(

[gkasprow]

be careful with adding too many capacitors at the SMPS output - it can easily became unstable. Inductors have Isat = 950mA, so there is no way to saturate them. Ripples at the output of SMPS are something normal, I simply didn't expect 44kHz while designing the LC filter

[gkasprow]

Almost the same power supply is used on ADC board and I get the noise floor at -120dB easily.

[hartytp]

I simply didn't expect 44kHz while designing the LC filter Almost the same power supply is used on ADC board and I get the noise floor at -120dB easily.

Right! And, my guess is that once we stop killing the DACs this ripple will go away. It's almost certainly a symptom of DAC death rather than a problem in itself.

[hartytp]

@gkasprow @cjbe At this point, I think we should just stop testing this version of Zotino! Replacing those DACs is too time consuming and expensive! Whatever the exact failure mechanism is, it's most likely related to the thermal island -- and, even if it's not, I think we need to eliminate the island as a potential source of problems.

Can we just fix the known issues (see top post of this thread) and order a couple more boards?

[hartytp]

Let's do this as soon as reasonably possible, since I really want to get a version of this PCB that we can actually test.

[hartytp]

@cjbe @gkasprow One thing though:

If either of you have a chance, it would be interesting to measure the noise spectrum of the filter with the DAC removed from the board (just ground the DAC end of R24).

This at least lets us check the filter's noise. Hopefully with the DAC gone, the SMPS will also start behaving.

[gkasprow]

Tom, let me test the Kasli first. Just after that I will focus on DAC board.

[hartytp]

@gkasprow Sounds like a good plan.

[cjbe]

@gkasprow I just monitored the reference output as the Zotino overheated from cold. The reference output stayed at 5.00V even after everything has reached thermal equilibrium.

For reference, the temperature sensor was reading 27 degrees with the board just turned on from cold, and ~85-88 degrees when the board was at thermal equlibrium.

[gkasprow]

@cjbe thermal equilibrium means overheated negative regulator :) You can check value of negative rail in such condition.