3U DAC (Zotino) tests & rev 1 errata

[gkasprow]

Issues:

• [x] Use two output resistors in series/parallel to improve power handling (200mW short-circuit)
• [x] reverse C143, C145, C147, C84
• [x] mirrored SOT23 diode footprint due to Altium bug - part of component is on top layer, pads are on bottom layer.
• [ ] add Shottky diodes at the LDO outputs
• [ ] 3mm higher EMC shield on PSU
• [x] reverse IDC connector on BNC to IDC board
• [ ] Zero-index all channels
• [x] swap BNC connector GND terminals
• [ ] either add ~2mm plastic washers on BNC barrel behind front panel or move BNC connectors 2mm towards FP. This is to enable fixing the BNC connectors to the panel using the screws.
• [x] Please remove the annotation "This module connects to Kasli ..." and replace it with something like "EEM connector: IO are LVDS, I2C is 3V3 LVCMOS, P3V3_MP up to 20mA, P12V up to 1A."
• [ ] run PI and thermal simulations
• [ ] Can we run the digital interface from 2V5?
• Temperature controller issues:
  • [x] Decrease the thermal resistance between the thermal island and the main PCB/increase the thermal mass of the island by at least 1 order of magnitude -- without active stabilization, the DAC should only be 1-2C hotter than the main PCB. This should also improve the electrical ground connection between the main PCB and the thermal island.
  • [x] Remove current temperature controller and DAC over temperature shutdown circuitry.
  • [x] Add mounting holes/pads for peltier + heat sink controller by external circuit.
  • [x] Leave temperature probe on the thermal island, bring it out to an external connector, which should also provide +12V power, ground and I2C (maybe use I2C-controlled GPIO instead? This way we can do bit-banged SPI if we need to).
  • [ ] Assuming the peltier + heatsink mount on the top side of the PCB, do we want to add a stick-on insulation on the bottom side to screen the DAC from external air currents?
• [x] Silk + schematic + panel should be "v1.x" not "rev1.x"
• [x] Only have one set of +-12V regulators, not two!
• [x] Reduce SMPs outputs slightly, since we're only using +-12 regulators

Tests that can be done without an Artiq driver:

• [ ] Measure step response. This means measuring the 90%-10% rise time for a full-scale signal. If easily possible, also measure the 1% settling time.
• [ ] Measure noise PSD (0.1Hz to 10MHz) with full scale and mid scale output signal
• [ ] Glitches on LDAC, major carry, etc (trigger off the SPI)
• [ ] Confirm that temperature controller works. Measure temperature stability of DAC in lab conditions
• [ ] Measure drifts with temperature controller running. Set one channel to Vmax/0V, and look at drift over ~24 hours to confirm gain and offset drifts. Measure board temp co (e.g. use hot air gun to heat the board up while measuring output drifts).
• [ ] susceptibility to external EMI sources (i.e. neighbouring boards). I will test it with EMC equipment.

Tests that need an Artiq driver:

• [ ] Measure cross-talk (full-scale square wave at ~50kHz on one channel, another channel at 0V)
• [ ] Measure spurs due to SMPSs, digital-cross talk, pickup etc. Have one channel updating as fast as possible and measure noise on another channel
• [ ] Check for thermal issues (drifts) when outputs are slewing at max rate (e.g. testing for issues due to self-heating in the OpAmps). What’s the best way to test this?
• [ ] Measure harmonics with a full-scale sinusoid output at 50kHz.

Configuration for measurements is: DAC->3m SCSI cable->HD68 to IDC->IDC to BNC->scope/DVM/noise analyser. CMCs and a LPF on the IDC-BNC board as previously discussed. LPF chosen to avoid degrading the DAC noise or bandwidth too much.

[hartytp]

@dtcallcock good point. It also provides a good diagnostic: turn the heater on and then read out the thermal shutdown bit. If that's triggered, we know what the issue is...

[gkasprow]

I will check it.

[gkasprow]

The DAC without cooling heats up itself to 36 degrees. I took the DAC board with new chip. So I suspect that when I heat it to 50 degrees, Its power consumption increases and then it heats itself until destruction. So when it's cold in the lab it works well, when it's slightly warmer this triggers uncontrolled heating "chain reaction" that temperature controller cannot stop. So there is one solution - instead of using heating components, use little Peltier device that can do both heating and cooling and keep temperature at let's say 20 degrees. I will modify it in such way.

[gkasprow]

This is DAC temperature in self-heating mode. Sudden drop happened when I touched DAC package with my finger so it means that heat capacity of the island is very low.

[gkasprow]

Another option is to add external thermal shutdown. In this way when somebody switches the board without configuration and cooling, the DAC will sill have a chance to survive. I tried with Pelter but it works only with airflow. But with airflow it works also without Peltier :) Once it exceeds 50 degrees, airflow cooling cannot stop the heating trend so we must stabilise the temperature at much lower level.

[hartytp]

The DAC without cooling heats up itself to 36 degrees. I took the DAC board with new chip. So I suspect that when I heat it to 50 degrees, Its power consumption increases and then it heats itself until destruction.

I still don't understand this. So, without the heater, the DAC heats itself from something like 20C (room temperature) to 36C, so the temperature rise is about 15C. (Given that the DAC dissipates about 1/4W, that means that the impedance of the thermal island is about 60C/W, which is indeed very low).

From the curves in the data sheet, the DAC's power dissipation hardly changes between 20C and 80C. But, even if the DAC's power dissipation doubled, that would only add an extra 15C to the temperature, which still leaves us at something like 65C, well below the 80C that the DAC is specified to operate at. So, I can't see that the DAC's power dissipation is the cause of this, as you describe.

[hartytp]

My best guess is still that the temperature controller overshoots massively during turn-on, and that this transient is what kills the DACs -- while the DAC power dissipation doesn't really change up to 80C, I can believe it runs away beyond that.

The temperature controller can supply ?W which, given the island's tiny thermal impedance, is enough to heat the PCB up to something like 115C. This heating will occur very rapidly given how little thermal mass the island is.

Maybe the best bet is to double the size of all those heating resistors (or even a little more than double). That should keep the DAC within the safe operating point no matter what the heater does.

[hartytp]

Greg, the temperature controller is of secondary importance for this design. So, unless decreasing the heating resistor value (or lowering the loop gain to reduce overshoot) fix this problem, my vote is to leave the temperature controller disabled for the time being and move on to the other tests.

Either way, I'm satisfied that we've reached the point where you can ship two boards to us now.

Edit: we can come back to fixing the temperature controller when we look at version 1.2

[hartytp]

Some other thoughts:

1. Given how low the thermal impedance of the thermal island is, we can probably afford to connect it to the main board better (wider/shorter tabs). This will help to give the island more thermal mass, making it less sensitive to temperature fluctuations, and (by providing more ground) will also improve signal integrity.

2. We should make sure that the final temperature controller has plenty of phase margin. The loop gain will depend a bit on ambient conditions (temperature/air flow), so we need to make sure it has acceptable phase margin over a wide range of environmental conditions. This requires a lot of extra phase margin.

3. Given that the island has low thermal impedance and the temperature controller loop isn't going to be too tight (given point 2 above) the temperature of the DAC is going to be very sensitive to air currents (cf your finger measurement above). To achieve decent stability I suspect that we will definitely need to screen air currents away from the DAC. Can we add small/cheap screening cans on both sides of the board around the thermal island. These should connect to the main PCB ground.

[gkasprow]

I modified resistor values and added external shutdown. Now when it exceeds 55degrees, DAC negative rail gets disconnected and LED starts blinking. After cooling down it starts working normally. Temperature stabilisation loop works at 42 degrees. Below is log of temperature. Interesting moments: 0: power on 5.22 fans voltage reduced from 12V to 9V 10.45 - my colleague enters the room 12.50 - a person leaves the room 15:00 fans voltage changed to 12V

[gkasprow]

changes for rev2:

• add full shutdown, also for positive rail. Now positive regulator does not have SHDN pin.
• increase thermal capacity of the island. At the moment it reacts too nervous to slight temperature changes in the room
• fine tune regulator. I will do it with another rev1 board but I want to send this board to @hartytp tomorrow.

[gkasprow]

Another issue is that thermal regulator loop time constant depends strongly on airflow. So as @hartytp wrote shielding should help a little.

[gkasprow]

@hartytp 2 boards were modified - overtherm comparator was set to 55 degrees and it shuts down the negative LDO. Tomorrow morning they will be sent to UK.

[hartytp]

@hartytp 2 boards were modified - overtherm comparator was set to 55 degrees and it shuts down the negative LDO. Tomorrow morning they will be sent to UK.

Great, thanks!

[hartytp]

I still don't understand why the DACs were breaking before. Looking at figures 15 and 16 in the datasheet shows that the DAC's power consumption does not change significantly between 20C and 85C (die temperature). The DAC's die should be approx 1/4W*25C/W=6C above ambient, so if you are only heating the thermal island to 50C (approx 56C die temperature), there really shouldn't be any problem; this is well within the specified operating range of the DAC.

So, I really can't see why the temperature controller was breaking the DACs before. Are you sure that there wasn't a temporary thermal overshoot when the temperature controller was turned on? Since the thermal mass of the island is so low, the overshoot could be quite brief and easy to miss. That's really the only explanation I can see for this, since the DAC shouldn't go into thermal run away at this temperature. Or, am I missing something?

[hartytp]

Looking at the temperature plot you posted, the temperature control loop looks quite close to oscillation (lots of ringing). As I said before, the loop gain will depend on ambient temperature/air flow/etc, so we need to make sure this loop has lots of phase margin. If it's ringing now, it could easily oscillate under different conditions.

[hartytp]

increase thermal capacity of the island. At the moment it reacts too nervous to slight temperature changes in the room

This should help a bit. But, if we increase the coupling between the island and the main PCB too far, the power required by the temperature controller will become unacceptable.

Another issue is that thermal regulator loop time constant depends strongly on airflow. So as @hartytp wrote shielding should help a little.

I think this will be important. Either use cheap metal screening cans, or some foam glued onto the PCBs (whichever is easier/cheaper and sufficiently robust).

[dhslichter]

What's wrong with a Peltier and putting the setpoint at ~25 C or so? Then make a good thermal connection between the DAC and the rest of the board (to provide thermal mass), choose the DAC setpoint to be approximately ambient, and then the Peltier + heat sink allows it to servo heating and cooling to maintain.

[hartytp]

@dhslichter More hassle than it's worth IMO (e.g. needs a heat sink). Easier just to heat a little and add some insulation.

[gkasprow]

@dhslichter There is nothing wrong. It has almost advantages - much better stability of the step response vs air flow. It takes roughly 100..300mW of power to keep temperature at the level of 30deg. I tested today such solution. There are however two drawbacks:

• mechanical construction, the stack need to be installed below the shield which will act as a heatsink. So this is some additional manual work needed.
• small (<3cmx3cm) Peltiers are usually rated at 2V, so additional DC/DC converts would be necesary.

[dhslichter]

@hartytp very debatable that it's "more hassle than it's worth", given the issues we are seeing here. If we had a Peltier none of this would be occurring because we would be choosing a lower setpoint.

[hartytp]

If we had a Peltier none of this would be occurring because we would be choosing a lower setpoint.

True. So long as you have a nice way of doing the mechanics...

[gkasprow]

@hartytp I plan to order tiny Peltier and use nice chip from Maxim that integrates all functions. So with little additional PCB I can try to stabilize the DAC temperature.

[gkasprow]

Another option is this, much cheaper chip from ADI

[dhslichter]

No heat island/isolation, tie everything together with nice ground strapping. Clear space on the underside of the board for Peltier? Not sure what the clearances are there. If it has to be on the top side, you can't run under a can, but you rely on having the Peltier outside the can but close to the DAC, and you make the thermal conductivity through the ground plane layers high so that you have a good thermal link. Remember the goal should just be stabilizing temperatures against noise/drifts, probably accepting a dc temperature offset which might vary from rack to rack.

[gkasprow]

@dhslichter What about placing DAC on bottom side of the PCB, then on top goes Peltier, then the heatsink protrudes through the shielding can which can be much lower than now. I'd like to isolate the DAC from other ICs, mainly output drivers that may generate some heat when loaded. And since QFP package conducts the heat mainly through bottom thermal pad, and we want to stabilize temp of the DAC, such solution seem to be optimal. That was my intention of this island to isolate the DAC from neighbouring opamps. I did not take into account that its thermal capacity is so low and that regulator time constant varies a lot with airflow and it might be difficult to provide such phase margin that it would work reliably in all conditions. I noticed today when I was playing with the board that regulator works quite well with window open, it reaches temperature without oscillations. But when I closed the window, it started oscillating itself. The board is placed just about the window and outside is really cold.

[hartytp]

I noticed today when I was playing with the board that regulator works quite well with window open, it reaches temperature without oscillations. But when I closed the window, it started oscillating itself. The board is placed just about the window and outside is really cold.

Well, that explains the mystery of the spontaneously exploding DACs ;)

[hartytp]

through the shielding can which can be much lower than now.

Do we even need the screening can?

My vote is this: let's forget about the temperature controller for the time being and focus on testing the rest of the board. One test we plan to do anyway is the EMI sensitivity test. If we find that the DAC doesn't need screening to protect from adjacent noise sources (Kasli etc) then we can scrap this screening can, which makes the peltier mechanics a bit easier.

[gkasprow]

@hartytp the oscillations I observed today were less than 1K. Initial regulator components gave very short time constant so the heater wouldn't be able to kill DAC. It worked like hysteresis regulator. I had to extend regulator time constant 20 times to make the regulator somehow stable. Overheating could have happened if the regulator was too slow. Anyway I will investigate it with original component values. The same situation happened with 3 boards so at least it is consistent :) So first of all, what needs to be measured is step response/ bode plot of the thermal circuit in different airflow conditions. Then we can check if it is possible to choose PID values that would make such regulator stable. If not, then Peltier would be the only way to go.

[dhslichter]

One thing to think about with Peltier is that we will probably want to run in analog mode, rather than PWM mode. There is an efficiency tradeoff, but it's probably not good to have ~amp-level pulsed currents directly next to your DAC, in terms of noise pickup :)

[dhslichter]

DAC on bottom of PCB is fine with me. I don't think the DAC needs a shielding can necessarily. Also, I forget how cluttered the inner layers of the board are, but if you ran the signals on inner layers with grounds on the outside, it would provide a nice "shielding can" for pickup in the long traces between DAC and amplifiers/filters, which I think is our main potential issue in terms of pickup.

[gkasprow]

@dhslichter good idea. But anyway we need to generate low voltage, high current supply for Peltier. In my case I needed 1A and 0.2V to keep DAC at 30 degrees.

[hartytp]

Reminding myself how good the temperature stability needs to be...

• Target is 10ppm drifts max, ideally more like 3ppm
• DAC stability (gain + offset): 5ppm/K
• reference stability: +-2ppm/K max
• So we'd ideally like these components stabilised to better than +-1K (0.25K would be great). That's easily achievable in good lab conditions, but probably not in a rack with DDSs etc. So, stabilisation isn't an absolute requirement, but nice to have.

OpAmps (not actively stabilised):

• offset voltage: +-3uV/K max = +-0.15ppm/K max. For +-3ppm, need temperature stability better than +-20K. (!)
• bias current: hard to read from graphs. Very pessimistic estimate is +-5nA*11k=+-55uV=2.75ppm. That is the change between -40 and +125K, so not likely a problem. NB almost all the change is above 50C, so if we can keep to <50C this is really a non-issue.
• self heating: worst-cast power dissipation is driving 10/2=5V into the 500R=100mW. Thermal impedance is 50K/W (junction to PCB, this ignores other sources of heat dissipation), so that's 5K worst-case self-heating.
• Conclusion: the OpAmps are not going to be a source of drift! NB Ignoring temp co of output resistor since I assume the load is essentially capacitive.

[hartytp]

Anyway, pelter sounds good if not too expensive, but it'd be great to see the other tests done before we spend too much time on the temperature controller!

[dhslichter]

@hartytp agreed on this count, but we have to figure out a way to keep the DACs from cooking themselves into oblivion before we can do the tests :)

The output resistor will matter if one is making ac waveforms, though.

[gkasprow]

@dhslichter I implemented simple protection mechanism that switches the DAC rail off when too hot. One can modify other boards as well or use without heater as they are now.

[hartytp]

The output resistor will matter if one is making ac waveforms, though.

Not necessarily: for the DC one wants ppm stability, but in most cases 0.1% drifts in the AC amplitude aren't a problem. In any case, the AC drift will likely be dominated by the caps, not the Rs.

@gkasprow Given that these resistors can take as much as 10V^2/500R=200mW (DAC short circuited) is it worth paralleling two resistors for the outputs to improve thermal performance?

[gkasprow]

@hartytp sure, we can add, there is plenty of space.

[hartytp]

@cjbe @klickverbot If you have time, do you think you can help out with some of the above tests? Particularly the ones requiring Artiq, since @gkasprow doesn't have Artiq up and running atm. Anything you can do will help speed things along (I'm keen to get onto the ADC v2 design so M-Labs can finish the noise eater).

[gkasprow]

Today I repaired and modified the DAC board I have. I left original regulator compensation components to see what could be initial cause of DAC failure. Only temperature set point is different : 40 degrees instead of 50 degrees. Below is temperature log. After 3 mins I switched on the fans. So now I can play with optimisation before Urukul arrives. Technosystem promised they will be ready on Friday.

[jordens]

Then I take it that Kasli has been handed of to Technosystem as well? IMHO doing the redesign for Novo with the new ADC would fit the timeline nicely right now. No?

[gkasprow]

Yes, Kasli is in production right now.

[hartytp]

IMHO doing the redesign for Novo with the new ADC would fit the timeline nicely right now. No?

If there is time to make significant progress on Novo v2.0 before Friday then I'd agree: that should take priority over further testing of Zotino. Once that is done, we have a prototype of all EEMs currently planned, so users can start testing and M-Labs can write software.

[hartytp]

Yes, Kasli is in production right now.

Fantastic!

[jordens]

Great. @marmeladapk is Kasli/v1.0rc3 == Kasli/v1.0? Could you do the release process as described? @gkasprow Do you need input on Novo v2.0?

[gkasprow]

I played a little with RC filter values. Now the components are: Rint = 200k, Cint = 47n II (10uF+10k)

Initial transients were caused by door opening. Then all people left the lab. It keeps working, I measure the temperature with bench multimeter. The scale is 10mV/K. Anyway, I'd love to put there small CPU as a temperature controller. Then PID coefficient could be updated via SPI or it could select them depending on the airflow. The algorithm could be trivial - just measure the time constant of the heating circuit and select one or another set of coefficients. The only drawback is that it needs programming and code to maintain but I can take care of it.

[hartytp]

@gkasprow I really worry about making this temperature stabilization circuit overly complex, as the stability of the DAC is almost good enough to get away without the stabilization circuit anyway.

Let's put this issue aside for now and come back to it once we've finished testing the rest of the DAC's functionality (and designing the v2 ADC).

[hartytp]

@gkasprow Here are my thoughts about the temperature controller:

• It was a nice idea to have a really simple fixed-gain temperature controller like the one we currently have. However, as we've seen, this is unlikely to work given the potential range of ambient temperature/air flow. As a result, v1.2 will not feature this kind of temperature controller. As a result, we should not spend more time tuning this temperature controller.

• The current design for the thermal island creates problems, since the thermal mass is so low it make the DAC much more sensitive to ambient temperature fluctuations. As a result, v1.2 should either feature no thermal island (DAC directly on the main PCB) or a redesigned island with more thermal conductivity to the main PCB.

• To meet our stability target (+-10ppm) the board must be stable to +-2K.

• For v1.2 we have two options. Either scrap the thermal island + temperature controller and leave it up to the users to ensure that the board temperature is sufficiently stable for their needs. Or, add a more fancy temperature controller, probably featuring a microprocessor and a peltier. If we go for the second option, I would prefer the temperature controller to be on a separate PCB that mounts on top of the DAC/thermal island. That way users who don't care about stability don't have to mount it. In this way, we can first design the DAC board without the temperature controller board, and design the temperature controller board later on when needed.

• Edit: if we do want a microprocessor temperature controller, we could consider one of these Thorlabs ICs...: https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=10626

• In any case, since the temperature controller will need redesigning from scratch, there is no point doing any more tests on it in this version of the design.

[hartytp]

@gkasprow A couple of questions about power supplies/sequencing on the DAC:

• Why are there two separate +-12V supplies? Can't the DAC use the sample supplies as the OpAmps?
• Is it a problem if the DAC supplies turns on (off) before (after) the OpAmp supply? Would it make sense to put diodes between these supplies to ensure that the DAC can't drive the OpAmps beyond their supply rails (e.g. diode between DAC +12V and OpAmp +12V)?

[gkasprow]

@hartytp well, it is purely historical. Initial idea assumed different supply levels for DAC and amplifiers.Then we decided to have 10V range instead of 12 due to the price of precise resistors so I only modified the LDO resistors. One can also make DC/DC voltage lower 13V would be enough. Diodes are fine.