[RFC] New AMC: Shuttler (high-speed multi-channel DAC)

[marmeladapk]

From @hartytp on 2017-08-16 05:24

I've started sketching out requirements for a fast DAC to be used for ion shuttling/splitting. It's on the Wiki.

All comments welcome...

[marmeladapk]

From @gkasprow on 2017-08-16 17:43

We are able to drive 32 channels x 10MS/s from 4 EEMs. The question is if Kasli is able to refresh such amount of channels simultaneously. And to build 320 channel system we would need 5 Kaslis.

15.08.2017 11:24 PM "hartytp" notifications@github.com napisał(a):

I've started sketching out requirements for a fast DAC to be used for ion shuttling/splitting. It's on the Wiki https://github.com/m-labs/sinara/wiki/Fast_DAC.

All comments welcome...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/m-labs/sinara/issues/253, or mute the thread https://github.com/notifications/unsubscribe-auth/AEH-vqvfLzFd5M4rTLLnqwHQOHib0maFks5sYn0FgaJpZM4O4b0u .

[marmeladapk]

From @gkasprow on 2017-08-16 18:11

On AMC we can fit 64 or even 128 DAC channels with stacked SCSI connectors.

[marmeladapk]

From @dhslichter on 2017-08-16 18:29

@gkasprow is that 32 channels, EACH at 10 MS/sec? I think that probably serial control is going to be challenging to meet the kind of update rates we want.

The DAC used on the PDQ boards is the Analog Devices AD9726, It has 0.5 LSB typical DNL, 10 ppm/C gain and offset drift, and takes parallel LVDS data at up to 400 MSPS. Noise is at the Johnson noise limit for -6 dBFS output, runs up about 5 dB at 0dBFS. Dissipation is about 0.6 W/channel (!!!) In the PDQ, it is run at 50 or 100 MSPS. AFAICT most of the other DACs available with sufficient speed to drive > MHz signals have at least 2 LSB typical DNL, and 100 ppm/C gain coefficients, both of which are problematic. The downside of the AD9726 is that it is a "not recommended for new designs" product, i.e. they are phasing out the lifecycle of the chip.

The LTC1668 is a potential alternative part, with specs somewhat worse than the AD9726 but better than a lot of the competition. The noise is a factor of a few above the Johnson noise limit (50 pA/rtHz, or 5 nV/rtHz into 100 ohms), but it goes to 50 MSPS and the power dissipation is much lower (180 mW/channel).

Another chip worth looking at is the LTC-2000-16. This is more expensive, about $90/channel, and has export restrictions which could be problematic, although I think Poland is OK (China/Hong Kong is definitely not OK). Power dissipation at 100 MSPS would be ~300 mW, DNL is 0.5 LSB, gain drift is 5 ppm/C. It's in a BGA package, which is a pain.

[marmeladapk]

From @jordens on 2017-08-16 18:35

If we start with selecting the DAC, then we can match N channels to an FPGA which should probably be on the same board as the DACs for wiring and throughput reasons. The interface between that board and the next upstream device (Kasli or Metlino) is likely to be higher level (DRTIO) and not EEM for throughput and simplicity reasons. And then we arrive at AMCs as the form factor.

[marmeladapk]

From @dhslichter on 2017-08-16 19:54

I agree with @jordens here in general, although if price point is highly sensitive on this one we could basically implement "Kasli PDQ" -- effectively N DACs (AD9726 perhaps) linked to a medium-sized Artix on an EEM. If much of the PDQ gateware could be ported over, including the SPI communications over EEM, this would be a simple board to implement that would also be inexpensive, and require only the inexpensive chassis rather than the more pricey uTCA. The price you pay is obviously that you would have less bandwidth to update the output waveform of the DACs, feedback becomes slower, etc.

NIST would be very interested in an EEM-form-factor PDQ replacement card with similar spec/properties (although with much deeper memory, and possibly higher sample rates).

[marmeladapk]

From @jordens on 2017-08-16 19:59

@dhslichter ACK. It is just that restricting the link to SPI (or anything we can run over EEM) would likely lead to the same split-brain effect we see with PDQ (or any other non-DRTIO device with significant data): data here, control there and data needs to be uploaded ahead of time, synchronized, maintained and can not be swapped or parametrized easily.

[marmeladapk]

From @hartytp on 2017-08-16 20:16

@dhslichter Thanks for the DAC recommendations.

PDQ, it is run at 50 or 100 MSPS

Do we really need this much bandwidth? IIRC, if we go for 10MSPS, we get more choices of DAC (and a less crazy power budget!).

I agree with @jordens here in general, although if price point is highly sensitive on this one we could basically implement "Kasli PDQ" -- effectively N DACs (AD9726 perhaps) linked to a medium-sized Artix on an EEM.

If you're serious about going to 200+ channels, then I really think AMCs are the only way to go. Not just because of the wiring complexity/space required for large numbers of Kasli/EEMs, but also because you need a chassis with really good thermal management (otherwise, thermal drifts will be a killer due to the amount of power dissipation on these boards). At that point, you need something like a proper uTCA rack.

For a cheaper approach, you an always buy a single unit AMC chassis and run the AMC over DRTIO fibre.

[marmeladapk]

From @gkasprow on 2017-08-16 20:24

I've been looking at lower resolution DACs. Did we discuss that in such application 12bits is sufficient?. Then we could make very low cost EEM with such 32 DACs with constant update rate of 10MS/s. And MTCA with high performance DACs.

[marmeladapk]

From @gkasprow on 2017-08-16 20:32

I meant something like dac7822.

[marmeladapk]

From @hartytp on 2017-08-16 20:32

Did we discuss that in such application 12bits is sufficient?

12 bit would not be sufficient for us.

Then we could make very low cost EEM with such 32 DACs with constant update rate of 10MS/s. And MTCA with high performance DACs.

Given the reasons discussed above, I don't think we'd use the EEM DAC board.

@gkasprow Will the EEM + Kasli alternative really be cheaper than the AMC? My guess is that in the end, the AMC solution will be cheaper and more convenient.

[marmeladapk]

From @hartytp on 2017-08-16 20:36

Big questions for me about this design:

• What analog bandwidth/sample rate do we really need? Let's try not to over specify this.
• Are we happy separating the AFE from the DAC board? Downside is power dissipation/space taken up near the trap. Upside is that we won't run into issues (self-heating etc) due to trying to drive 10V into 200pF of cabling capacitance at 1MHz.
• What pass-band and stop-band frequencies do we need for the AFE, and what stop-band noise do we need? This determines the order of the filter on the AFE (and hence the power consumption).

[marmeladapk]

From @gkasprow on 2017-08-16 20:41

If we pack 64x 90$ DACs then platform cost does not matter:)

[marmeladapk]

From @hartytp on 2017-08-16 20:43

Exactly.

[marmeladapk]

From @jordens on 2017-08-16 20:48

Hey @camacazio could you explain to us why you went for 50 MHz with PDQ? Would 10 MHz work as well?

[marmeladapk]

From @jordens on 2017-08-16 20:51

• I really like the detached frontend. It also helps people that have just +-1V voltages or +-50V.
• IMHO 10 MHz would be fine as well but probably actual users should determine the minimum from their physics and filter electronics.

[marmeladapk]

From @dhslichter on 2017-08-16 20:51

I would argue we want to be able to synthesize signals with bandwidths up to typical secular frequencies, if we are looking to do fast (diabatic) transport. I would say that 10 MSPS is the very low end of what one would want,

I'm fine with AMC in the end but I think we should try to pack as many DACs in as possible in this case.

[marmeladapk]

From @dhslichter on 2017-08-16 20:53

Detachable AFE is definitely a good thing, and will allow people to customize a great deal. I would keep it inside the chassis. I think given that customizability, we want the underlying DAC board not to be underpowered sample-rate-wise, especially if we are going to be in an expensive chassis.

[marmeladapk]

From @hartytp on 2017-08-16 20:58

I'm fine with AMC in the end but I think we should try to pack as many DACs in as possible in this case.

I'm happy shooting for 64 channels per AMC.

The only downside of packing the channels in is that the silicon cost is quite high, so users who only want, say, 10 channels are wasting a lot of money on unused DACs.

IMHO 10 MHz would be fine as well but probably actual users should determine the minimum from their physics and filter electronics.

Agreed, we shouldn't go below this. And, if we can find a good DAC that's much faster, and doesn't have much worse noise/etc then it'd be great to go faster. But, let's take this as a lower limit on the BW for now.

Detachable AFE is definitely a good thing, and will allow people to customize a great deal.

Let's copy Zotino and supply +-12V + GND on the HD68 cable. That way, the front-end can be powered via the SCSI cable assuming it doesn't need too much current.

[marmeladapk]

From @dtcallcock on 2017-08-16 21:00

Would 10 MHz work as well?

If the DAC update rate might be resonant with any ion secular frequencies you have to think a lot harder about filtering and/or frequency planning. In Be+ and Mg+ surface traps you are often right around 10MHz so I think at least 20MSPS would be preferable so that this isn't an issue.

Also to echo Daniel's comment, there is interest in pushing transport further into the diabatic regime at NIST. We'd have to have to ask the experts (eg. Didi) to see what their aspirations are.

drive 10V

The Sandia HOA 2.0 can take +/- 20V so I think this would need to be an option. I guess AFE spec is a somewhat separate discussion though.

[marmeladapk]

From @gkasprow on 2017-08-16 21:01

We could follow rtm approach and make mezzanines:) i.e. fmc ones or smaller ones with i.e. 8 dac channels...

[marmeladapk]

From @gkasprow on 2017-08-16 21:04

We can also make just 32channel fmc boards and plug them into std amc carrier i.e. AFC(K) OH design which has HPC FMC

[marmeladapk]

From @hartytp on 2017-08-16 21:09

We can also make just 32channel fmc boards and plug them into std amc carrier i.e. AFC(K) OH design which has HPC FMC.

That could be a nice solution if you can fit that many DACs on an FMC, as well as the required regulators and FPGAs (probably not enough IO on an FMC to run these DACs directly from the connector)! Assuming you can get 2 FMCs on an AMC, you could then have up to 64 channels per AMC.

---

Another thought: we should make sure that this board has front-panel clock, and trigger input SMAs. That would extend the range of uses (e.g. for non-ARTIQ users)

[marmeladapk]

From @gkasprow on 2017-08-16 21:15

I did such 16 channel OH boards nased on dac1401d125

16.08.2017 15:04 "Grzegorz Kasprowicz" kasprowg@gmail.com napisał(a):

We can also make just 32channel fmc boards and plug them into std amc carrier i.e. AFC(K) OH design which has HPC FMC

[marmeladapk]

From @hartytp on 2017-08-16 21:16

Also to echo Daniel's comment, there is interest in pushing transport further into the diabatic regime at NIST. We'd have to have to ask the experts (eg. Didi) to see what their aspirations are.

Good idea!

The Sandia HOA 2.0 can take +/- 20V so I think this would need to be an option. I guess AFE spec is a somewhat separate discussion though.

Okay, let's see if we can get the AFE to +-20V.

I did such 16 channel OH boards nased on dac1401d125

Were there FPGAs on the mezzanine? How did you do the IO for the DACs?

[marmeladapk]

From @gkasprow on 2017-08-16 21:23

HPC FMC has really lot of pins. I used all pins of biggest ARTIX chip

16.08.2017 15:16 "hartytp" notifications@github.com napisał(a):

Also to echo Daniel's comment, there is interest in pushing transport further into the diabatic regime at NIST. We'd have to have to ask the experts (eg. Didi) to see what their aspirations are.

Good idea!

The Sandia HOA 2.0 can take +/- 20V so I think this would need to be an option. I guess AFE spec is a somewhat separate discussion though.

Okay, let's see if we can get the AFE to +-20V.

I did such 16 channel OH boards nased on dac1401d125

Were there FPGAs on the mezzanine? How did you do the IO for the DACs?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/m-labs/sinara/issues/253#issuecomment-322901861, or mute the thread https://github.com/notifications/unsubscribe-auth/AEH-vg2_dMV4F8g4QoGisJoctDlAsgaUks5sY1wpgaJpZM4O4b0u .

[marmeladapk]

From @sbourdeauducq on 2017-08-17 02:57

This shouldn't be called "fast DAC", it's slower than Sayma. Maybe "medium speed DAC" or something along those lines.

[marmeladapk]

From @jordens on 2017-08-17 05:34

If we put that on FMCs (and 16 channels per FMC with two FMCs per AMC would already be nice IMHO) then we can also do an EEM with a (little) FPGA being an FMC carrier, i.e. have the choice of having the FMC on AMC or on EEM.

And let's just use a code name instead of "something DAC".

[marmeladapk]

From @hartytp on 2017-08-17 13:53

@jordens Agreed: if we can get this on an FMC, that's great. As you say, if anyone really wants to use it with Kasli, we can then make an adapter board. This could also be a generic (passive) FMC to many IDC board. One option would be to give this female header, so that it can mate directly with Kasli, mounting via spacers and the mount holes.

---

Having said that, maybe the best approach is to put the discussion about form-factor on hold for now.

Let's first figure out how much bandwidth people need, and what order of filter the AFE needs. Once we've done that and chosen DACs/OpAmps, we'll be in a better position to figure out whether it can fit on an FMC/etc.

[marmeladapk]

From @gkasprow on 2017-08-17 17:22

To fit 32 channels on FMC we would need to go for quad dacs with 16 bit lvds port and max frequency twice higher than we need. In this way we would connect 2 chips in parallel to each port and use 4 x 16 channels ports of HPC FMC with control lines.Example is DAC3484. It would be hard to physically fit and route 16 dual channel dacs on FMC

17.08.2017 07:53 "hartytp" notifications@github.com napisał(a):

@jordens https://github.com/jordens Agreed: if we can get this on an FMC, that's great. As you say, if anyone really wants to use it with Kasli, we can then make an adapter board. This could also be a generic (passive) FMC to many IDC board. One option would be to give this female header, so that it can mate directly with Kasli, mounting via spacers and the mount holes.

Having said that, maybe the best approach is to put the discussion about form-factor on hold for now.

Let's first figure out how much bandwidth people need, and what order of filter the AFE needs. Once we've done that and chosen DACs/OpAmps, we'll be in a better position to figure out whether it can fit on an FMC/etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/m-labs/sinara/issues/253#issuecomment-323080233, or mute the thread https://github.com/notifications/unsubscribe-auth/AEH-vnpRgSPIQVAXdF2ieZgV4jz2JV76ks5sZEXZgaJpZM4O4b0u .

[marmeladapk]

From @gkasprow on 2017-08-17 20:44

@hartytp Look at this MAX5898 chip. It has DNL=1 and INL=3 and 2 channels, but it has 110ppm so would need thermal management or compensation This MAX5895 is similar. The MAX5888 has 50ppm with ext reference but only 1 channel.

[marmeladapk]

From @jordens on 2017-08-17 21:03

On the sample rate, it does look like non-adiabatic transport tends to interact with the transverse modes as well. Now it depends strongly on how steep your filters are and whether they are also excellent anti-aliasing filters for the DAC. I'd be worried about the stuff in the second Nyquist zone colliding with the transverse modes, i.e. up the data rate or better just use e.g. 4x DAC interpolation and 10 Mhz data rate. MUXing multiple DACs over one LVDS and time-sharing gateware would be useful as well.

[marmeladapk]

From @hartytp on 2017-08-17 22:05

Let's put together the DAC options (feel free to add more)...

Sorting by: sample rate >=10MSPS, DNL <=1LSB, 16-bit

PN	fs (MSPS)	channels	power (W)	DNL (LSB)	output (mA)	NSD	temp co (ppm/K)	interp	package dims	comments	cost ($)/CH	IFC
AD9726	400	1	0.6	+-0.5	20	45pA/rtHz (-173 dBFS/Hz)	+-10	no	14x14 mm, leaded	end-of-life (not recommended for new designs)	58	LVDS16
LTC1668	50	1	0.2	+-1	10	50pA/rtHz (-166 dBFS/Hz)	+-30	no	8 x 10 mm, SSOP		15	CMOS16
LTC-2000-16	2500	1	2@2G, 0.7@500M, 0.4@250M	+-0.5	20	-165dBFS/Hz@65M	+-5	no	9 x 15 mm, BGA		96	LVDS16, LVDS32
MAX5895	500	2	0.5@100M, 0.83@250M	+-1	20	-157dBFS/Hz@16M	+-110 incl +-50 of ref	yes	10x10 mm, 68 QFN		13	CMOS16
MAX5898	500	2	0.5@100M, 0.83@250M	+-1	20	-157dBFS/Hz@16M	+-110 incl +-50 of ref	yes	10x10 mm, 68 QFN		13	LVDS16
MAX5888	500	1	0.13	+-1.3	20	-165dBFS/Hz@16M	+-50	no	10x10 mm, 68 QFN		45	LVDS16
DAC3484	1250	4	1.27	+-2	20	-160dBFS/Hz@10M	+-15	yes	12x12 mm BGA, 9x9 mm double-pitch QFN		18	LVDS16

[marmeladapk]

From @gkasprow on 2017-08-17 22:13

We can go down to about 50..60ppm with external ref. And this MAX589x is low cost with channel price ~15$. We could fit 16 such chips on fmc providing that we reduce their power. At 100MHz they consume about 0.5W of power so for FMC would be still within the limit but would require intensive cooling.

[marmeladapk]

From @klickverbot on 2017-08-17 22:37

The first generation of transport DACs in Zurich used the MAX5898, so there is quite a bit of experience with that. As Greg points out, drifts are definitely an issue (although we didn't stabilize the temperature at all).

[marmeladapk]

From @jordens on 2017-08-18 04:34

@klickverbot What do the other generations use and why? ;)

I'd be disappointed if we end up going for single channel chips when building a multichannel dac.

[marmeladapk]

From @hartytp on 2017-08-18 05:18

@gkasprow If you have time, could you help me check the table above, and add any other DACs you find that could be suitable.

AFAICT, the best choice in terms of performance would be LTC-2000-16. I don't see any export issues when looking at Farnell... Downside is that it's quite expensive and only single channel. Plus a lot of power when run at high clock frequencies...

[marmeladapk]

From @gkasprow on 2017-08-18 05:25

@hartytp I just was filing the table and got info that someone is editing it. I'll do it.

[marmeladapk]

From @klickverbot on 2017-08-18 06:42

@jordens: I left the group since, so... As far as I am aware, the second version (more like 1.1) just used an external reference with the same DACs, though. Did you talk to Vlad?

[marmeladapk]

From @hartytp on 2017-08-18 15:08

@gkasprow Thanks for doing that!

To check: is that the complete list of potential DACs satisfying the filter criteria (<=1LSB DNL, >=10MSPS)? e.g. is that all the offerings from ADI, LTC, TI, MAX, etc.?

Could you convert all noise units to dBFS to make the comparison easier?

[marmeladapk]

From @hartytp on 2017-08-18 15:13

Oh, and given that this is fast and low noise, I suggest "Holding" as the name...

[marmeladapk]

From @gkasprow on 2017-08-18 15:41

Indeed the MAX spec says that with external ref the drift is reduced to roughly 50..60ppm

[marmeladapk]

From @klickverbot on 2017-08-18 15:42

I don't think they are quite happy with the performance yet, though. (The external reference was just a quick fix rather than part of a bigger redesign.)

[marmeladapk]

From @dhslichter on 2017-08-18 16:05

LTC-2000-16 would be a very nice DAC to use because one could in principle generate RF/microwave tones with it as well as transport/shuttling tones. Obviously it is more expensive and consumes more power if we run to higher clock speeds. I think passive stability (relying less on temperature control) is definitely a big plus.

[marmeladapk]

From @dhslichter on 2017-08-18 16:11

If this is going in a uTCA rack, I think we should consider RTM instead of FMC. We'll have much more real estate for DACs, which allows for more channels. We might consider making an AMC card which is very simple, and basically just passes backplane connections through the RTM connector (no FPGA needed on AMC, very stripped down). We also already have high quality clock distribution on the RF backplane, which we can use to generate the DAC clocks.

[marmeladapk]

From @gkasprow on 2017-08-18 16:13

@hartytp I checked much more vendors like NXP, IDT, Intersil and so far found only these.

[marmeladapk]

From @dhslichter on 2017-08-18 16:16

RTM would also allow us to more easily include integrated AFE daughterboards for user customization; with FMC we would either need to make a standalone box for AFE (potentially problematic for various reasons), or else make different FMC variants with different integrated front ends.

[marmeladapk]

From @hartytp on 2017-08-18 16:22

Thanks Greg!

Since we've already got RTMs for Sayma, I'm happy putting this on an RTM if that helps...

[marmeladapk]

From @hartytp on 2017-08-18 16:28

Narrowing tings down a little:

eliminating the DAC3484 because the DNL is too high; it's basically a 14-bit DAC.

eliminating the AD9726 as it's end-of-life and will become hard to source at some point in the not so distant future. We could, in principle, buy a stock now to keep us going for a while, but I'd rather focus on a good longer term solution.

DHS -- adding package dimensions

PN	fs (MSPS)	channels	power (W)	DNL (LSB)	output (mA)	NSD	temp co (ppm/K)	interp	package dims	comments	cost ($)/CH	IFC
LTC1668	50	1	0.2	+-1	10	50pA/rtHz (-166 dBFS/Hz)	+-30	no	8 x 10 mm, SSOP		15	CMOS16
LTC-2000-16	2500	1	2@2G, 0.7@500M, 0.4@250M	+-0.5	20	-165dBFS/Hz@65M	+-5	no	9 x 15 mm, BGA		96	LVDS16, LVDS32
MAX5895	500	2	0.5@100M, 0.83@250M	+-1	20	-157dBFS/Hz@16M	+-110 incl +-50 of ref	yes	10x10 mm, 68 QFN		13	CMOS16
MAX5898	500	2	0.5@100M, 0.83@250M	+-1	20	-157dBFS/Hz@16M	+-110 incl +-50 of ref	yes	10x10 mm, 68 QFN		13	LVDS16
MAX5888	500	1	0.13	+-1.3	20	-165dBFS/Hz@16M	+-50	no	10x10 mm, 68 QFN		45	LVDS16

[marmeladapk]

From @hartytp on 2017-08-18 16:36

I'd basically be happy with any of these options.

Having said that, I'm tempted to go for the LTC-2000-16, in particular because of the low temp co -- this board is going to run hot, so thermal issues will likely be a killer for a DAC with 30-50ppm/K. Sillicon cost is roughly 2kGBP for a 32ch board, which is high but acceptable IMO.

32 channels on an RTM?

Anyway, let's give the prospective users a chance to vote... e.g. I'd be interested to hear from the likes of @klickverbot @cjbe @jbqubit Vlad, Christian Ospelkaus, etc

[marmeladapk]

From @gkasprow on 2017-08-18 16:40

For large number of channels we will need high pin count fpga on board. It might be easier to modify sayma amc. If we get rid of one ddr controller and fmc we then can put such amount of dacs instead. And we keep rtm, sfps and ability to control rtm boards because we have no other use for gtx channels. If we want to use rtm for dacs and AFE modules then we need either jesd204b or to upgrade artix fpga to highest pin count one. On rtm we already have decent clock distribution so this can be good option. With 200$ artix chip you get 16 gtp channels and enough ios to control 64 DACs