Kasli clocking

[jordens]

• [x] Is the LTC6957 needed? IMHO we can skip it.
• [x] The SMA assignment is not the way we'd want it. I'd like to be able to get the recovered DRTIO clock out of Kasli, then feed it through some external PLL/cleaner, and then input it into Kasli for distribution to the EEM fanout again. For SMA1, there would be two use cases:
  • [x] (master mode): SMA1: DRTIO ref clock input
  • [x] (slave mode): SMA1: recovered DRTIO clock
• [x] SMA2 should always be the input to the fan-out to the EEMs.
• [x] Only if it is really easy, provide a way to feed DRTIO clock (same as the one going to the FPGA logic and transcievers) to the EEM-fanout.
• [x] Also feel free to use a bulkhead SMA connector and then a cable (or pigtail) for SMA2 if that helps with front panel space.
• [x] IC3 IN_SEL needs a jumper to select the input.
• [x] Remove R30 and R33 as IC3 (ADCLK948BCPZ) has internal termination.
• [x] No need for TR1 and TR2. Drive IC2 (Si5324C-C-GM) and IC3 (ADCLK948BCPZ) single ended. Reduce the number of MMCX connectors to 7. Drive the backplane (TR3) directly from the freed-up diff output. Saves one BOM line. @hartytp OK? @marmeladapk We should probably actually remove these components rather than just marking them as DNP to save board space.
• [ ] Please double check CK_FPGAIO_P and _N bias. Really OK to just bias _P?
• [x] Make all biasing resistors for negative outputs on IC3 DNP by default (apart from the diff pair going to the BP). CLK_CDR_SMA_OUT_N biasing resistor (R11) should also be DNP.

[hartytp]

@jordens What max EEM clock do you want? 500MHz? 1GHz? Faster?

[jordens]

1 GHz.

[hartytp]

@jordens How to you want to do the EEM fan out? Have 8 (4?) RF connectors on Kasli?

[jordens]

Yes. If that fits. But I am worried about space. Maybe it is best to just have a 3U card that is the fanout... That frees space on Kasli, could feed 12 or more EEM and decouples problems/cost by making that clock distribution optional. @gkasprow ?

[gkasprow]

@hartytp We plan to have several clock outputs from fanout chip. They are distributed in parallel with IDCs using MMCX cables.

[hartytp]

@jordens @gkasprow Either approach works for me. If there is room, I'd prefer having the MMCX connectors directly on Kasli, as it reduces the number of 3U boards one has to buy and fit into a rack.

[gkasprow]

@hartytp @jordens will do our best to fit all stuff on signgle board.

[hartytp]

Only if it is really easy, provide a way to feed DRTIO clock (same as the one going to the FPGA logic and transcievers) to the EEM-fanout.

I'd be keen on this if it can be done in a convenient manner: IMO the most common EEM clock will be the DRTIO clock, whether Kasli is operated in master or slave mode. Given that, it makes sense to have an easy way of routing it to the EEM fanout buffer without having to externally jumper the 2 SMAs together.

[hartytp]

Otherwise, all looks good to me.

[gkasprow]

@hartytp @jordens @jbqubit It seems we will manage to fit clock distribution to Kasli board. Clocks will be routed using right angled MMCX connectors The clock distribution became more complex but this should make everybody happy

[hartytp]

Greg, thanks for posting that! That all looks good to me.

A few minor points about the clocking:

• Can we replace the LVPECL clock buffers with LVCMOS equivalents (must be good to 1GHz)? This should shave a bit off the cost/power budget.
• The current arrangement of MMCX connectors looks very tight. I'd be happy reducing the number to, say, 6 if that allowed them to be spread out a bit/placed in a more convenient location. That way, we could also drive the backplane connection directly from a differential output (no balun).
• Are you using a standard 100mil jumper to switch the SMA between input and output? Is that okay at 1GHz? Would it be better to use either a solder jumper or (my preference if conveniently possible) a bi-directional LVCMOS clock buffer for this?
• Can we scrap the baluns (drive the inputs single-ended, terminating the unused inputs with a capacitor to ground)? I don't think they're required for this design.
• Can we ac couple all clock inputs, please? With the current baluns, the clock inputs are shorts to Kasli ground at DC, and I've seen some nasty noise/grounding issues in the past when doing this.
• Should IC3 have a jumper/piano switch connected to its input select pin?

[marmeladapk]

Is that okay at 1GHz?

Wasn't SMA1 clock supposed to be at 100MHz?

The current arrangement of MMCX connectors looks very tight.

We'll move them up, above Artix and space them out.

[hartytp]

Wasn't SMA1 clock supposed to be at 100MHz?

Sorry, my mistake, SMA1 is at 125MHz, not 1GHz. IIUC, it might move to ~250MHz in the future if we increase the RTIO_COARSE frequency to allow for a faster CPU.

Are you happy with the jumper at 125MHz?

[sbourdeauducq]

RTIO and CPU frequencies are not linked, it's only accidental that they are both 125MHz in current systems (but typically derived from different oscillators).

[hartytp]

@sbourdeauducq thanks for clarifying.

In any case, it still seems that this hardware should support RTIO_CLK to at least 200MHz. For example to enable the 1.6GHz DAC_CLK, 800MHz f_data configuration for Sayma in artiq hardware.

[gkasprow]

@hartytp Yes, we can replace the buffer with CMOS one, the question is what additive jitter do you accept? It's hard to find 1GHz capable CMOS buffer since there is no CMOS single ended standard that supports it. If we limit frequency to 200MHz then the LMK00101SQ looks good, has low jitter and it has 10 outputs so we get rid of balun. The standard jumper should work fine at 200MHz, at 1GHz we may get troubles. One can always solder 0805 resistor to the pads. Yes, we can get rid of baluns and use single ended configuration. AC coupling is also possible. Of course, IC3 must have a jumper and/or be controlled by FPGA.

[hartytp]

It's hard to find 1GHz capable CMOS buffer since there is no CMOS single ended standard that supports it.

Yes, for 1GHz the lower cost/power alternative to LVPECL is LVDS, not LVCMOS. Oops!

Yes, we can replace the buffer with CMOS one, the question is what additive jitter do you accept?

A good starting point is the phase noise floor for the Si5324 and AD9910 PLL (cf my last comment on #201). IMO, anything that won't degrade them is fine for the Kasli fanout; if users really want an ultra-low noise fanout they're probably better off using a dedicated one on a separate card.

The standard jumper should work fine at 200MHz, at 1GHz we may get troubles. One can always solder 0805 resistor to the pads.

Okay.

Of course, IC3 must have a jumper and/or be controlled by FPGA.

Jumper control is probably better IMO.

[jordens]

Looks good!

• Let's not take the SI5324 as the benchmark for that fanout's allowed noise contribution but just the AD9910 PLL. There will be use cases where the clock cleaner is not used.
• I'm also slightly worried about the jumpers. Would be happy with resistor pads. Default to using that SMA as an input (CLK_CDR_IN)
• I am also OK with cutting down the number of MMCX conns 6 or even 4 and would like some (i.e. 2) of clocks on the backplane. Those connectors also tend to be expensive.
• OK to kill the baluns if that helps with cost or board space. Would strongly prefer AC coupling as well.
• Optimizing for low power and sufficiently small noise contribution with LVCMOS/LVDS sounds fine.

[hartytp]

would like some (i.e. 2) of clocks on the backplane.

Alternatively, we could stick with a single clock pair on the bp connector and put a fanout buffer on the bp itself (it already has power, so this is trivial):

• this allows us to save a couple of pins on the 96-pin bp connector for gnd/power, while still having 1 differential clock line for each slot on the bp.
• having a dedicated clock line to each slot on the bp is nice anyway, as it means we don't need to worry about the unterminated stubs in the clocking network that arise when a single clock line is routed to multiple slots. It also saves the EEMs from requiring muxes to select between BP_CLK_0 and BP_CLK_1 (if we have two BP clock lines), as we wouldn't want two EEMs using the same clock (as they should both terminate it)

[gkasprow]

At the moment we have only one clock pair on BP. MMCX are not that expensive - below 1$$. We can always not mount all of them. It would be better to route diff clock over BP connector. At the moment we don't have any free pins on BP. Number of GNDs is barely sufficient. So I'd like to keep single clock diff line over BP. So what about LMK00101SQ and balun to make clock diff over BP pins ? It has ~30fs of jitter and works up to 200MHz

[hartytp]

At the moment we have only one clock pair on BP.

Let's stick with 1 diff clock pair to the BP, and add a fanout buffer on the BP if required.

MMCX are not that expensive - below 1$$. We can always not mount all of them.

That sounds good. So long as they're not too hard to access, the current number is fine then.

So what about LMK00101SQ and balun to make clock diff over BP pins ? It has ~30fs of jitter and works up to 200MHz

We would like to have the option of running the BP clock up to 1GHz (e.g. DDS with no PLL), so can we use a faster LVDS buffer than this? I've seen a few suitable ones around from ADI and others.

Otherwise, that all looks really good @gkasprow! Great job!

[jordens]

• ACK the one diff pair to the BP
• ACK the MMCX price/mounting option
• I'd also like to be able to push 1 GHz through the fanout. But the typical 300 fs jitter sounds prohibitive. If I have to choose between 1 GHz/300 fs and 200 MHz/30 fs, I'd take the 30 fs.

[gkasprow]

@jordens @hartytp I'm not sure if LVDS is good choice as single ended coaxial line driver. It has current output which means that don't like capacitive loads which quickly degrades clock edges and thus increases jitter. It is dedicated to drive low capacitance diff lines on PCB. For this reason I'd keep LVPECL or use version with 10 outputs (ADCLK950BCPZ) so could get rid of balun. Example 1G capable LVDS driver - LMK01010 costs 12$ - more than ADCLK948BCPZ (9$). ADCLK950BCPZ costs 11$. LMK01010 LVDS version cosumes 160mA, LVPECL (LMK01020) - 340mA so there is 0.5W more power in case LVPECL.

[hartytp]

But the typical 300 fs jitter sounds prohibitive. If I have to choose between 1 GHz/300 fs and 200 MHz/30 fs, I'd take the 30 fs.

Jitter doesn't need to be a problem for LVDS muxes/buffers. See e.g. ADCLK854 fig 12. This is better than the AD9910 PLL noise, so not a problem. Cost is $6USD in quantity from Digi-Key.

I'm not sure if LVDS is good choice as single ended coaxial line driver. It has current output which means that don't like capacitive loads which quickly degrades clock edges and thus increases jitter. It is dedicated to drive low capacitance diff lines on PCB. For this reason I'd keep LVPECL. Example 1G capable LVDS driver - LMK01010 costs 12$ - more than ADCLK948BCPZ (9$). LVDS version cosumes 160mA, LVPECL (LMK01020) - 340mA so there is 0.5W more power in case LVPECL.

@gkasprow Is this still a problem when launching into properly terminated 50Ohm coax (i.e. not a capacitive load)? Or, is the issue that you'd need a balun on each clock output to do this properly?

I don't feel strongly about the LVDS v LVPECL issue, so I'm happy for you to go with whichever you think will be best.

[gkasprow]

@hartytp I don't have strong arguments that LVDS would have issues with coax cables. It is more intuition than knowledge. Generally people use LVPECL for low jitter clock distribution with coax cables. What we can do is to use LMK01000 series chip that in same package has LVDS and LVPECL option. So let's use LMK01010 with LVDS and if we experience jitter issues, will switch to LMK01020 with LVPECL outputs and solder termination resistors.

[gkasprow]

In case of input baluns, we can mark them DNP and install coupling capacitors below them. Then in case of jitter issues we can still mount them.

[jordens]

@gkasprow , @hartytp all ACK.

[hartytp]

I don't have strong arguments that LVDS would have issues with coax cables. It is more intuition than knowledge. Generally people use LVPECL for low jitter clock distribution with coax cables.

Good to know! I haven't got a huge amount of experience with this kind of thing, so I'd tend to trust your intuitions.

So let's use LMK01010 with LVDS and if we experience jitter issues, will switch to LMK01020 with LVPECL outputs and solder termination resistors.

FWIW, another option which could be nice, is the HMC6832:

• 2:8 mux/fanout
• more expensive than LMK01020 at $16.53 from Digi-Key in singles, $11.4 in 500
• TTL input (e.g. resistor strapping) allows switching between LVPECL and LVDS, so we don't need to swap components to choose.
• 3.5GHz BW, so it could be useful if we ever use an AD9914 DDS
• ultra-low phase noise (not that it's at all required for this design!)
• For 3V3 supply, power consumption at full-load: 1W for LVPECL, 400mW for LVDS

The LMK01010/LMK01020 is almost certainly fine. Comments:

• I didn't see phase noise plots on the data sheet, so it's hard to evaluate it's performance (e.g. can't directly compare to the AD9910 PLL noise)
• It has quite a lot of unnecessary features like delays and dividers. I haven't looked into it in detail (e.g. is the default configuration okay for us?), but I wouldn't want to introduce another chip that requires programming/configuration unless we have to.

Given your concerns with LVDS as a coax driver, I'd be happy sticking with the current ADCLK948 -- maybe it's overkill for our application, but at least we have a high confidence that it will work in the first revision. Do we really want to get into testing the phase noise of clock buffers to try to save a Watt on Kasli? (I'd also probably prefer ADCLK948 without baluns to LVDS + baluns).

In case of input baluns, we can mark them DNP and install coupling capacitors below them. Then in case of jitter issues we can still mount them.

I'd prefer to avoid the baluns if possible, as they'll take up quite a lot of space and inflate the BOM. Based on our past experience, my guess is that they're not needed. We could add a balun on only 1 channel for now, so we can test the phase noise with and without before making a final decision if you think it's a good idea?

[hartytp]

Another benefit of sticking with the ADCLK948 is that we're already using it elsewhere (Sayma/Baikal/etc), so it's already a "known quantity". It's always good to minimise the number of IC datasheets one has to read to understand the system...

[gkasprow]

@hartytp I meant baluns on SMA clock inputs. If you are sure that we don't need it, that's OK. So with AD948 we would use 7 x MMCX + one output to the backplane. We can try not to load unused LVPECL outputs to save power.

[hartytp]

I meant baluns on SMA clock inputs. If you are sure that we don't need it, that's OK.

Baluns shouldn't be needed for the clock inputs in this design (so long as the input power level is sensible). IIRC @WeiDaZhang looked at this with ADCLK948 eval boards when we were testing our ultra-low noise PLL designs, and found the improvement from the baluns is generally pretty marginal. Just make sure to AC couple the inputs.

So with AD948 we would use 7 x MMCX + one output to the backplane.

Okay, let's stick with that for the prototype board.

(The other option would be to buy an LVDS buffer evaulation board and measure the noise spectrum when launching into 50Ohm coax. But, personally I don't think this is worth the time it would take to make this measurement).

We can try not to load unused LVPECL outputs to save power.

You mean using piano switches and FETs to switch off the output biasing? If there is room on the PCB, that could be nice, but I don't think it's absolutely necessary.

[gkasprow]

@hartytp I mean unused negative outputs. Positive LVPECL outputs are loaded with 200G to GND and AC coupled with MMCXs while negative goes to GND via 200R. And here we can save some power.

[hartytp]

I mean unused negative outputs. Positive LVPECL outputs are loaded with 200G to GND and AC coupled with MMCXs while negative goes to GND via 200R. And here we can save some power.

Good point, I hadn't thought of doing that...

I've never tried doing that but, assuming it doesn't affect the clock buffers performance negatively, it sounds like a nice trick. Then, the power consumption is roughly the same as LVDS so there's no reason not to use the LVPECL buffers.

So long as we leave pads for the other resistors, we can always populate them if needed...

[hartytp]

It seems like we've reached a consensus on clock distribution. Summary of the above discussion:

• Keep ADCLK PECL clock buffers.
• [x] 7 outputs from the ADCLK948 go to the MMCX connectors, one differential pair to the BP
• [x] For the ADCLK948 outputs going to the MMCX connectors, use the positive outputs for the connectors, and leave the 200R resistors for the negative outputs DNF by default to save power
• [ ] Try to place/arrange the MMCX connectors to make them as accessible as possible
• [x] Use a resistor/solder jumper instead of the 100mil pin jumper to select the direction of the SMA clock input. Default configuration is as an input to the Si5324.
• [x] Scrap all baluns
• [x] AC couple all clock inputs/outputs
• [x] IC3 input select via jumper/dip switch

Edit: also, not to do with clocking (and doesn't necessarily have to be done right now), but:

• [x] Please can we have a power budget for Kasli on the schematic?
• [x] Can you add an annotation giving the max 12V current draw? This will be limited by the barrel connector or something like that
• [x] Can you add an annotation giving the max total 12V current for the EEMs (assuming that the connectors/wiring won't limit the current per individual EEM)

Combining these points with issues #217 and #228 gives the full to do list for Kasli. Unless anyone objects, let's freeze the specification for this prototype round once these changes are made.

[dhslichter]

Late the the party, but one more possibility: use SATA cables/connectors to do distribution of differential LVDS clocks. Cables and connectors are inexpensive, naturally well suited for differential signals, not sure about physical footprint but might be workable. Since the cables have two differential pairs, one could also have the option of running two different clock frequencies.

[gkasprow]

They are also well shielded and some of them are very flexible.

[gkasprow]

One can also think of using second pair for synchronisation.

[dhslichter]

SATA cables would work for LVPECL or LVDS, give a nice clean 100-ohm differential impedance, and then since everything remains differential out the receiving EEM board there is potentially better noise immunity and certainly simpler interfacing to/from the cable.

@hartytp @jordens thoughts? @gkasprow would the physical footprint work OK?

[gkasprow]

@dhslichter Let me check it.

[gkasprow]

@dhslichter @hartytp @jordens This could work providing that Kasli occupies 8 HP (2 standard slots). The only way to connect such number of SATA cables is using vertical connectors: Right Angled ones won't fit.

[dhslichter]

This mockup from @gkasprow has above looks pretty reasonable to me, actually. The need to move to 6 HP or 8 HP from 4 HP seems to me to be not that much of a problem, but others can say how much this is an issue. Assuming a backplane such that Kasli can drive up to 12 IDC connections, then we have up to 12 EEMs, which may be a mix of 4 and 8 HP (call it average 6 HP?), then this yields 72 HP. This means to me that we are in the right ballpark for one fully loaded Eurorack run by one Kasli, even if it goes to 8 HP width.

Some considerations:

• do we need matched lengths for clock cables to different Urukul boards? Zeroth order I would argue no.
• where will clock distribution cables and IDC cables be routed? Card guides will restrict over/under routing. Are we relying on EEM cards being "short" and not plugging in to a backplane, with IDC and clock distribution going around behind? If so, will we need multiple variants of cards so that we can both accommodate cables (short version) or use a backplane IDC (long version)?
• We will probably start running in to thermal issues if we are packing 12 EEM plus a Kasli into a single Eurorack, so a little extra width for the Kasli may not be a bad thing.

[hartytp]

@dhslichter Agreed that routing clocks over differential cabling is nice. Also agreed that using cheap PC cables is nicer than having to splash out on MMCXs. Assuming that they're good for 1GHz analog clocks, and that they work out mechanically, I don't object to using SATA cables.

If we do use 100Ohm differential cables, we should use LVDS for our clock distribution, rather than LVPECL to keep the power dissipation down. I'd go for ADCLK854, or anything cheaper that gives a proper phase noise specification and goes up to 1GHz.

The need to move to 6 HP or 8 HP from 4 HP seems to me to be not that much of a problem

That doesn't worry me too much.

do we need matched lengths for clock cables to different Urukul boards? Zeroth order I would argue no.

I'd argue not as well.

If so, will we need multiple variants of cards so that we can both accommodate cables (short version) or use a backplane IDC (long version)?

We have the same issue for the ribbon cables as well. AFAICT, so far no EEMs have been designed to support a backplane, so cables can be routed around the back of the rack for now.

[jordens]

I don't think we need differential clock distribution. The other channel is also unused since we won't drive it. And I have not seen that unshielded clocks over SATA cables are comparable to single ended shielded coax. Doubling the width of Kasli is not worth that IMHO. Those cables are also thicker and use more space than coax cables.

[gkasprow]

@jordens SATA cables are well shielded -these are twinax cables with aluminium foil shield and some braid on it. Look here One can also use 3M sata twinax cables that works well up to 20GHz. That's true - they are much more rigid and take much more space.

[gkasprow]

On the other hand, MMCX are available in plenty of forms and lengths - MMCX to MMCX jumpers, SMA pigtails, so one can use them to distribute clock to front panel bezel with SMAs on it. They are also much more durable than u.fl connectors and are much cheaper than SMPs because are widely used in GSM and GPS applications.

[jordens]

@gkasprow Ack. My mistake. But the connectors don't seem to be shielded AFAICT. Yes. Being able to wire an alternative clock distribution to the frontpanel or in the back of the rack without SATA adapter and translation boards for those clocks would be another reason to go for regular coax connectors IMHO.

[hartytp]

I'm happy with either MMCX or SATA (although, personally I'd opt for MMCX).

At this point, my top priority is finishing the design and getting the prototypes sent off for manufacture.

[dhslichter]

I am fine with either solution as well, I just wanted to throw the SATA option out there for discussion. I don't have any hard data on the performance of SATA vs coax in terms of clock distribution, although the data speed ratings for SATA cables (~6 Gbps) would give some idea just because of the jitter requirements to meet the corresponding eye diagrams. There are certainly advantages to being able to run clocks out through the front panel as well if we want to, and coax is more natural for that.

Would it be worth running a quick test (or could it be simulated?) of the LVPECL-over-coax signal integrity to make sure it will perform OK?

I still think it would be good to make Kasli a little wider so we could use straight (not right angle) MMCX connectors.

[hartytp]

Would it be worth running a quick test (or could it be simulated?) of the LVPECL-over-coax signal integrity to make sure it will perform OK?

@WeiDaZhang Do you still have the ADCLK948 eval board? If so, would you mind testing the phase noise in the following configuration:

• Input: 100MHz from Wenzel oscillator, AC coupled onto positive input, with negative input grounded via a capacitor (no input balun).
• Output: positive input AC coupled to SMA (200R biasing resistor), negative input floating (no biasing resistor), no balun.

Assuming the phase noise in that measurement is better than the AD9910 PLL off phase noise, I'd argue that LVPECL over coax is fine for this design.

I still think it would be good to make Kasli a little wider so we could use straight (not right angle) MMCX connectors.

You mean adding extra (unused/blank) width to the front panel? I'd prefer to leave the panel as narrow as possible. If the user wants more width, they can leave the adjacent slot(s) empty and add a blanking plate.

[gkasprow]

@hartytp user can use any type of MMCx connectors one wants. 4HP Euro panels are available in every catalogue so one can buy them and use.