Closed jordens closed 7 years ago
@jordens What max EEM clock do you want? 500MHz? 1GHz? Faster?
1 GHz.
@jordens How to you want to do the EEM fan out? Have 8 (4?) RF connectors on Kasli?
Yes. If that fits. But I am worried about space. Maybe it is best to just have a 3U card that is the fanout... That frees space on Kasli, could feed 12 or more EEM and decouples problems/cost by making that clock distribution optional. @gkasprow ?
@hartytp We plan to have several clock outputs from fanout chip. They are distributed in parallel with IDCs using MMCX cables.
@jordens @gkasprow Either approach works for me. If there is room, I'd prefer having the MMCX connectors directly on Kasli, as it reduces the number of 3U boards one has to buy and fit into a rack.
@hartytp @jordens will do our best to fit all stuff on signgle board.
Only if it is really easy, provide a way to feed DRTIO clock (same as the one going to the FPGA logic and transcievers) to the EEM-fanout.
I'd be keen on this if it can be done in a convenient manner: IMO the most common EEM clock will be the DRTIO clock, whether Kasli is operated in master or slave mode. Given that, it makes sense to have an easy way of routing it to the EEM fanout buffer without having to externally jumper the 2 SMAs together.
Otherwise, all looks good to me.
@hartytp @jordens @jbqubit It seems we will manage to fit clock distribution to Kasli board.
Clocks will be routed using right angled MMCX connectors
The clock distribution became more complex but this should make everybody happy
Greg, thanks for posting that! That all looks good to me.
A few minor points about the clocking:
Is that okay at 1GHz?
Wasn't SMA1 clock supposed to be at 100MHz?
The current arrangement of MMCX connectors looks very tight.
We'll move them up, above Artix and space them out.
Wasn't SMA1 clock supposed to be at 100MHz?
Sorry, my mistake, SMA1 is at 125MHz, not 1GHz. IIUC, it might move to ~250MHz in the future if we increase the RTIO_COARSE frequency to allow for a faster CPU.
Are you happy with the jumper at 125MHz?
RTIO and CPU frequencies are not linked, it's only accidental that they are both 125MHz in current systems (but typically derived from different oscillators).
@sbourdeauducq thanks for clarifying.
In any case, it still seems that this hardware should support RTIO_CLK to at least 200MHz. For example to enable the 1.6GHz DAC_CLK, 800MHz f_data configuration for Sayma in artiq hardware.
@hartytp Yes, we can replace the buffer with CMOS one, the question is what additive jitter do you accept? It's hard to find 1GHz capable CMOS buffer since there is no CMOS single ended standard that supports it. If we limit frequency to 200MHz then the LMK00101SQ looks good, has low jitter and it has 10 outputs so we get rid of balun. The standard jumper should work fine at 200MHz, at 1GHz we may get troubles. One can always solder 0805 resistor to the pads. Yes, we can get rid of baluns and use single ended configuration. AC coupling is also possible. Of course, IC3 must have a jumper and/or be controlled by FPGA.
It's hard to find 1GHz capable CMOS buffer since there is no CMOS single ended standard that supports it.
Yes, for 1GHz the lower cost/power alternative to LVPECL is LVDS, not LVCMOS. Oops!
Yes, we can replace the buffer with CMOS one, the question is what additive jitter do you accept?
A good starting point is the phase noise floor for the Si5324 and AD9910 PLL (cf my last comment on #201). IMO, anything that won't degrade them is fine for the Kasli fanout; if users really want an ultra-low noise fanout they're probably better off using a dedicated one on a separate card.
The standard jumper should work fine at 200MHz, at 1GHz we may get troubles. One can always solder 0805 resistor to the pads.
Okay.
Of course, IC3 must have a jumper and/or be controlled by FPGA.
Jumper control is probably better IMO.
Looks good!
would like some (i.e. 2) of clocks on the backplane.
Alternatively, we could stick with a single clock pair on the bp connector and put a fanout buffer on the bp itself (it already has power, so this is trivial):
At the moment we have only one clock pair on BP. MMCX are not that expensive - below 1$$. We can always not mount all of them. It would be better to route diff clock over BP connector. At the moment we don't have any free pins on BP. Number of GNDs is barely sufficient. So I'd like to keep single clock diff line over BP. So what about LMK00101SQ and balun to make clock diff over BP pins ? It has ~30fs of jitter and works up to 200MHz
At the moment we have only one clock pair on BP.
Let's stick with 1 diff clock pair to the BP, and add a fanout buffer on the BP if required.
MMCX are not that expensive - below 1$$. We can always not mount all of them.
That sounds good. So long as they're not too hard to access, the current number is fine then.
So what about LMK00101SQ and balun to make clock diff over BP pins ? It has ~30fs of jitter and works up to 200MHz
We would like to have the option of running the BP clock up to 1GHz (e.g. DDS with no PLL), so can we use a faster LVDS buffer than this? I've seen a few suitable ones around from ADI and others.
Otherwise, that all looks really good @gkasprow! Great job!
@jordens @hartytp I'm not sure if LVDS is good choice as single ended coaxial line driver. It has current output which means that don't like capacitive loads which quickly degrades clock edges and thus increases jitter. It is dedicated to drive low capacitance diff lines on PCB. For this reason I'd keep LVPECL or use version with 10 outputs (ADCLK950BCPZ) so could get rid of balun. Example 1G capable LVDS driver - LMK01010 costs 12$ - more than ADCLK948BCPZ (9$). ADCLK950BCPZ costs 11$. LMK01010 LVDS version cosumes 160mA, LVPECL (LMK01020) - 340mA so there is 0.5W more power in case LVPECL.
But the typical 300 fs jitter sounds prohibitive. If I have to choose between 1 GHz/300 fs and 200 MHz/30 fs, I'd take the 30 fs.
Jitter doesn't need to be a problem for LVDS muxes/buffers. See e.g. ADCLK854 fig 12. This is better than the AD9910 PLL noise, so not a problem. Cost is $6USD in quantity from Digi-Key.
I'm not sure if LVDS is good choice as single ended coaxial line driver. It has current output which means that don't like capacitive loads which quickly degrades clock edges and thus increases jitter. It is dedicated to drive low capacitance diff lines on PCB. For this reason I'd keep LVPECL. Example 1G capable LVDS driver - LMK01010 costs 12$ - more than ADCLK948BCPZ (9$). LVDS version cosumes 160mA, LVPECL (LMK01020) - 340mA so there is 0.5W more power in case LVPECL.
@gkasprow Is this still a problem when launching into properly terminated 50Ohm coax (i.e. not a capacitive load)? Or, is the issue that you'd need a balun on each clock output to do this properly?
I don't feel strongly about the LVDS v LVPECL issue, so I'm happy for you to go with whichever you think will be best.
@hartytp I don't have strong arguments that LVDS would have issues with coax cables. It is more intuition than knowledge. Generally people use LVPECL for low jitter clock distribution with coax cables. What we can do is to use LMK01000 series chip that in same package has LVDS and LVPECL option. So let's use LMK01010 with LVDS and if we experience jitter issues, will switch to LMK01020 with LVPECL outputs and solder termination resistors.
In case of input baluns, we can mark them DNP and install coupling capacitors below them. Then in case of jitter issues we can still mount them.
@gkasprow , @hartytp all ACK.
I don't have strong arguments that LVDS would have issues with coax cables. It is more intuition than knowledge. Generally people use LVPECL for low jitter clock distribution with coax cables.
Good to know! I haven't got a huge amount of experience with this kind of thing, so I'd tend to trust your intuitions.
So let's use LMK01010 with LVDS and if we experience jitter issues, will switch to LMK01020 with LVPECL outputs and solder termination resistors.
FWIW, another option which could be nice, is the HMC6832:
The LMK01010/LMK01020 is almost certainly fine. Comments:
Given your concerns with LVDS as a coax driver, I'd be happy sticking with the current ADCLK948 -- maybe it's overkill for our application, but at least we have a high confidence that it will work in the first revision. Do we really want to get into testing the phase noise of clock buffers to try to save a Watt on Kasli? (I'd also probably prefer ADCLK948 without baluns to LVDS + baluns).
In case of input baluns, we can mark them DNP and install coupling capacitors below them. Then in case of jitter issues we can still mount them.
I'd prefer to avoid the baluns if possible, as they'll take up quite a lot of space and inflate the BOM. Based on our past experience, my guess is that they're not needed. We could add a balun on only 1 channel for now, so we can test the phase noise with and without before making a final decision if you think it's a good idea?
Another benefit of sticking with the ADCLK948 is that we're already using it elsewhere (Sayma/Baikal/etc), so it's already a "known quantity". It's always good to minimise the number of IC datasheets one has to read to understand the system...
@hartytp I meant baluns on SMA clock inputs. If you are sure that we don't need it, that's OK. So with AD948 we would use 7 x MMCX + one output to the backplane. We can try not to load unused LVPECL outputs to save power.
I meant baluns on SMA clock inputs. If you are sure that we don't need it, that's OK.
Baluns shouldn't be needed for the clock inputs in this design (so long as the input power level is sensible). IIRC @WeiDaZhang looked at this with ADCLK948 eval boards when we were testing our ultra-low noise PLL designs, and found the improvement from the baluns is generally pretty marginal. Just make sure to AC couple the inputs.
So with AD948 we would use 7 x MMCX + one output to the backplane.
Okay, let's stick with that for the prototype board.
(The other option would be to buy an LVDS buffer evaulation board and measure the noise spectrum when launching into 50Ohm coax. But, personally I don't think this is worth the time it would take to make this measurement).
We can try not to load unused LVPECL outputs to save power.
You mean using piano switches and FETs to switch off the output biasing? If there is room on the PCB, that could be nice, but I don't think it's absolutely necessary.
@hartytp I mean unused negative outputs. Positive LVPECL outputs are loaded with 200G to GND and AC coupled with MMCXs while negative goes to GND via 200R. And here we can save some power.
I mean unused negative outputs. Positive LVPECL outputs are loaded with 200G to GND and AC coupled with MMCXs while negative goes to GND via 200R. And here we can save some power.
Good point, I hadn't thought of doing that...
I've never tried doing that but, assuming it doesn't affect the clock buffers performance negatively, it sounds like a nice trick. Then, the power consumption is roughly the same as LVDS so there's no reason not to use the LVPECL buffers.
So long as we leave pads for the other resistors, we can always populate them if needed...
It seems like we've reached a consensus on clock distribution. Summary of the above discussion:
Edit: also, not to do with clocking (and doesn't necessarily have to be done right now), but:
Combining these points with issues #217 and #228 gives the full to do list for Kasli. Unless anyone objects, let's freeze the specification for this prototype round once these changes are made.
Late the the party, but one more possibility: use SATA cables/connectors to do distribution of differential LVDS clocks. Cables and connectors are inexpensive, naturally well suited for differential signals, not sure about physical footprint but might be workable. Since the cables have two differential pairs, one could also have the option of running two different clock frequencies.
They are also well shielded and some of them are very flexible.
One can also think of using second pair for synchronisation.
SATA cables would work for LVPECL or LVDS, give a nice clean 100-ohm differential impedance, and then since everything remains differential out the receiving EEM board there is potentially better noise immunity and certainly simpler interfacing to/from the cable.
@hartytp @jordens thoughts? @gkasprow would the physical footprint work OK?
@dhslichter Let me check it.
@dhslichter @hartytp @jordens This could work providing that Kasli occupies 8 HP (2 standard slots). The only way to connect such number of SATA cables is using vertical connectors:
Right Angled ones won't fit.
This mockup from @gkasprow has above looks pretty reasonable to me, actually. The need to move to 6 HP or 8 HP from 4 HP seems to me to be not that much of a problem, but others can say how much this is an issue. Assuming a backplane such that Kasli can drive up to 12 IDC connections, then we have up to 12 EEMs, which may be a mix of 4 and 8 HP (call it average 6 HP?), then this yields 72 HP. This means to me that we are in the right ballpark for one fully loaded Eurorack run by one Kasli, even if it goes to 8 HP width.
Some considerations:
@dhslichter Agreed that routing clocks over differential cabling is nice. Also agreed that using cheap PC cables is nicer than having to splash out on MMCXs. Assuming that they're good for 1GHz analog clocks, and that they work out mechanically, I don't object to using SATA cables.
If we do use 100Ohm differential cables, we should use LVDS for our clock distribution, rather than LVPECL to keep the power dissipation down. I'd go for ADCLK854, or anything cheaper that gives a proper phase noise specification and goes up to 1GHz.
The need to move to 6 HP or 8 HP from 4 HP seems to me to be not that much of a problem
That doesn't worry me too much.
do we need matched lengths for clock cables to different Urukul boards? Zeroth order I would argue no.
I'd argue not as well.
If so, will we need multiple variants of cards so that we can both accommodate cables (short version) or use a backplane IDC (long version)?
We have the same issue for the ribbon cables as well. AFAICT, so far no EEMs have been designed to support a backplane, so cables can be routed around the back of the rack for now.
I don't think we need differential clock distribution. The other channel is also unused since we won't drive it. And I have not seen that unshielded clocks over SATA cables are comparable to single ended shielded coax. Doubling the width of Kasli is not worth that IMHO. Those cables are also thicker and use more space than coax cables.
@jordens SATA cables are well shielded -these are twinax cables with aluminium foil shield and some braid on it. Look here One can also use 3M sata twinax cables that works well up to 20GHz. That's true - they are much more rigid and take much more space.
On the other hand, MMCX are available in plenty of forms and lengths - MMCX to MMCX jumpers, SMA pigtails, so one can use them to distribute clock to front panel bezel with SMAs on it. They are also much more durable than u.fl connectors and are much cheaper than SMPs because are widely used in GSM and GPS applications.
@gkasprow Ack. My mistake. But the connectors don't seem to be shielded AFAICT. Yes. Being able to wire an alternative clock distribution to the frontpanel or in the back of the rack without SATA adapter and translation boards for those clocks would be another reason to go for regular coax connectors IMHO.
I'm happy with either MMCX or SATA (although, personally I'd opt for MMCX).
At this point, my top priority is finishing the design and getting the prototypes sent off for manufacture.
I am fine with either solution as well, I just wanted to throw the SATA option out there for discussion. I don't have any hard data on the performance of SATA vs coax in terms of clock distribution, although the data speed ratings for SATA cables (~6 Gbps) would give some idea just because of the jitter requirements to meet the corresponding eye diagrams. There are certainly advantages to being able to run clocks out through the front panel as well if we want to, and coax is more natural for that.
Would it be worth running a quick test (or could it be simulated?) of the LVPECL-over-coax signal integrity to make sure it will perform OK?
I still think it would be good to make Kasli a little wider so we could use straight (not right angle) MMCX connectors.
Would it be worth running a quick test (or could it be simulated?) of the LVPECL-over-coax signal integrity to make sure it will perform OK?
@WeiDaZhang Do you still have the ADCLK948 eval board? If so, would you mind testing the phase noise in the following configuration:
Assuming the phase noise in that measurement is better than the AD9910 PLL off phase noise, I'd argue that LVPECL over coax is fine for this design.
I still think it would be good to make Kasli a little wider so we could use straight (not right angle) MMCX connectors.
You mean adding extra (unused/blank) width to the front panel? I'd prefer to leave the panel as narrow as possible. If the user wants more width, they can leave the adjacent slot(s) empty and add a blanking plate.
@hartytp user can use any type of MMCx connectors one wants. 4HP Euro panels are available in every catalogue so one can buy them and use.
IN_SEL
needs a jumper to select the input.