Kasli v1.0 tests & errata

[marmeladapk]

To be changed for rev1.1

• Misc:
  • [x] R57 must be 300R
  • [x] Improve signal integrity for backplane LVDS lines (remove reference plane cut) (one cut between 2v5 and 12V0 seems unavoidable)
  • [x] Add two more USER_LEDs
  • [x] (v1.1/rj) consider adding a PCB-mount heatsink (one of the FPGAs died from overheating during testing)
  • [ ] Document power supply design parameters (#317)
  • [x] update
  • [x] and double check power budget after changes.
  • [x] add a PN for that clock balun on the schematic
  • [ ] Change revision to 1.1 on schematics
• USB:
  • [x] (v1.1/rj) Swap I2C with UART on FTDI chip (channel C doesn't have MPSSE)
  • [x] (v1.1/rj) fix direction of the IC19 (both channels)
• I2C:
  • [x] (v1.1/rj) add TVS on EEM I2C
  • [x] (v1.1/rj) add ID chip 24AA02E48T-I/OT, same I2C port as the Si5324
  • [x] (v1.1/rj) Connect reset of the I2C routers IC14 IC15 to the FPGA
  • [x] Add TCA9517 to Si5324 I2C
  • [x] Connect IC10:A1 (I2C_SW_RESET) to one of the unused BDBUS lines (BDBUS7 or BDBUS5 if you can) and not to CDBUS4.
• Backplane:
  • [x] (v1.1/rj) Inconsistent power connector PN 712RA vs L712RA (712RA is the correct one) (it was just wrong footprint name, PN was correct; PK)
  • [x] (v1.1/rj) Remove second power connector J17.
  • [x] (v1.1/rj) Populate the backplane connector J20.
• SFP:
  • [x] Move SFP LEDs closer to panel (same as was done with USB)
  • [x] Free up six pins on the FPGA by having MOD_DEF1 and MOD_DEF2 routed through the I2C switch only and not additionally directly to the FPGA.
  • [x] Remove the I2C level converters IC10 on those SFP I2C busses.
  • [x] (v1.1/rj) Connect RATE_SELECT1 (pin 9 of the SFP) to the FPGA
• Clock distribution
  • [x] (v1.1/rj) Remove the 50 MHz oscillator OSC1. Not needed as the 125 MHz GTP input can be routed to fabric.
  • [x] (v1.1/rj) Remove IC3
  • [x] Remove the CLK_SEL logic of IC3 (free up FPGA pin)
  • [x] (v1.1/rj) Remove J19 and related components
  • [x] (v1.1/rj) Remove J5-J7 and related components
  • [x] Remove the R98 path of CLK_OUT (IC1:Q2 to the front panel SMA)
  • [x] (v1.1/rj) Add an LDO for the power supply of IC1 and IC2 (also worth adding a ferrite/inductor between the SMPS and the LDO to kill high-frequency spurs).
  • [x] (v1.1/rj) Connect four MMCX (J1-J4) to the Q2, Q2B, Q3, Q3B outputs of IC1
  • [x] (v1.1/rj) Mark J2 and J4 (on the negative Q2B and Q3B outputs) clearly on the PCB
  • [x] (v1.1/rj) Connect backplane clock CLK_EXT_P/N to CKOUT2+- of IC2
  • [x] (tph) change 0R resistor to 100k to float SMA clock input
  • [x] (tph) consider floating MMCX connector grounds in same way as SMAs (only if you have time!)
  • [x] Change the transformer TR1 to TC2-1TX+. No changes to layout. Footprint and package are compatible. But pinout is rotated 180 degrees.
  • [x] DNP C13, C9
• Layout:
  • [x] Either teardrops everywhere or nowhere. Right now there are teardrops on roughly half the pins.
  • [x] Set up consistent rules
  • [x] Properly clean up polygons
  • [ ] Don't do arcs where they are not needed.
  • [x] Remove weird teardrop on FTDI
  • [x] Connect PCB edge GND stitching vias to top and bottom GND pour. Stick with @gkasprow recommendation on via density.
  • [x] Check XADC connection to components
  • [x] Replace C49 with a 0R (connect VREFN to XADC_GND) to use internal reference (connected without jumper, PK)

[jordens]

@gkasprow Doing I2C through the switch only also gets rid of IC10{ABC}. I'd rather like to use these FPGA pins for:

• RATE_SELECT1 (pin 9 of the SFP).
• Reset of the I2C routers. If there lines shorted, on the I2C output ports (as there is on my board), there is no way to get out of that situation and reconfigure the I2C routers other than power cycling. That's bad.
• Add ~test points and~ two more USER_LED. (Forget about the test points. The EEM connectors are perfect test points).

[jordens]

• You can put the EEPROM on the same I2C port as the SI5324. But I guess you have figured that out as well.

[jordens]

• And I have changed my mind on the second power connector that is pointing towards the backplane. That can go (removed entirely). If someone wants to power internally, then just use the backplane connector, that EMI filter, and a connector on the backplane. In turn please populate the backplane connector.

[hartytp]

I am pretty sure the effects on routability, SI and cleanliness will be there. If the 7x MMCX array and the ADCLK948 are gone, that's a significant amount of space on all layers.

Well, it's not a deal breaker for me, but I'd still prefer not to remove these unless @gkasprow and @marmeladapk really think it will help (they are the ones best placed to make that call). I would have no objection to cutting down to 4-5MMCX connectors if that helped (although, we discussed this before and, IIRC, @gkasprow and @marmeladapk said there was no point removing them since the cost is negligible and removing them doesn't really help with the design).

And I have changed my mind on the second power connector that is pointing towards the backplane. That can go (removed entirely). If someone wants to power internally, then just use the backplane connector, that EMI filter, and a connector on the backplane. In turn please populate the backplane connector.

Great! I completely agree. Let's just make the BP adapter with a power jack on it.

[gkasprow]

@jordens true, I met some SFP modules that used these pines in other way, but they were custom CERN designs that also used both RD and TD as outputs :)

[gkasprow]

@jordens @marmeladapk On two kasli boards we noticed failed I2C switch. This may be due to ESD. The I2C are open drain with no protection diode. So we should add TVS to each EEM connector I2C lines.

[hartytp]

@jordens re the clock buffer: this is still a high value feature for us.

• I don't anticipate any noise issues with the 125MHz clock from the Si chip (assuming we add an LDO for it or at least some better PSU filtering). If we get close to matching the datasheet performance then it will as good as the AD9910 PPLs. Also, most of our Urukul applications aren't that noise sensitive anyway, so a few spurs won't hurt us.
• For Kasli DRTIO slaves, it's really convenient not to have to supply a clock via coax. This makes wiring much nicer. So, it's really nice to have a clock source output at the RTIO_COARSE frequency. That way, Kasli alone can supply everything (power, DIO and clocks) needed to run EEMs.
• We generally have 100MHz references but not 125MHz/150MHz references, so it's really useful to have Kasli produce the 125MHz/150MHz we need to clock our DDSs from. It's also nice that we don't need to buy new reference oscillators for our DDSs if we change the RTIO frequency.
• We can use an external clock buffer such as clocker if we loose the fanout buffer on Kasli (so long as we keep at least an MMCX clock output). But then we have to source power for that (NB we plan to use all EEM connectors on Kasli, so there won't be one spare to power clocker in many cases). It also increases the cost of the overall system, takes up more rack space etc.
• The clock fanout isn't just a fanout but also a mux for the EEM clock fed to the backplane connector. If we loose the mux then we loose the ability to feed an external clock to the BP connector (we then only have the Si chip).
• While I understand your concerns, we have so far seen no evidence that the clock fanout chip on Kasli causes any SI issues etc on the prototype, so there is no actual evidence to back your concerns up. Also, @gkasprow and @marmeladapk -- the experts doing the design -- haven't yet agreed that the concerns are well founded.

[hartytp]

But, if you're really set on removing it then a possible compromise would be:

• Route the RTIO_coarse clock to the BP
• Keep a single MMCX on Kasli which also outputs the RTIO_coarse clock from the Si chip
• Put a 4x fanout + MMCXs on the Kasli BP adapter. Then we can use that for our applications.

[jordens]

@hartytp I am OK with exposing the clock at a few places (the backplane, an Kasli MMCX and the panel SMA). But I have the following observations with the 8x clock fanout:

• We definitely need additional space for an LDO for IC2, IC1.
• Kasli is extremely dense. I have been looking at the layout for the last couple weeks and I really don't like the fact that the density has forced Pawel to place all kinds of unrelated logic groups inside of each other. That makes debugging hard, makes reading the layout hard, and is just bad style. The fact that you can cram that much logic onto a board does not mean that you should do it. There are enormous downsides. It is also bad when changing things in the design as there are unnecessary knock-on changes and large parts have to be rerouted. The attitude of making future changes harder just because there is minute short term benefit is not a good one.
• The 7 or 8 coax and MMCX connectors make assembly more tricky and lead to a worsening of the cooling situation. It obstructs cooling (they run straight over the FPGA where the heatsink should be). Moving the 8x buffer off of Kasli somewhere else would save a significant amount of power (and temperature) especially in the high-density areas of IC1 and IC2.
• You can still do your 125 MHz fanout. You'll get that in three places: the SMA at the front, the backplane and the MMCX on Kasli. Just use Clocker. Otherwise, what purpose does Clocker serve since you would already be getting 7 MMCX from Kasli? Using Clocker (or a fan-out on the backplane) makes wiring nicer as it reduces contention in the area where things are the worst.
• You have to decide what's important for you. You say that the 125 MHz are "rare", but then you still want the option to not use them but feed the backplane with something else. This weakens both arguments.
• How would you use the MMCX input to the mux on Kasli if not from Clocker? That makes the mux on Kasli pretty useless as Clocker already gives you those exact outputs.
• Have the 2x8 fanout/mux on the backplane. You get the same mux options as before. And 8 outputs is sufficient to drive even setups with 8 Urukul/Mirny. That's unlikely to be standard.
• Sure there is evidence for SI issues. As Pawel and Greg stated, the routing is extremely dense and that lead to SI issues in the EEM connectors.

[hartytp]

The 7 or 8 coax and MMCX connectors make assembly more tricky and lead to a worsening of the cooling situation.

I would be totally happy with reducing that to 4.

You have to decide what's important for you. You say that the 125 MHz are "rare", but then you still want the option to not use them but feed the backplane with something else. This weakens both arguments.

IIRC, you were the one who wanted that. I'd be fine with loosing the external input and only supporting backplane clocks from the Si chip. Just pointing out that we loose the muxing capacity if we remove the fanout. But, ...

Have the 2x8 fanout/mux on the backplane. You get the same mux options as before. And 8 outputs is sufficient to drive even setups with 8 Urukul/Mirny. That's unlikely to be standard.

That's true. I hadn't considered that. The only downside of this is that there is no convenient way of feeding this mux via the front panel.

Otherwise, what purpose does Clocker serve since you would be getting 7 MMCX from Kasli?

That was more targeting ultra-low noise cases. I'd prefer not to have to use if for simple fanout unless really necessary.

[hartytp]

But, having said all that, I would be okay with moving the mux + fanout to the BP adapter.

[marmeladapk]

@jordens Removing clock fanout won't help with anything (unless we notice SI issues and pinpoint them to lines being too close or smth else). Right now routing is a solved problem and removing for example W1 or MMCXs won't change anything because I won't reroute lines in proximity (there's no need to).

(I will reroute lines that are too close to SFP TX and RX lines.)

Sure there is evidence for SI issues. As Pawel and Greg stated, the routing is extremely dense and that lead to SI issues in the EEM connectors.

Haven't seen any TBH. Greg neither. And the point about removing reference plane cut is more of a precaution as I didn't observe any differences between signal on lines moving over the cut and on lines from eem0-8.

[marmeladapk]

@jordens And routing is very dense but near and under FPGA (eems routing was very tricky) not near clock circuits.

[jordens]

Making room to cool Kasli properly is high priority to me. A block of 8 EEM connectors, 8 ribbon cables, 8 MMCX connectors and 8 coax cables is an almost perfect wall. The opposition to adding 4 HP of space to Kasli is the wrong place to save space IMHO. Having racks filled in the front and in the back (like AMC/RTM) would be easier if HPs are at a premium and a the backside operation is apparently a small inconvenience. The EEM crates are really not deep right now. You can stuff boards from both sides in about 50 cm of crate depth. You can put Samplers on one side and Urukul on the other (at the same side as the RFPA which is much deeper). But please allow 8 HP for Kasli.

There is also the mux isolation issue. With the 75 dB spurs on Urukul with the capacitors placed to route the other input and the switch switched to the SMA, I'd expect something similar on the Kasli mux just by proximity. And it won't be easily curable there.

[hartytp]

8 coax cables is an almost perfect wall.

Can reduce this to 4. But, even 8 coax cables aren't really an airflow problem compared to the ribbon cables.

There is also the mux isolation issue. With the 75 dB spurs on Urukul with the capacitors placed to route the other input and the switch switched to the SMA, I'd expect something similar on the Kasli mux just by proximity. And it won't be easily curable there.

I have no issue with scrapping the mux and only keeping the fanout.

[hartytp]

Anyway, so long as we have a mux on Kasli or the BP adapter, I'm less fussed about where it is. So, will leave final call to @jordens and @marmeladapk. No more comments from me on this

[jordens]

@marmeladapk Ack. Thanks for the input. IMO any future layout changes would still be made easier if the 8x fanout and 7 MMCX were gone. @hartytp What about the following: We scrap the idea of having the SMA connector output the RTIO clock and use that for input exclusively. Then we add four MMCX on the four remaining outputs of the 4x fanout (IC1, two positive and two negative). The fact that there is a 180deg phase shift on two of them is not a problem IMHO. You can now go with a nice short pigtail from Kasli to its 8HP front panel if you want a RTIO clock output there. You can also go with 4 MMCX connectors to 4 Urukul/Mirny boards. You'd still have the additional differential output to the backplane (CLKOUT2 from IC2).

[hartytp]

@jordens that would be fine for me. I don't anticipate using the SMA clock output, since I imagine doing all clock distribution inside the rack. In the rare case that it's done one can add a pigtail/clocker (I don't mind doing that in this case since I don't think it will be very common, so it's okay if we don't make it quite as convenient).

[jordens]

@hartytp @marmeladapk Ok. Thanks. Then let me summarize:

• Remove IC3
• Remove the CLK_SEL logic of IC3 (free up FPGA pin)
• Remove J19 and related components
• Remove J5-J7 and related components
• Remove the R98 path of CLK_OUT (IC1:Q2 to the front panel SMA)
• Add an LDO for the power supply of IC1 and IC2
• Connect four MMCX (J1-J4) to the Q2, Q2B, Q3, Q3B outputs of IC1
• Mark J2 and J4 (on the negative Q2B and Q3B outputs) clearly on the PCB
• Connect backplane clock CLK_EXT_P/N to CKOUT2+- of IC2

(all the errata added to the top post)

[jordens]

@marmeladapk Why does R57 need to be 300R?

UG470:

DONE has an internal pull-up resistor of approximately 10 kΩ. There is no setup/hold requirement for the DONE register. These changes, along with the DonePipe register software default, eliminate the need for the DriveDONE driver-option. External 330Ω resistor circuits are not required but can be used as they have been in previous generations.

[marmeladapk]

@jordens @hartytp Please tell me if you consider v1.1 to be a final version or do you expect v1.2?

IMO any future layout changes would still be made easier if the 8x fanout and 7 MMCX were gone.

That part of the board was not dense and I had no trouble routing it. Removing anything is more work than DNPing it, unless you anticipate significant rework of clock recovery circuit schematics. So please consider if removing IC3 is still needed not on a premise that the board is dense, but on something else. If cost and power savings are worth it or if we're worried about cables obstructing cooling then we can DNP parts and move CLK_EXT to Q3 of IC1.

mux isolation issue. With the 75 dB spurs on Urukul

What caused it? If we can measure similar effect on Kasli and it's not easily fixable then it defeats the purpose of fanout.

Add an LDO for the power supply of IC1 and IC2

We could supply IC1-3 with TPS79601 at 3,05 V from 3,3 V.

Alternatively, if we get rid of IC3 what do you think about supplying IC1 and IC2 with 2,5V from LT3045EDD? Both chips can be powered with 2,5 V.

Connect backplane clock CLK_EXT_P/N to CKOUT2+- of IC2

Do you want it to stay as LVPECL or change lines to LVDS?

Why does R57 need to be 300R?

IDK, but with 2k2 Xilinx platform cable didn't recognise that the FPGA is programmed.

[hartytp]

Please tell me if you consider v1.1 to be a final version or do you expect v1.2?

I currently don't have any more changes to suggest (although, I haven't played with these boards much yet).

That part of the board was not dense and I had no trouble routing it. Removing anything is more work than DNPing it, unless you anticipate significant rework of clock recovery circuit schematics. So please consider if removing IC3 is still needed not on a premise that the board is dense, but on something else. If cost and power savings are worth it or if we're worried about cables obstructing cooling then we can DNP parts and move CLK_EXT to Q3 of IC1.

I'm not fussed either way.

Do you want it to stay as LVPECL or change lines to LVDS?

IIRC, @gkasprow thought that LVPECL is better for driving cables etc than LVDS. I'd prefer to stick with it unless we need to change.

We could supply IC1-3 with TPS79601 at 3,05 V from 3,3 V. Alternatively, if we get rid of IC3 what do you think about supplying IC1 and IC2 with 2,5V from LT3045EDD? Both chips can be powered with 2,5 V.

LGTM. Those regs are nice low noise regulators which should do the job. Anything alone those lines that meets the supply requirements of the chips is fine by me!

[marmeladapk]

@jordens Is there any reason other than visual that you brought back arc issue?

[jordens]

I hope that this will be the final revision of the v1.x series.

I am fine with DNPing instead of removing if it achieves the same. Let me mark the changes that I definitely want to see in v1.1 (marked as (v1.1/rj)).

@marmeladapk Yes visual. I don't think the electrons care. But it looks amateurish, conveys a false sense of what is needed and important and seems to indicate that there is some hidden magic going. It's not a bug that prevents something from working but it is not good and something that I would like to see changed.

[jordens]

And if routing that clock network was as trouble-free as you say (which I find surprising), then re-routing it is even easier! (That comment on the visual aspect of the layout does not refer to your skills which I admire. I hope that's clear).

[gkasprow]

@jordens existing resistor caused JTAG error that DONE pin did not go high. And once I changed it to 300R the problem was solved.

[jordens]

That's weird.

[gkasprow]

I had same issue with Spartan chips so didn't think too much about any other fix :)

[marmeladapk]

@jordens @hartytp So what's the consensus? Should I remove IC3 or should I just DNP J5-J7 and move CLK_EXT to Q3 of IC1?

And is there anyone else that should weigh in?

[jordens]

@marmeladapk I thought that was clear. Sorry if it wasn't. You can interpret the "Removes" from the items in the top post as "DNP" if you prefer. That's fine with me.

But just "DNP J5-J7" won't help. If we are doing any of this, I'd like the items marked v1.1/rj ticked off. That's more than DNPing a few components. It will involve re-routing some signals (e.g. driving J1-J4 from IC1 instead of IC3).

[marmeladapk]

@jordens If we drive J1-J4 from IC1 then and we DNP IC3 then we won't drive CLK_EXT with anything. I don't think that's intentional.

[jordens]

@marmeladapk

• (v1.1/rj) Connect backplane clock CLK_EXT_P/N to CKOUT2+- of IC2

[marmeladapk]

Ok, after some amount of discussion @jordens convinced me that shaving off 1 W and preventing possible isolation issues is worth deleting IC3.

We plan to add this heatsink to BoM.

[marmeladapk]

Remove the R98 path of CLK_OUT

I also removed R97 - no point in keeping it now.

[jordens]

Ack. Thanks!

[hartytp]

@jordens @hartytp So what's the consensus? Should I remove IC3 or should I just DNP J5-J7 and move CLK_EXT to Q3 of IC1?

I'd vote for completely removing it rather than DNPing it. If you make the other changes that @jordens is suggesting then this component no longer makes any sense.

Plus IIRC by removing it, you can simplify things by powering the other clock components from 2V5.

[hartytp]

@marmeladapk Will the heat sink be glued down? How robust is that? Would it be better to use one that mounts robustly onto the PCB like on Sayma?

Edit: I've found that self-adhesive tape isn't always strong enough for long-term use in a lab environment

[marmeladapk]

@hartytp

Plus IIRC by removing it, you can simplify things by powering the other clock components from 2V5.

It doesn't really matter if they will be powered from 3 V or from 2,5 V.

Would it be better to use one that mounts robustly onto the PCB like on Sayma?

Perhaps yes (@gkasprow ?) if we find one, but right now it would require significant rerouting near FPGA and I won't be able to finish it this week.

[hartytp]

Perhaps yes (@gkasprow ?) if we find one, but right now it would require significant rerouting near FPGA and I won't be able to finish it this week.

Okay, do whatever you think is best.

But, just my two cents, I'd be a bit nervous about using one of those. There is so much cabling here that could knock a glued on heatsink off. I don't want to be in a situation where pulling a cable somewhere in the rack results in the heatsink falling off and then the FPGA breaks! So, I'd ideally like the board either to not rely on a heatsink or to have one that's sufficiently robust for the application.

[hartytp]

Anyway, all changes look good to me.

[gkasprow]

Glued heatsink is robust. I tried to remove one from Sayma and did not manage. There are heatsinks that are attached to the FPGA edge so don't need mounting holes. I used them for bigger ARTIX chips.

[hartytp]

@gkasprow if you're happy with it then it's fine by me. I've probably used too much cheap self-adhesive tape and not enough of decent glue :)

Use whatever approach you think will work best.

[gkasprow]

Such heatsink I sent requires quite a lot of clearance around FPGA. If it does not fit, let's use glued one.

[hartytp]

@marmeladapk it's probably adding an inductor/ferrite between the SMPS and the new clock buffer LDO to make sure we kill high-frequency noise on that power rail.

[hartytp]

Also, if we're ever going to do it then let's tackle #317 while we're at it (should be pretty quick!).

[marmeladapk]

I also removed R131 and R133.

[hartytp]

One final comment (but not fussed if you don't want to make the change at this point):

• [ ] Should we float the SMA clock input by making that 0R resistor 100k?

My thinking here is that floating is generally what one wants for this. IIRC our only reason for not floating it originally was not wanting to have to fuss around with insulating washers. Since we're actually fitting an insulating washer in Kasli at the moment I can't see any reason not to float this input. Just requires changing one component value, so nice and easy to implement!

[hartytp]

FWIW, it probably would have been better to have floated the MMCX clock connectors on Kasli as well. If you can be bothered then let's do that.

However, if this is in any way hard to do (or if you just don't have time) then I don't mind you not bothering with it (it's not critical since these are generally only used for distribution over short distances within the rack, so ground loops shouldn't be too bad and, if this is a problem, one can use clocker)

[hartytp]

FWIW, one other thing we haven't looked at with Kasli is the robustness of the clock input. In particular, what happens when a clock is applied while the board is not powered (which always happens sooner or later!).

Quick calculation:

• Assume we apply a 2Vpp signal (this is the largest one typically uses) from a generator with a 50R output impedance
• Source voltage is thus 4Vpp=1.4Vrms, so short-circuit current is 28mA (actually a bit less that that, since current only flows once the voltage is greater than +-0.3V to forward bias the protection diodes)
• ADCLK chips can take 40mA current (presumably CW) so should be fine
• LTC6957-1 are only rated to 10mA input current, so may be damaged by applying a full-scale input when the board is unpowered (which is another argument against using them IMHO)
• I couldn't find a specification for those Si Labs chips, so who knows.

Anyway, this is probably fine but if we observe any damage then we could consider adding protection to the clock inputs (also applies to Urukul).

As discussed elsewhere, this is hard to do with diodes or protection resistors without degrading the performance of the clock chips, so adding one of the ADCLK buffers before the Si chip might be the easiest way of making the inputs more robust if there does turn out to be a problem...

[hartytp]

Other comments on that:

• If this does turn out to be a problem, adding diodes alone isn't necessarily enough, since the clock chip already has Schottky diode junctions at its inputs. So, we would probably need diodes and a current limiting resistor, placed between the diodes and the clock chip
• I don't want to add those components in, because it can be hard to do that without compromising the clock chip's noise performance. Doing this would probably need a bit of (time consuming) trial and error to determine things like maximum resistor value etc
• One thing I would be open to, if @gkasprow and @marmeladapk think it's worth it, would be to place a pad for back to back RF (fast switching, low capacitance) diodes (GND and the supply rail) and a current limiting resistor, but make the diodes DNP and start with the resistor 0R. That way if we do find a problem it can be fixed without changing the layout. We should probably also make sure to add a ferrite + capacitor filter between the diodes and supply rail (again, can be DNP for now) to avoid coupling noise onto the inputs.
• Maybe we should have also done this for Urukul v1.2, but hopefully it's not needed in any case.

What do you think?