Kasli v1.0 tests & errata

[marmeladapk]

To be changed for rev1.1

• Misc:
  • [x] R57 must be 300R
  • [x] Improve signal integrity for backplane LVDS lines (remove reference plane cut) (one cut between 2v5 and 12V0 seems unavoidable)
  • [x] Add two more USER_LEDs
  • [x] (v1.1/rj) consider adding a PCB-mount heatsink (one of the FPGAs died from overheating during testing)
  • [ ] Document power supply design parameters (#317)
  • [x] update
  • [x] and double check power budget after changes.
  • [x] add a PN for that clock balun on the schematic
  • [ ] Change revision to 1.1 on schematics
• USB:
  • [x] (v1.1/rj) Swap I2C with UART on FTDI chip (channel C doesn't have MPSSE)
  • [x] (v1.1/rj) fix direction of the IC19 (both channels)
• I2C:
  • [x] (v1.1/rj) add TVS on EEM I2C
  • [x] (v1.1/rj) add ID chip 24AA02E48T-I/OT, same I2C port as the Si5324
  • [x] (v1.1/rj) Connect reset of the I2C routers IC14 IC15 to the FPGA
  • [x] Add TCA9517 to Si5324 I2C
  • [x] Connect IC10:A1 (I2C_SW_RESET) to one of the unused BDBUS lines (BDBUS7 or BDBUS5 if you can) and not to CDBUS4.
• Backplane:
  • [x] (v1.1/rj) Inconsistent power connector PN 712RA vs L712RA (712RA is the correct one) (it was just wrong footprint name, PN was correct; PK)
  • [x] (v1.1/rj) Remove second power connector J17.
  • [x] (v1.1/rj) Populate the backplane connector J20.
• SFP:
  • [x] Move SFP LEDs closer to panel (same as was done with USB)
  • [x] Free up six pins on the FPGA by having MOD_DEF1 and MOD_DEF2 routed through the I2C switch only and not additionally directly to the FPGA.
  • [x] Remove the I2C level converters IC10 on those SFP I2C busses.
  • [x] (v1.1/rj) Connect RATE_SELECT1 (pin 9 of the SFP) to the FPGA
• Clock distribution
  • [x] (v1.1/rj) Remove the 50 MHz oscillator OSC1. Not needed as the 125 MHz GTP input can be routed to fabric.
  • [x] (v1.1/rj) Remove IC3
  • [x] Remove the CLK_SEL logic of IC3 (free up FPGA pin)
  • [x] (v1.1/rj) Remove J19 and related components
  • [x] (v1.1/rj) Remove J5-J7 and related components
  • [x] Remove the R98 path of CLK_OUT (IC1:Q2 to the front panel SMA)
  • [x] (v1.1/rj) Add an LDO for the power supply of IC1 and IC2 (also worth adding a ferrite/inductor between the SMPS and the LDO to kill high-frequency spurs).
  • [x] (v1.1/rj) Connect four MMCX (J1-J4) to the Q2, Q2B, Q3, Q3B outputs of IC1
  • [x] (v1.1/rj) Mark J2 and J4 (on the negative Q2B and Q3B outputs) clearly on the PCB
  • [x] (v1.1/rj) Connect backplane clock CLK_EXT_P/N to CKOUT2+- of IC2
  • [x] (tph) change 0R resistor to 100k to float SMA clock input
  • [x] (tph) consider floating MMCX connector grounds in same way as SMAs (only if you have time!)
  • [x] Change the transformer TR1 to TC2-1TX+. No changes to layout. Footprint and package are compatible. But pinout is rotated 180 degrees.
  • [x] DNP C13, C9
• Layout:
  • [x] Either teardrops everywhere or nowhere. Right now there are teardrops on roughly half the pins.
  • [x] Set up consistent rules
  • [x] Properly clean up polygons
  • [ ] Don't do arcs where they are not needed.
  • [x] Remove weird teardrop on FTDI
  • [x] Connect PCB edge GND stitching vias to top and bottom GND pour. Stick with @gkasprow recommendation on via density.
  • [x] Check XADC connection to components
  • [x] Replace C49 with a 0R (connect VREFN to XADC_GND) to use internal reference (connected without jumper, PK)

[sbourdeauducq]

If this does turn out to be a problem, adding diodes alone isn't necessarily enough, since the clock chip already has Schottky diode junctions at its inputs. So, we would probably need diodes and a current limiting resistor, placed between the diodes and the clock chip

What would be the problem - excessive current through those diodes? Then what about adding a current-limiting resistor in series with the power pin of the clock chip (+decoupling of course), a diode, or a MOSFET-based switch? (semi-related: https://hackaday.com/2011/05/20/using-an-avr-as-an-rfid-tag/)

[jordens]

I'm ok with floating the SMA connector if that's risk free. Why does no scope or spectrum analyzer manufacturer seem go that way e.g. on their reference inputs/outputs?

I'd have preferred to have a non-transmission line galvanically isolating bal-un instead of that TLT (TR1). We should change that for Kasli v2.0. I overlooked it twice now...

@marmeladapk Schematics look good!

[hartytp]

I'm ok with floating the SMA connector if that's risk free.

The major risk I'm aware of here is that if one attaches a connector where the ground is at a significantly different potential (e.g. floating with ESD) then that now doesn't have a low-impedance path to circuit ground. There is a similar issue with using isolation transformers.

This is generally not an issue, but in a few cases it can lead to ESD-like damage of components (e.g. when ground potentials rapidly change relatively to each other). For the kind of lab work we're going to do I wouldn't anticipate this being an issue. Still worth being aware of.

Why does no scope or spectrum analyzer manufacturer seem go that way e.g. on their reference inputs/outputs?

I can't give you a definitive answer on this (maybe Greg or Samer can) but a few comments:

• This is only an issue for measurements where low-level close-in phase noise matters (as, that's usually the only thing that ground loops on the clock inputs do). Most scopes and SAs aren't low enough noise for this to be a problem
• On scopes the grounding is typically a mess anyway as the inputs aren't floating (too high frequency for isolation amplifiers and need to be DC coupled), so it's not really worth worrying about ground loops in the reference input
• Things like our R&S FFT analyser (and also the SRS model) do support floated inputs for the signals at least to avoid this kind of ground loop problem.
• In general, one can always add an external inside-outside DC block so often manufacturers just rely on one doing that if needed.

I'd have preferred to have a non-transmission line galvanically isolating bal-un instead of that TLT (TR1). We should change that for Kasli v2.0. I overlooked it twice now...

IIRC the reason we decided not to do that was that the flux-cored baluns run out of steam rapidly above 1GHz. That means that you don't get a very nice square wave clock through them, so they're not great from a phase noise perspective.

For Kasli this may not be such an issue (since we don't care about phase noise too much), but I guess we figured that it was easiest/best to just do the same thing we're doing elsewhere (which would still be my preferred option).

[hartytp]

What would be the problem - excessive current through those diodes?

Exactly. But, only when a clock signal is applied while the clock chip is not powered.

When those things are properly biassed they're quite hard to kill. The diodes don't begin to conduct until the Vpp exceeds Vcc + 2*0.3V =3.1Vpp=1Vrms=+14dBm in our case. After that, the ADCLK chips can take 40mA through each diode. So, you're not likely to damage that with any standard signal generator.

When no supply is present, the positive-going part of the input signal sources current to the positive supply rail. That can be quite messy. But, in general, the supply rail charges up until enough circuitry powers up to provide a path to ground for this current. Usually to a volt or two. In rare cases, reverse biasing the LDO that normally supplies this rail can cause it to go into lockup (in which case, a diode is required across the LDO output).

Then what about adding a current-limiting resistor in series with the power pin of the clock chip (+decoupling of course), a diode, or a MOSFET-based switch? (semi-related: https://hackaday.com/2011/05/20/using-an-avr-as-an-rfid-tag/)

If you add resistance in series with the power rail, you also need to add it in series with the ground pin. These chips draw quite a lot of current, so to put enough resistance there to provide adequate protection can start to give a decent voltage drop across it, which is undesirable.

Also, FWIW, to get good close in phase noise, the power supply/ground needs to be low impedance all the way down to DC, so the decoupling capacitors aren't enough.

Not sure where you mean to put diodes? Between the connector and the clock chip? That's not a bad option, but:

• The clock chip already has Schottky diodes in it, so they will turn on at roughly the same time as the external diodes you add. So, it can be hard to ensure that the external diode actually draws the current away. To solve this, it's usual to add a resistor before the clock input to push more current through the external diode than the internal ones. The problem is that that external resistor can reduce the phase noise performance of the clock chips so some trial and error is needed to make sure that things actually work
• Since the signals we're applying are generally quite close to the supply rails, the external diodes can start to behave like non-linear capacitors to ground and cause SI issues. They can also couple noise from the supply rails to the very low noise clock inputs. Again, these issues can be overcome with good design, but it's easy to screw things up and kill the noise performance of the IC if one doesn't take due care and, again, some prototyping is generally needed.

Not sure what you mean by a MOSFET, but, again, there are the issues of ensuring that any components you add don't degrade the clock chip's performance, and thinking about how they operate when no supply is present.

---

This is all messy enough that it's probably not worth worrying about it for now and only fixing it if it turns out to be a problem in practice.

[hartytp]

One final thought about this: shall we populate the transformers on the clock inputs by default?

• they are really cheap, so they won't inflate the BOM cost noticeably. We'd only need to change a few population options, so it's really quick and easy to implement
• Standard oscillators like a Wenzel oscillator often put out +13dBm (2.85Vpp) or so. If we drive the clock chip single-ended then that's too much (they can only take something like 2Vpp). Adding the transformer reduces the voltage across each pin to 2.85Vpp/sqrt(2)=2Vpp (well, a bit less after 1dB or so of loss) so this becomes fine.
• Allowing bigger signals improves the phase noise performance when driven from a sinusoidal source
• For a similar reason, having the transformer makes these inputs more robust to having too much power applied.

I don't feel strongly about this, just raising it.

With all of this, @gkasprow and @marmeladapk just do whatever you think is best (or whatever you have time for).

[sbourdeauducq]

Not sure what you mean by a MOSFET

In series with the clock chip's power, just like the diodes, without the drop-out voltage but needing more complex circuitry to turn it on/off with the board (like a synchronous rectifier). Then the rectified clock could only "power" the clock chip, which could reduce current to acceptable levels.

[hartytp]

That doesn't sound like a god idea to me at all.

• You still have to make sure that your FET switch functions properly when not biassed (taking into account all parasitic diode junctions)
• Unless you float the entire IC (which is not something I'd consider), you'll still have return paths via the output biasing resistors, digital pins etc. so that won't work.

Let's not overcomplicate (and potentially break) the design to try to solve something that we don't know is a problem -- hopefully, the ESD diodes on that Si chip are relatively beefy and can take the current that a standard +13dBm clock source can throw at them. If they can't then I still feel that diodes and a current limiting resistor on the input are the best way to protect the clock chip (although, that would probably need some prototyping to get right). Adding some pads for DNP/0R components was just intended as a quick way of ensuring that we can add extra protection in the future if needed without changing the layout, without actually adding them now. Edit: but, I'm still not sure that even that is an entirely worthwhile thing to do (I'll leave the decision about how to handle all this to @gkasprow and @marmeladapk).

[sbourdeauducq]

You still have to make sure that your FET switch functions properly when not biassed (taking into account all parasitic diode junctions)

Sure; worst case use a small solid-state relay, that can't go wrong. Anyway, agreed we should not look too much into this unless it's a real problem.

[jordens]

I'd be fine with that as long as it doesn't delay anything and as long as we change them from TLTs to isolating bal-uns.

[hartytp]

I'd be fine with that as long as it doesn't delay anything and as long as we change them from TLTs to isolating bal-uns.

With populating the baluns you mean?

Out of curiosity, why do you want to go for a flux-cored balun? For Kasli I'm not particularly fussed, but it does seem like a shame to use something that can't pass RF and I can't see an obvious advantage in terms of isolation (why do this galvanically rather than the current capacitive coupling)? Also, if you want to use the balun for isolation then you'll need to change the layout a bit IIRC since one half of the balun would need to be on the "ground island" with the connector...

[jordens]

Yes. For 100/125/150 MHz I'd like a DC isolated bal-un instead of trifilar one currently in the schematic. And yes, in that case make it a ground island. I like them because i would guess that breakdown voltage is higher than with capacitors. And AFAICT this (and not capacitors) is what people use in the industry to provide galvanic isolation. I expect them to have good reasons. What is a "flux-cored" balun? AFAIK they all have flux in the core.

[marmeladapk]

@jordens Are you fine with some pin swapping? Either way there will be new pinout. ed: and @sbourdeauducq

[jordens]

@marmeladapk depends a bit where. But generally yes.

[marmeladapk]

@jordens Only control lines for SFP.

[jordens]

Oh yes. Feel free. Same goes for LEDs.

[hartytp]

What is a "flux-cored" balun? AFAIK they all have flux in the core.

Flux cored/magnetic cored/flux coupled is relatively standard terminology IIRC. But, let's not get into a discussion about terminology here...

I like them because i would guess that breakdown voltage is higher than with capacitors.

It certainly is, and AFAICT that's the reason they're usually preferred for lower frequency work.

FWIW, in our case I don't think the breakdown voltage is such an issue, as the clock source is unlikely to have a ground potential that differs from Kasli's by a significant amount. Also, we're capacitively coupling RF signals in various other places, so if this is a problem for Kasli then we need a rethink elsewhere and, if it's not an issue elsewhere then there's no point doing something different on Kasli.

As I said, other than it being nice to use the same transformer everywhere, the only real downside I can see of using the magnetic balun on Kasli is that it won't be able to pass nice square waves, which could make the phase noise performance a bit worse. But, that isn't going to be an issue as Kasli isn't ultra low noise anyway.

[hartytp]

And AFAICT this (and not capacitors) is what people use in the industry to provide galvanic isolation.

FWIW, capacitive inside-outside DC blocks are also quite common, particularly for microwave work (where they're the only option).

[jordens]

Oh sure, "flux-coupled". In the datasheet you cite, I can't find the term "flux-cored". I mean flux-coupled when I say "a DC-isolated transformer that is not a transmission line transformer". Yes, we are capacitively coupling in several places on the board, just that we are not doing that with the ground/shield and we are not doing it yet between devices. That would parallel what industrial devices seem to be doing. We won't have a noise problem with flux-coupled transformers on Kasli.

[hartytp]

Right on Kasli this is fine.

For urukul and Sayma it probably wouldn't be, which is why we have the pads for capacitivrly coupled grounds.

[a-shafir]

@hartytp it is a good question about "ground loops" in the labs. I think there is no real way to improve much on RF with coax/BNC in case if the connecting devices are not grounded nearly perfectly. Breaking DC (and some low freq range) loop is good for preventing dangerous currents via PCB. But IMHO need to be careful not to burn the sink resistor when it small or not to burn the following circuit with high voltages in case of some ESD or even DC dropped to the BNC/shield.

For mostly unbalanced i/o design like we have here the best approach is to have metal front panel and all the signals are referring to it so the back plane of the boards is "floating" to avoid currents trough the PCB(s). In case if the signals indeed can be affected by wide range device to device noises and parasitic currents need to switch somehow to balanced. I don't know if it sounds a crazy idea to use SFTP CAT-7A cables or similar approach?

Also for the clocks >10MHz will work 100pF or less capacitor for the signal and ground lines. It can easily stop DC, low freq noises and relatively high voltages as well . So unless we will put high RF on the input it will be safe and also will not affect the RF clock performance.

For the balun IMHO the main question how good the matching of the input (output) coax/BNC with the PCB diff circuit. For the critical nets it shall be simulated layout or tested with the network analyzer IMHO.

[dhslichter]

I think if you want sharp rise times with DC ground isolation and DC center pin isolation, inner/outer DC blocking caps seems like the most sensible way to go. You don't care about frequencies below ~100 MHz, so finding appropriate component sizes should be fine. The "standard"/non-transmission-line baluns usually tend to give up the ghost above ~400 MHz or so, which is why transmission line baluns were selected for this initially.

[marmeladapk]

Rerouting is almost done. Tommorow I'll improve power distribution and I'll run DC simulations.

[marmeladapk]

Simulations done. I must say, that removing back power connector made routing signals and power much easier. Upper 1/3rd of board was taken entirely by planes connecting to power connectors in middle layers.

Is that note clear enough?

[marmeladapk]

@jordens @hartytp Just to be sure: clock fanout does not need to be length matched now?

[marmeladapk]

I pushed RC version.

I can't remember what's our current stance on generated files, so didn't put PDF and BOM yet (seems that output folder dissapeared from Kasli, was it intentional?).

There's a bug with Altium, it didn't put 3d models for new parts, will solve that later.

Speaking of bugs, behold Altium generating teardrops:

[jordens]

Thanks! The silkscreen markings are good. No length matching needed. I don't know who removed the files. But that's ok. Just tag a RC release and upload the files to the release.

[hartytp]

@marmeladapk Can you post the PDF to a RC release please so we can have a look over it? (Or, you can just attach it to a post here if that's easier).

[hartytp]

@gkasprow @marmeladapk One thing about the releases: please can you upload the PDF files directly to the releases as well/instead of putting them in the zip/7z file (as was done for many of the releases currently posted). It helps to have direct access to the shcematic PDFs without having to download and unzip a larger archive! Thanks.

[marmeladapk]

KASLI.zip

(edit by @jordens): RC files: Kasli/v1.1rc1

[marmeladapk]

@hartytp @jordens But how should we do it in the long run? Adding tags for each commit doesn't seem sustainable. I can upload PDF zip to comment commit.

[jordens]

@marmeladapk excellent. Very clean and tidy! I couldn't find anything I don't like on first glance. Will check more after lunch. Commits and tags are diffferent. You should do commits when you change anything or when you want somebody else to look at the files in source form (using the viewer or the editor). When you want to do a release or release candidate, you'd tag the commit, create a release, and upload the files (not as a zip) to that release. That is much better manageable than PDF zips in comments on issues or comments to commits. We have stable and semantic links to those files. That also makes space usage much more benign. We can delete/archive old releases or the data for releases and it is not a burden.

[marmeladapk]

@jordens Some arcs are still there. ;)

So I can assume that for incremental changes you'll use Altium Viewer?

[jordens]

• (!) Would have been nice to do the I2C switch reset also (or predominantly) from the FTDI interface to the I2C tree. That's what's been bugging me so far (testing EEM I2C trees over USB, also for fabrication/QA). And the two switches don't need separate resets. Sorry, I wasn't clear on those two things.
• (!) On the SFPs, RS1 (Pin 9), is there a way to add a ~2k2 in series after IC13:A2? The MSA says that the controller needs to be fine with that pin shorted to GND (VEER) by the SFP module. Maybe the 245 doesn't like it too much. OTOH this is not too critical as we expect to only drive that pin high when there is actually a module inserted that listens to it.
• Yes. For incremental things I'll try to use a viewer. But when am I looking at incremental changes that are not RC or releases?
• I've seen the arcs ;)
• On the underside two of the I2C TVS are not populated. I guess that's Altium's fault.
• It would have been nice if the three USER_LEDs had been towards the front panel. But it's not worth the hassle now.

[marmeladapk]

@jordens

I2C switch reset also (or predominantly) from the FTDI interface

Pyftdi (which, I assume, we want to use) doesn't support GPIO and I2C right now, so it would have to be connected to UART or 4th unused port. Also no point now in connecting the two resets together.

This mode is mutually exclusive with advanced serial MPSSE features, such as I2C, SPI, JTAG, … If you need to use GPIO pins and MPSSE interface on the same port, you need to use the dedicated API. For now, this shared mode is only supported with the SPI API. source

[jordens]

Fast I2C over MPSSE is nice but not critical. It works fine at lower speeds with GPIO. I'd rather like to be able to reset the switches using GPIO. You can always switch to GPIO mode, do the reset, and then switch back to MPSSE mode. But anyway, this is minor. (1) I don't expect to need that anymore now that we have the TVS on there and (2) we can reset over JTAG/FPGA.

[marmeladapk]

On the underside two of the I2C TVS are not populated. I guess that's Altium's fault.

That's a bug, 2V5 LDO and SATA also don't have 3d model.

It would have been nice if the three USER_LEDs had been towards the front panel.

For me that's not a problem, but we would have to change panels. @gkasprow ?

[marmeladapk]

You can always switch to GPIO mode, do the reset, and then switch back to MPSSE mode.

Good point.

[jordens]

• Yes. Don't bother moving the LEDs if the panels are already in the works.
• I really like the LT3045!
• (!) If there is no major rerouting fallout it would be nice to have the I2C_SW1/2_RESET on CDBUS4. They really don't need to be separate.
• (!) But do increase the pull up resistance on those I2C_SW1/2_RESET pins to 10k if you need to pull them up to 3.3V or pull them up to 2.5V only: the FPGA is at 2.5 V and that pin should not see 3.3V at 2k2 impedance.
• Good work keeping the GTP diff pairs further apart!
• I would have put the MMCX shield grounding component groups like C43/R29/C44 closer to the center conductor capacitor C20 (i.e. on that side of the MMCX connector where the input signal is located). That would keep the return current loop smaller and decrease emission/coupling. But that's a minor tweak as well.
• I see that there are SATA cables wich connectors angled any of the four possible ways but there seems to be one preferred bend style. If we ever do boards with SATA connectors in tight spaces, rotate the connector so that the most widely available cable exits into the preferred direction (to the back).

Other items from our check list

• There isn't much clearance around the FPGA (large capacitors) for larger heat sinks but I'll accept that.
• EMI ground via stitching at the PCB edges?

[hartytp]

• [ ] @marmeladapk TR1 still needs to be populated and surrounding components changed.

[hartytp]

Overall though, looks great!

[hartytp]

In the power budget, are you double counting the power draw from the clock buffers, since this is listed under the 2V5 column and under the 3V3 column?

[jordens]

Ok. I am done reviewing. Let me know what you think about the four items marked (!) above. Great work!

[hartytp]

@marmeladapk Where are the PCB mount clips that hold the heatsink onto the board?

[jordens]

@hartytp IIRC we said there is no space. Needs to be a glued heat sink.

[hartytp]

@jordens you're right, I'd forgotten about that. Thanks for reminding me.

[marmeladapk]

@jordens

I2C_SW1/2_RESET on CDBUS4 increase the pull up resistance on those I2C_SW1/2_RESET pins to 10k

And disconnect it from FPGA or keep it? If we're keeping we should add resistor in series. Connecting RESET to CDBUS4 won't be a problem.

Good work keeping the GTP diff pairs further apart!

I didn't touch them ;) I only removed lines in-between them and routed them where unfiltered P12V0 ran in v1.0.

put the MMCX shield grounding component groups like C43/R29/C44 closer to the center conductor capacitor

ACK.

rotate the (SATA) connector

ACK.

@hartytp

TR1 still needs to be populated and surrounding components changed.

ACK. I didn't see a clear conclusion from your discussion, that's why I didn't change it.

In the power budget, are you double counting the power draw from the clock buffers

No, total power draw is only calculated from 3V3, 2V5, 1V5 and 1V0.

[jordens]

@marmeladapk

• I am fine with having I2C_SW1/2_RESET go only to the FTDI chip and not to the FPGA. Situations where the FPGA needs to talk I2C and the I2C tree is broken must be fixed by some other means. That also removes the issue of pull-up strength on those signals. And merging them is fine as well.
• On the SATA connector. Really don't bother rotating it if that poses any hassle. If you do rotate it, I think the best rotation would be 90deg counter clockwise with the notch pointing down.
• I am for populating the bal-un as well. I like bal-uns. Let's grant us that luxury.

[jordens]

For the SATA connector it doesn't matter wither it is 90 degree CCW or 90 degree CW. Both seem to be equally present.

[marmeladapk]

@jordens Rotating clockwise is easier.

I need to add an additional buffer between FTDI and I2C switch for this reset signal.

[marmeladapk]

EMI ground via stitching at the PCB edges?

Not sure if this is necessary because all high-speed signals are routed in the middle of the board (X/Y/Z) (except maybe SATA), however I have no experience with so @gkasprow comment is needed.