Kasli 1.2 WR support

[gkasprow]

We have just a few free BGA balls which are not sufficient to support WR-like clocking. we have some spare pins: 2.5V bank: W19 (clock input), F21, T3, Y17 1.5V bank: E3 But we don't need all SFP control lines like rate select, LOS, FAULT. We don't use them and can re-route to I2C extender. In this way we get another 10 pins or so. @jordens any objection? In all my new designs I route all SFP control lines to dedicated per SFP I2C extender so they don't consume valuable FPGA lines. They don't cosume I2C switch ports either because SFPs are already connected to I2C. In this way we would remove bunch of level translators and replace them by 3 8bit I2C extenders which enable SFPs be default.

[jordens]

We are using the SFP lines. But I am OK with moving them somewhere else. I don't know whether I2C or SPI would be better here. And I am unclear how the software/gateware support will be implemented and who will do that. Also there are a few things that should be wired to the FPGA, like the I2C switch reset.

[marmeladapk]

@jordens https://github.com/sinara-hw/sinara/issues/349#issuecomment-360517401

[jordens]

Sure. But since we are now doing a big revision that will lead to a bunch of incompatibility and we just freed up a lot of pins, it's fair to revisit that perspective and enable resetting the I2C switches from the FPGA. Right now we don't make much use of the I2C bus. That is likely to change. If e.g. the FTDI interface leaves the I2C bus stuck and forgets to reset it, the FPGA can't even boot. As a sidenote on the I2C front I have had problems with talking to SFP modules. It seems a bit fragile but it could also be a problem with the modules. One thing that I would consider is increasing the pullup resistances on the downstream switch ports to e.g. 10k-20k. Otherwise the pullup impedance is 1k in use or even lower when the tree is deeper than one switch level. That might be a bit low. AFAICT there is no situation where a bus segment would have a too high impedance or the entire bus would have too high impedance when in use.

[gkasprow]

@jordens which ones are you using? Lines like Rate Select of TX Fault are not supported by most SFPs We need just a few of them.

[jordens]

LOS is supported and we use it to drive the LED. We are using the cheap ones from FiberStore in different incarnations but also a bunch of genuine Aruba/HP ones (SFP-SX).

[hartytp]

cf https://github.com/sinara-hw/meta/issues/15

[hartytp]

Continuing from https://github.com/sinara-hw/Kasli/issues/5#issuecomment-490073738

(@marmeladapk) Do we want to implement WR CDR as in Sayma and Metlino?

Side note: should we stop calling this WR, since we're not implementing most of the WR stack (PTP etc)? Shall we just call it CDR PLL or something generic? Or call it WR PLL to indicate the history?

I definitely want a DCXO PLL on Kasli v2.0 (making sure we have resources for this was the main reason we funded Kasli 2.0). Let's copy Sayma, but leave the DFF off.

(@jordens ) WR CDR once the first iteration has flown successfully beyond bread boards and the Si5324 should be scrapped then.

I'd strongly prefer to keep the Si5324 on for v2.0, with the aim of removing it in v2.1 Ideally, I'd like a mux to select between Si5324 and DCXO PLL, but solder jumpers are fine too.

1. I don't want to rely on Sayma for the DCXO PLL development (issues with limited hardware availability for testing, likely to be delays in getting working boards ready for testing, complex design to test, potential issues with Ultrascale FPGAs etc). Ideally, we would have demonstrated the DCXO PLL on Kalsi before freezing the Sayma design, but that wasn't possible. I don't want to put off Kasli v2.0 until we've been able to demo the DCXO PLL working reliably on Sayma v2.0
2. I don't want to remove the Si5324 until we've done a few months of field testing with the DCXO PLL. I don't expect any major issues, but best to be safe.

I think it's fine to support both paths in v2.0, with the aim of moving on to v2.1, which will only have one option, fairly quickly.

[jordens]

Is there really no way we can reassure everyone sufficiently to decide on one CDR solution instead of dragging multiple mutually exclusive approaches for the same problem around? In retrospect I have the impression that each time we did this it was relatively easy to predict which option would be the only one used in practice. In addition to the mux/jumper at the output, you also need fanouts or solder jumpers on each of the two inputs to either CDR solution or on the only input if that ends up being after a SMA/FPGA mux. The latter changes the usage of the Si5324 inputs. I would really like to get a full view of the effect of having both solution on board area, pinout complexity, pin shortage, power supplies, added fanouts/muxes.

[marmeladapk]

Assuming that we add two Si549:

• We would need to switch from ADCLK944 to 948 to switch from main DCXO to Si5324 (IHO that a good thing, Kasli is the only board using 944)
• We have a free MRCC input on the FPGA for helper DCXO
• We would need to add two I2C buses straight to the FPGA - we have free pins for that
• I would put output enables on I2C extenders, these aren't critical signals (just for the sake of easier routing)
• We would need a beefier LDO to power Si5324 and Si549. Both Si549 and 570 support 2,5V supply (though Si570 has a different part number for 2,5V version)
• Considering board area - it might be a bit tricky to put those chips in the same area as Si5324 so I think I'll place them on the right side of MMCX connectors

[hartytp]

Is there really no way we can reassure everyone sufficiently to decide on one CDR solution instead of dragging multiple mutually exclusive approaches for the same problem around?

Let's see. On the hardware side, the DCXO/WR PLL has been quite well tested using evaluation boards and a KC705: we've checked it locks reliably (well, say 10 out of 10 times); we've used a scope on persist to verify that no cycles are slipped over days; we've measured phase noise; we've measured long term phase stability with an interferometer (mixer). It all seems good. Moreover, we've done tests both with unloaded and with heavily loaded FPGAs.

However, the cautious side of me would still like to spend a few weeks testing this on a live Artiq experiment before we commit to it as our only CDR scheme. For the reasons spelled out above, I don't think Sayma will be a good test bed for that and would much prefer to use Kasli. We could put only the DCXO/WR PLL on Kasli v2.0 and test like that, however if we do find an issue with the design those boards will be effectively useless.

On the software side: while we have working POC code, it's not close to being ready to merge into ARTIQ (although @WeiDaZhang is working on it). So, we either need to put both the SI5324 and DCXO on Kasli v2.0 or accept that we won't be able to use Kasli 2.0 until the code is ready for ARTIQ integration. I'd prefer to press on with both solutions and switch to the DCXO when the code is tested and working.

If @marmeladapk feels that keeping both the SI5324 and DCXOs will cause SI/PI/layout issues then that would change things. But, if it's not a problem to have both then that's what I'd like to do for the next version. We can finally drop one in v2.1.

[marmeladapk]

@hartytp By default CMOS DCXOs are mounted. Do we plan to use LVPECL variant at all? I couldn't find stocked LVPECL variant anywhere. LVPECL and LVDS variants are shown to have nearly the same phase noise. If we'll use only CMOS and LVDS then I could simplify schematic for helper DCXO and route both output signals directly to the FPGA and (in CMOS case) terminate _P line to ground.

I also need to add buck from 12V to 5V and then LDO from 5V to 3,3V, as ADCLK948 can only be used with 3,3V.

There will be plenty of space for this since I remove one SFP cage.

[jordens]

• The ADCLK948 is pretty hungry (288 mA at 3.3 V versus 139 mA at 2.5 V, both static).
• If the external SMA then only goes to the FPGA and to the SI5324, do we need another fanout? If we go for capacitors then we can do that just as well at the input to the downstream fanout and don't need the ADCLK948.
• Does the WR PLL also work well with 100 MHz or 10 MHz external reference? Since 100 MHz is the dominant use case so far it should definitely be verified.

[marmeladapk]

@jordens I wanted to change ADCLK944 to 948 to have clock input selection, so it's not another fanout. External SMA only goes to Si5324.

[jordens]

Sure, I understand that. Is that worth the increased power consumption? Not a path from the sma to the fpga if the dcxo is used means the external sma becomes dependent on the si5324 blocking it's removal.

[hartytp]

External SMA only goes to Si5324

Not a path from the sma to the fpga if the dcxo is used means the external sma becomes dependent on the si5324 blocking it's removal.

For the WR master, we need an external coax (MMCX/SMA) connection to the FPGA.

Can we get away with only routing the clock to the FPGA and then use the FPGA as a mux to route it to the Si5324 if necessary?

[hartytp]

Does the WR PLL also work well with 100 MHz or 10 MHz external reference? Since 100 MHz is the dominant use case so far it should definitely be verified.

We've verified 100MHz and 125MHz so far. I'm not aware of any limitation that would make 10MHz worse (assuming clean edges), however it needs different parameters and we don't currently have a nice design tool to determine them (it's on the to do list).

[hartytp]

@hartytp By default CMOS DCXOs are mounted. Do we plan to use LVPECL variant at all? I couldn't find stocked LVPECL variant anywhere. LVPECL and LVDS variants are shown to have nearly the same phase noise. If we'll use only CMOS and LVDS then I could simplify schematic for helper DCXO and route both output signals directly to the FPGA and (in CMOS case) terminate _P line to ground.

I'm not aware of any reason to use LVPECL since the LVDS specs are fine. We used LVDS in our tests. LVCMOS isn't specified as well an we haven't tested it yet. But, if I had to guess, I'd say it's probably fine too (generally, there is another PLL after the WR DCXO like the HMC830 to clean up the broad-band noise).

[hartytp]

Does the WR PLL also work well with 100 MHz or 10 MHz external reference? Since 100 MHz is the dominant use case so far it should definitely be verified.

@jordens most of the tests @WeiDaZhang has done so far were with a 100MHz reference and 125MHz DCXO frequency. He used an FPGA PLL (MMCM?) to convert the 100MHz to 125MHz before feeding it to the DDMTD DFF. Interestingly, he did not observe any degradation in the noise spectrum or long-term stability due to the FPGA PLL (even under heavy load). So, this seems to work fine.

I guess the close-in noise for those PLLs is fine, and the WR PLL takes out all the high-frequency noise with an aggressive loop filter. In any case, it seems that this will be fine for the vast majority of use-cases.

To get an RTIO frequency significantly different to 125MHz one has to re-optimize the loop-filter and potentially change some register values in the DCXO. I'd guess that the range 100MHz to 150MHz can be supported decently without retuning the loop, but @WeiDaZhang would need to confirm this.

[marmeladapk]

I pushed updated schematics, see if this is what you had in mind. (Si549 OE needs translation)

[hartytp]

@WeiDaZhang

[hartytp]

I pushed updated schematics, see if this is what you had in mind.

Can you post a pdf please?

[marmeladapk]

KASLI.PDF

[hartytp]

Thanks! That looks good to me.

[hartytp]

@marmeladapk are we sure it's okay to pull the NC pins on the Si549 high?

[hartytp]

Wide have both reviewed the WR/clocking and are happy (modulo my above question). Assuming @sbourdeauducq / @jordens can't see any issues, I think we're good to go.

@marmeladapk what's the timeline for finishing Kasli 2.0? I'm keen to get the next batch of boards off asap as we're out of hardware in the lab.

[hartytp]

@WeiDaZhang has produced an FMC with DCXOs etc on it for prototyping the WR interface on a KC705. Can you post the schematic here, please?

The test board has DFFs on it, so we can look at the loop performance both with the DDMTD DFFs inside the FPGA and with them externally. The broadband noise is about 6dB lower for the PLL external DFFs than for the FPGA DFFs. That suggests that we can have a somewhat broader-bandwidth/lower-noise lock with the external DFFs, which is nice, but not a game changer. IMHO that's not sufficient grounds to use the DFFs on Kasli yet since I'd prefer to optimize for simplicity over squeezing out the last few dB of phase noise at this stage.

Still to do: measure close-in phase noise, temp-co and long-term stability with both approaches.

[hartytp]

One interesting thing that did come up is that the Si549 close-in phase noise seems to degrade with increasing DCXO temperature. This figure shows the open-loop phase noise of the DCXOs on Weida's test boards. Red is with a large heat-sink on the DCXO, orange is with a small heat-sink and yellow is no heat sink.

The difference is all basically in the 0.1Hz to 10Hz region. That's a bit close to the typical loop bandwidth we want to run at. It can be dealt with to a certain extent by using an aggressive (probably >3rd order) loop filter. But was unexpected and needs thought.

I don't think the DCXO thermal grounding was particularly good on Weida's board. Next week, he'll put the test setup in an oven and measure the open-loop phase noise for a couple of different DCXO case temperatures. (@WeiDaZhang NB we should ask SiLabs if this is expected behaviour on the forums once we have that data).

[marmeladapk]

@hartytp Close this issue if there's nothing else to do here.

[hartytp]

@weida I think we now agree that there is no point having an ext dff since the transceiver PLLs dominate. In which case, this design is optimal, right?

[marmeladapk]

@hartytp Do you prefer to have Si549 on top, shielded from airflow by SFP cage and panel or on the bottom with no obstructions? Package height is 1,35 mm.

[hartytp]

My feeling is that Si549 on top shielded from airflow is probably best. I'm considering both thermal fluctuations and the chance of damaging the cards (it's nice to keep bulky components on the top).

What do you think?

[jordens]

I would actually put it as far away from fluctuating and strong thermal sources/sinks: away from fan, clock fanout, FPGA, SFP, power supplies, SDRAM. Their conducted thermal coupling is likely stronger than the indirect convective coupling.

[marmeladapk]

Uh oh. Well, we can't have everything. This excludes the whole board ;) I'd also like to keep main dcxo near clock fanout so that the clock doesn't pick up noise from travelling across the board. Perhaps above I2C fanouts and beneath JTAG connector? This seems like the most quiet place + away from everything @jordens mentioned (esp SFP cage may be dumping a lot of heat). But Si549 won't be shielded from airflow then.