SYS/RTIO clock merge

[Spaqin]

So for Kasli we'd like to use the RTIO clock (output of Si5324) as SYS clock, avoiding clock domain transfers, with a simple bootstrap clock before the bootloader changes the clock. It seems possible in general, but comes with its own challenges - e.g. no support for 100/150MHz clocks (unless we set the MMCM/PLL parameters to match these frequencies? change in gateware + software, essentially, different build).

Marking this PR as WIP as I've been having difficulties with it and it's nowhere near finished. There's also kc705 support in there to be done as well, and I suppose it should be eventually merged in tandem with ARTIQ part.

Few things that I encountered though that I would like an explanation for:

• why is the div2 clock used in MMCM/PLL, rather than the 125MHz one?
• why was sysclk running at ~114MHz instead of 125 (see clkfbout_mult_f @ 14.5 instead of 16)? this looks like an oversight?
• Kasli QPLL took the buffered 125MHz from the reference directly (see line 209/256 of kasli.py) - I cannot use the "chosen clock" from BUFGMUX, as Vivado complains about unbuffered output, and sending the BUFGMUX output through BUFG/BUFH/BUFR resulted in unroutable design. Which should it use, is there a way to choose a clock for it too?

At the point of making this PR this method doesn't meet timing constraints and I can't seem to find a way to improve it (e.g. putting the si5324 clock through MMCM and then to BUFGMUX makes it even worse). Unless there's a different, better way to switch between the two clocks (probably is). I managed to run it at some point, but more by luck I suppose. I get garbage output on serial most of the time, possibly due to unmet timing, quite badly too.

------------------------------------------------------------------------------------------------
| Design Timing Summary
| ---------------------
------------------------------------------------------------------------------------------------

    WNS(ns)      TNS(ns)  TNS Failing Endpoints  TNS Total Endpoints      WHS(ns)      THS(ns)  THS Failing Endpoints  THS Total Endpoints     WPWS(ns)     TPWS(ns)  TPWS Failing Endpoints  TPWS Total Endpoints  
    -------      -------  ---------------------  -------------------      -------      -------  ---------------------  -------------------     --------     --------  ----------------------  --------------------  
     -3.074       -5.992                      2                36450        0.056        0.000                      0                36446        0.264        0.000                       0                 15627  

Timing constraints are not met.

Open for suggestions, changes, criticism.

[sbourdeauducq]

• why is the div2 clock used in MMCM/PLL, rather than the 125MHz one?

Because of routing restrictions of IBUFDS_GTE2. You can't have the same output to the QPLL and to the fabric.

• why was sysclk running at ~114MHz instead of 125 (see clkfbout_mult_f @ 14.5 instead of 16)? this looks like an oversight?

Kasli 1 with the lower speed grade FPGA cannot meet timing at 125MHz. I guess we will have to introduce a slower RTIO clock to support legacy systems.

• Kasli QPLL took the buffered 125MHz from the reference directly (see line 209/256 of kasli.py) - I cannot use the "chosen clock" from BUFGMUX, as Vivado complains about unbuffered output, and sending the BUFGMUX output through BUFG/BUFH/BUFR resulted in unroutable design.

Indeed this is unroutable - the QPLL/transceiver needs to use the dedicated clocking pins and dedicated clocking network. And you don't want to switch that clock, it needs to be exactly 125MHz for Ethernet. And the Ethernet clock domain is asynchronous to all other domains. So just keep that direct IBUFDS_GTE2 -> QPLL connection, it is fine (and there aren't many other options anyway).

Your code cannot work reliably because the system PLL/MMCM may unlock/glitch when you switch clocks. You cannot put a BUFGMUX at the input of a MMCM without precautions, which are difficult to take here because the MMCM needs to be continuously clocking the CPU. It seems to confuse the Vivado timing analyzer as well.

I suggest you redesign it based on this scheme from Xilinx UG472:

The top-left IBUFG would take the Si5324 output, and the top-right BUFG would become a BUFGMUX with the main input from the MMCM (as shown) and the IBUFDS_GTE2 ODIV2 output on its other input. It is fine if the system bootstraps at 62.5MHz, just correct the UART baudrate register. Note that BUFG and BUFGMUX correspond to the same physical resource inside the FPGA (BUFG is just a software abstraction with some BUFGMUX signals hidden), so the deskew should also work. The feedback BUFG would stay as on the UG472 diagram.

[Spaqin]

Although this looks to be what we need, I cannot get the system to boot anymore (no code is run at all, no UART output). Probably something I missed in the log.

With the BUFGMUX for cd_sys and just BUFG from MMCM for cd_sys4x, there will be a slew of warnings:

WARNING: [DRC REQP-1580] Phase alignment: Unsupported clocking topology used for ISERDESE2 ISERDESE2. This can result in corrupted data. The ISERDESE2/CLK / ISERDESE2/CLKDIV pins should be driven by the same source through the same buffer type or by a BUFIO / BUFR combination in order to have a proper phase relationship. Please refer to the Select I/O User Guide for supported clocking topologies of the chosen INTERFACE_TYPE mode.

For all ISERDESE2 and OSERDESE2 - makes sense, Vivado doesn't know the ISERDESE2 will be used after the clock hand-over, with proper relationship - but it does clutter the log.

[sbourdeauducq]

Although this looks to be what we need, I cannot get the system to boot anymore

Can you blink an LED with a counter in the sys domain?

[Spaqin]

Still in WIP because after moving the reboot to the firmware it seems rather unreliable - sometimes it reboots after firmware sets the clock switch bit, sometimes it doesn't and just continues onward.

[Spaqin]

PulseSynchronizer sometimes wouldn't synchronize the pulse - not sure why, but ~half the time it would fail. An even simpler approach seemed to have worked, though - reboots every time.