low latency PCIe link for Kasli

[gkasprow]

We have a few QC applications where the latency of Ethernet is dominating. We plan to develop a direct PCIe link that would run in parallel to Ethernet and would be dedicated to dumpling memory content directly with a delay of a few tens of us. There are a few ideas how to do so.

• Standard PC main-board uses SSC technique so it means the PCIe clock must be delivered to the PCIe device together with data. Kasli do not have suitable clock input. We would need to add a dedicated MGT clock input to Kasli instead of 125MHz oscillator.
• Another way, already tested in our lab is to use a special PCIe to QSFP adapter equipped with PCIe bridge that isolates the PCIe clock domains so the data sent over fibre do not use SSC and CDR on the device side can recover the data correctly. This is known to work for PCIe gen3, need tests for Gen 2 and Gen1.
• Yet another way of establishing the PCIe link is to use Thunderbolt, either natively or via PCIe extension card. On the other side of the cable we would place passive adapter that fits into SFP cage.

[sbourdeauducq]

• a special PCIe to QSFP adapter equipped with PCIe bridge

Doesn't this increase latency? Would it make sense to use Infiniband instead at this point?

[jordens]

Why not do all of them? I.e. add a variant that clocks MGTREFCLK1 from an external source for those that want to play with it and as a backup option to PCIe over SFP+/QSFP. For Gen2 SSC CDR also known to work if you use a PCIe switch that supports it (https://www.renesas.com/us/en/document/apn/715-multi-port-ssc-support-idt-pcie-gen2-switches?language=en e.g.)

[gkasprow]

@jordens I don't see in this note a scenario where PCIe switch could retrieve data from SSC -clocked link without having additional SSC clock channel... @sbourdeauducq Of course, additional switch will add latency but it is in order of <1us usually. For example PEX8749 has 126ns max latency. But PCIe is common and cheap interface unlike the Infiniband. You can even hook up a laptop with Thunderbolt which has exposed PCIe...

[gkasprow]

You are right @jordens : https://www.eetimes.com/pcie-goes-clockless-achieving-independent-spread-spectrum-clocking-without-ssc-isolation/

[gkasprow]

But still, I don't think Artix FPGA can work clockless with SSC PCIe link..

[jordens]

Isn't this what you want?

[gkasprow]

@jordens this is the scenario I described in second point (clock isolation) and we use it already. But that requires PCIe switch or purchase of existing PCIe to QSFP adapter.

[jordens]

Exactly. I only mentioned it because you were unsure whether it works with Gen 2.

[gkasprow]

I know switches work with Gen 2. The question is if Artix 7 will work with clock-less Gen 2.

[gkasprow]

@jordens does Kasli use this 125MHz clock for non-MGT purposes ?

[jordens]

yes. it also clocks large parts of the the fabric with it.

[gkasprow]

PCIe core in 7-series FPGA can be clocked with 125MHz, so we only need to check if it works in asynchronous mode. And the easiest way to go is to use the SSC isolation switch.

[gkasprow]

@jordens would it be a problem to clock this logic by 100MHz SSC clock (+/-30kHz) ?

[jordens]

For most applications not. And in your use case very unlikely