[RFC] EEM FMC carrier

[dhslichter]

There have been various discussions around the DI/OT FMC carrier in #56, but it seems for the time being like there is not a consensus yet. I am wondering what people think about the idea of developing an EEM FMC carrier card with an onboard FPGA. This would allow for some future developments to be carried out as FMC-daughtercard-only developments, which may mitigate some risk and simplify some designs (depending on the design), while also potentially reduce cost/iteration time because the FPGA/EEM side is stable and verified (this is, after all, the point of FMC!). However, this would probably only useful/appropriate for a subset of Sinara designs. Such a carrier would also potentially enable the use of COTS FMC cards with Sinara, which would be a nice capability and increase the breadth of Sinara's (and ARTIQ's) appeal. Because it would be an EEM card, and not a DRTIO satellite or master, one wouldn't need to have as much complexity on the board as for Kasli/Kasli-SoC. Obviously the bandwidth constraints of the EEMs mean that the FMC carrier card would have to do a lot of processing internally to feed high-speed DACs or ingest high-speed ADC data (or use both and implement digital servos), so it would need to have an FPGA onboard.

In my view, such a card might have the following properties (comments encouraged!):

• single FMC connector (fully LPC compliant, and ideally at least subset/partially HPC compliant, to enable the use of a wider range of COTS FMC daughtercards)
• mechanical design to accommodate single-width FMC daughtercard per standard. 10 mm vs 8.5 mm stack height?
• 1, ideally 2 EEM connectors (increase throughput to/from Kasli if used for high-speed DAC output)
• suitable FPGA (not too huge/expensive/power hungry, but big enough to enable the desired tasks)
• SDRAM (enough for currently envisioned use cases, plus maybe some general future-proofing)
• possibly some off-card gigabit links (over SATA connectors?)
• onboard power regulators (to run all ICs on the carrier card, give VADJ rails for FPGA and FMC, plus +12V and +3.3V for FMC per standard)
• simple clock inputs (internal MMCX clock from Kasli plus the possibility to accept an external clock from the FMC daughtercard, potentially) and appropriate clock fanout
• would one want a clock cleanup circuit? I don't think it's necessary if we are just MMCXing from Kasli?
• flash, JTAG, etc. Set up JTAG chain to enable JTAG programming of FMC daughtercard or FPGA.
• perhaps a microSD card slot? not sure if this would add any value.

Now for a more specific proposal: it seems to me that the current Phaser design could be adapted to implement such a board with relatively minor modifications, which might make it quick/inexpensive to spin and test while also reducing risk. The FPGA is a reasonable size and performance for the task, there is a 4 Gb SDRAM onboard, two sets of GTP over SATA, plus two EEM connectors. If you remove all the DACs, ADCs, modulators, attenuators, etc., this would leave banks 13 (currently unused, partial bond-out), 15, 16, and 34 completely free to populate the FMC (with the exception of a few signals on bank 34, which could be moved if needed). This would provide enough pins to fully populate the 160 lines (80 diff pairs) for an HPC FMC's LA, HA, HB banks. We would subset the gigabit transceivers by having one TX pair and one RX pair (currently available on the FPGA), allowing full LPC compliance but only partial HPC compliance. In general, though, I think most of the relevant COTS FMC daughtercards won't care, unless they are JESD204B-based. We could choose to reallocate some more of the GTPs to the FMC connector if desired, but I don't have a strong preference there.

The primary reason I am thinking about this right now is that Shuttler is currently being designed as an FMC card, and will require a carrier card in order to operate and be tested. Being able to harness the work on Phaser to make a simple, suitable EEM-based carrier card would make it quicker and simpler to integrate the Shuttler FMC cards into a Sinara EEM crate. Of course, one would want to make sure that such a design of a carrier card is appropriate for a variety of other FMC applications. I think the biggest question in my mind is whether people think the Phaser FPGA (Artix-7 XC7A100T) plus 4 Gb SDRAM are big enough for a general-purpose EEM FMC carrier card.

How do people feel about an EEM FMC carrier card vs. a DRTIO-connected FMC carrier card? In the long run we might potentially want to have both, but at least right now the EEM one seems like it might be simpler, less expensive, faster to develop?

@gkasprow @jordens @hartytp @cjbe @dtcallcock @sbourdeauducq

[gkasprow]

The DI/OT board schematics are currently under review. The entire discussion moved to OHWR since it is a CERN-funded project https://ohwr.org/project/diot-pfc-ku/wikis/home We use Kintex US FPGA due to very attractive pricing (common with CERN step pricing), comparable with Aritx FPGA. It will serve either as EEM slave or DRTIO satelite.

[gkasprow]

It is designed with Shuttler, TDC, 4x1GS/s JESD ADC and quad SFP FMC in mind. All these boards are OH designs. We developed a lot of OH FMC boards for AFC/AFCK ecosystem, currently we are working on porting them to 1V8 VADJ. In the meantime we are working on new revisions of AFC/AFCK/AFCZ boards to have common ecosystem for both EEM/DIOT and MTCA. We also working on providing MISOC support for our carrier and FMC boards.

[dhslichter]

Will these be compatible with the 3U EEM form factor, or does it have to use a backplane? What about clocking from a Kasli, etc? And what is the status of the Kintex US gigabit transceivers working with ARTIQ? Is everything suitably debugged that they can be used? My concern is that this DI/OT design is overkill for our immediate purposes, and I want to make sure that the added complexity doesn't hurt our more modest goals.

[gkasprow]

This is a CERN project, so it will require backplane. We can make EEM version if there is demand, but the plan is to migrate entire SInara to backplane. We already produced prototypes of OH chassis with backplane and compatible supplies so we can keep the cost low.

[dhslichter]

What is the timescale/complexity/debugging level for moving everything to backplane? Certainly there are upsides, but there are also downsides as well. Will there be backplane-to-EEM adapters to help with legacy hardware?

[gkasprow]

Yes, most existing modules are compatible (at least new revisions) with adapters. This includes Kasli, Kasli SoC and other modules. "short" DIOs are replaced with dedicated 16-channel MCX and individually isolated MCX modules. More about the timescale knows @marmeladapk

[dhslichter]

Based on all of this it sounds like we should not try to make an EEM FMC carrier. Closing!

[dhslichter]

@marmeladapk can you provide information on the timescale? What do you think are the biggest potential roadblocks?

[gkasprow]

But I plan to make a simple EEM-to CPCIS adapter to make possible using DIOT boards in existing EEM ecosystem.

[marmeladapk]

FMC carrier is currently undergoing a schematic review at CERN.

You can check this repository, I'll upload files next week and also issues regarding schematic review will be there:

https://ohwr.org/project/diot-pfc-ku/issues

Right now I'm doing SI simulations for System board (Zynq ultrascale based).

Regarding CPCIS crate and other matters, pinging @akaminska.

[dhslichter]

For ARTIQ operation, do we require the "System board" as well? And we have to port ARTIQ to this "system board"? Sorry I have not been following all the details ☹️ Basically I am nervous about having a large number of dependencies before we can run Shuttler with ARTIQ. Will the adapter board discussed by @gkasprow enable the FMC carrier to be used via EEM with a Kasli? How much of the unpleasant interfacing "glue" work would need to be carried out to get this to work? What about clocking (over MCX from Kasli), etc?

[gkasprow]

No, we can use Kasli/Kasli SOC with adapter to drive CPCIS/DIOT backplane. The other scenario is to use KAsli/Kasli SOC/VVHDCI carrier with the DIOT FMC carrier using tiny EEM adapter. In such case the FMC carrier can be connected using EEM or DRTIO (SATA-SFP cable).

[akaminska]

To update the status of the CPCIS crate - we will have the first prototype assembled end of this week and move to testing.

[gkasprow]

This is an entirely open source CPCIS crate including the backplane and mechanics.