[RFC] DI/OT FMC Carrier

[marmeladapk]

I'm designing FMC carrier board for DI/OT project for CERN. The specification is here. Basically it's a board in the same ecosystem as the system board which Greg discussed with you earlier.

This board will have Kintex Ultrascale FPGA, DDR4 SODIMM slot and FMC HPC. I'll probably be able to also include SD card or eMMC storage (requested in spec). One of the points in spec which I need to clarify is presence of USB connector.

This board could be a good fit for Shuttler FMC.

Is there something else required for ARTIQ support other than Si549? It already has a MGT connection to system board.

@sbourdeauducq @jordens @dhslichter

[jordens]

Nice. I don't see anything that would be needed. And I think that the clock recovery mechanisms of wr and drtio should be unified. Assuming I read the spec correctly, are you sure you can leave the vadj fpga banks unpowered?

[marmeladapk]

are you sure you can leave the vadj fpga banks unpowered?

I read further and found that in principle VCCO is not strictly required for unused banks but

The IO protection ESD structures are designed to work while powered (best protection).

In other ARs and in UG they advised against leaving VCCO floating and didn't write about connecting it to GND, but I guess it's not worth the risk. Thanks for the tip!

[jordens]

Right. That was also my conclusion for the potentially unused bank on Phaser.

[marmeladapk]

@jordens @sbourdeauducq Would it be ok if I connect output from main Si549 to one of inputs of Si5341? In DIOT system board CERN is doing the same with their main VCXO - its output is connected to the input of Si5341.

[jordens]

To save a mux/fanout? Fine with me. Noise and determinism should not be affected, but that needs to be confirmed by someone else.

What's the CERN rationale for the Si5341? Is it a backup/alternative for WR clock recovery or just a synthesizer/buffer/clock fanout/divider?

[marmeladapk]

Yes, it's to simplify the design.

Si5341 seems to be all in one solution for clocks, Silabs even advertises it that way "clock tree on a chip". Clock from backplane is an input to Si5341.

[jordens]

di/ot zu clocking

They have no plan to use it as an alternative/successor for WR clock recovery? Ultimately you'd hope that the WR recovery scheme can be replaced by a smaller/cheaper/better integrated solution. For us the only real reasons were deterministic phase which wasn't available and maybe a bit the performance hit. The phase issue looks solved with the zero delay mode and the synchronized dividers. One could even consider putting more of the tree into the feedback loop. Performance-wise it looks better than the WR-VCXO.

[jordens]

@marmeladapk Skipping Kasli-SoC in favor of this would be very valuable IMO. Or at least making the transition from Kasli-SOC easy for those that want to stick to ribbon cables for now. Is there any effort on that compatibility front yet?

[marmeladapk]

@jordens Do you mean skipping KAsli-SoC in favor of DIOT System board?

Is there any effort on that compatibility front yet?

In what way?

@gkasprow consulted system board with you and there are Si549 oscillators there as well. Each slot gets 15 LVDS pairs + MGT in CERN variant or 16 LVDS pairs in Sinara variant. @gkasprow also designed CPCIS adapters for existing EEMs. What else do you have in mind?

[jordens]

@jordens Do you mean skipping KAsli-SoC in favor of DIOT System board?

Yes. Transition to cPCIS and SoC at the same time.

Is there any effort on that compatibility front yet?

In what way? @gkasprow consulted system board with you and there are Si549 oscillators there as well. Each slot gets 15 LVDS pairs + MGT in CERN variant or 16 LVDS pairs in Sinara variant. @gkasprow also designed CPCIS adapters for existing EEMs. What else do you have in mind?

The way I see it we might be able to still make small changes to your DI/OT FMC carrier but we might be able to make bigger changes to Kasli-SoC to align them better. I'd like profit from the CERN involvement (review, development, characterization, testing, volume, cost). Pretty sure that Kasli-SoC will not profit much from that.

• Clocking: IMO Kasli-SoC should use the DI/OT clocking tree (add 4 MMCX). It support all options (WR CR with VCXO, DCXO, and Si5x-based recovery).
• I guess the FMC on DI/OT could be used with 4 MGTs to 4xSFP for DRTIO. 1.8 Vadj is not a problem there?
• DDR4 might be a bit of a challenge. But certainly a step forward.
• Same for the Ethernet section. We'd have to go pass Ethernet through an SFP+MGT and PL to PS, AFAICT? Might even be preferrable compared to the copper PHY.
• I2C bus tree differences? Looks like bigger delta but less critical.
• Any chance to get an FTDI for JTAG on the DI/OT board? What does JTAG over backplane look like in the ZU case?
• Do we get a front panel clock input to the tree?

(Looking at the ZU DI/OT board)

[sbourdeauducq]

Yes. Transition to cPCIS and SoC at the same time.

I do not like this idea; the Kasli-SoC design is almost ready and it can support the existing cards already.

I'd like profit from the CERN involvement (review, development, characterization, testing, volume, cost).

There are also disadvantages to it, such has having even less control over the designs than we have currently.

[hartytp]

Without wanting to make a judgement on what anyone else should/will do, I agree that Kasli-SoC is a good next step for us.

A backplane is at best a nice to have for us (personally I don't think the ribbon cables are that big a deal). One of the lessons I took from the original Kasli is that keeping the designs minimal complexity and totally under our control was a real benefit in terms of getting working hardware. I'm apprehensive about coupling our use-cases to those of a big organisation like CERN.

[hartytp]

I'll be interested to follow this project, but we'll keep pushing Kasli-SoC forward for the foreseeable.

[marmeladapk]

Any chance to get an FTDI for JTAG on the DI/OT board? What does JTAG over backplane look like in the ZU case?

Unfortunately it's too late for that, the system board is already routed. Maybe in the next revision if there is one. I pasted FTDI schematics from Kasli/Sayma in my FMC carrier, we'll see if it sticks.

What does JTAG over backplane look like in the ZU case?

AFAICT DI/OT SB is the source for JTAG for other cards. When it asserts SERVMOD_N signal then cards should connect some BP signals to JTAG chain and also connect their I2C to BP. (see crate monitoring)

There are also disadvantages to it, such has having even less control over the designs than we have currently.

It's open hardware, so if something doesn't work we can modify the card to our liking. Even if we modify the card, we could benefit greatly from common BoM.

Anyway my goal is to look for possible changes that could be beneficial for us and neutral for CERN. I wouldn't axe Kasli-SoC, since we don't even have ARTIQ support for ZU.

[hartytp]

Anyway my goal is to look for possible changes that could be beneficial for us and neutral for CERN. I wouldn't axe Kasli-SoC, since we don't even have ARTIQ support for ZU.

I completely agree with this. It's definitely worth us having input on these projects and steering them where we can. I just wouldn't want that to be at the detriment of Kasli-SoC.

[jordens]

I am not aware of any data to support the idea that custom designs have lead to better or earlier working hardware. We have lots of data to the contrary: the first working hardware was KC705 (complete with "foreign" DDS boards) and the first working Zynq system looks to be the same. Furthermore hardware totally under our control comes at a significant cost that we frequently fail to pay: it requires a level of maintenance, review, characterization, and QA that we struggle to achieve. We do have a bunch of examples for that problem as well.

The really important thing I need to stress here is that a simple solution for an expert is frequently an unworkable solution for an end-user. Ribbon cables are an excellent example. You need a significant amount of experience to reliable and properly plug and unplug them. I've actually watched new end-users do it. You need to know how to hold and where to pull and how to counter the force. You need to open and close the crate and move/insert cards and card guides. You need know which side of the cable goes where. You need to know when the connector is properly inserted and you need to take care of ESD and no-hotplugging instructions. You need to tie down the ribbon cable tree for vibrations if you ship the device. They are simply not a solution for an end-user device. Having the inexperienced end-user unable to insert and remove modules easily/low-risk reduces the value of Sinara significantly and makes selling and supporting these devices difficult and costly.

I'm also not judging your decisions and plans, but the number of anachronisms and idiosyncrasies makes it much less valuable to us in the long run.

If Kasli-SoC is really a worthwhile intermediate step, then please at least take the lessons and existing designs into account and don't run it into a dead end. E.g. look at the suggestion for the clocking tree that I made above and consider planning for a future port to a cPCIS crate during design (i.e. mechanics and layout) now. Those are high-value low-risk changes.

[hartytp]

If Kasli-SoC is really a worthwhile intermediate step, then please at least take the lessons and existing designs into account and don't run it into a dead end. E.g. look at the suggestion for the clocking tree that I made above and consider planning for a future port to a cPCIS crate during design (i.e. mechanics and layout) now. Those are high-value low-risk changes.

I'm very receptive to ideas for Kasli-SoC -- the reason I'd like to fund a co-design is to get more input on this kind of thing. If you have suggestions can you open an issue on the Kasli-SoC repo?

[gkasprow]

the Kasli/Kasli-Soc to CPCIS adapter is currently under development. So we can try it in the CPCIS crate and decide if the next revision will support the backplane natively.

[marmeladapk]

Current schematics: DIOT_FMC_Carrier.PDF

It's still WIP, but the concept is there. I await for CERN input, so some things may change.

[marmeladapk]

@sbourdeauducq @jordens Are you ok with clocking in these schematics? Also, one signal pair from system board can be either 16th LVDS (Sinara variant) or MGT (CERN variant). Would you prefer having MGT connection to system board or an additional LVDS pair? (so, should I make a variant with LVDS connection).

[jordens]

That board is for a peripheral slot, not the big CPU slot, right? I'd clearly prefer the MGT connection.

[gkasprow]

@jordens Yes,

[marmeladapk]

https://ohwr.org/project/diot-pfc-ku