[RFC] 2x2 SFP

[jordens]

What about the following:

• Stacked SFP cages/connectors. Either 2x1 or 2x2. That would simplify routing, space, front panel openings, LEDs. It would also get rid of the two different SFP cages.
• Ditching the SATA connector and going for 4 SFP. (Or make SATA a capacitor jumper option if sombody insists). So far the use for SATA has been to drive Sayma AFAIK. But that would be doable via SFP as well. Also having multiple Kaslis in a crate could be done cheaply with DAC SFP cables.

[sbourdeauducq]

* So far the use for SATA has been to drive Sayma AFAIK.

No, it's also been useful for having two Kasli inside the same crate, after all EEM slots of one Kasli are exhausted.

having multiple Kaslis in a crate could be done cheaply with DAC SFP cables.

I prefer an internal cable.

[jordens]

I prefer an external cable.

[marmeladapk]

I'm for using at least one 2x1 - this will allow to better space power supply, USB and SMA connectors. And it would give more space for DCXOs.

[jordens]

But then -- and even if we keep the SATA connector -- it might make sense to go 2x2 and jumper the GT lines. Two BOM lines (connector and cage) and space between the cages saved.

[hartytp]

My preference would definitely be to go for 4 SPFs on the FP and scrap the SATA to make routing easier. A 2x2 SFP cage sounds like a great idea. IIRC the only reason we didn't do that on v1.0 was that we were trying to fit Kasli into a 4HP form factor. Now it's 8HP I don't see a good reason not to do this.

The internal cable between two Kasli seems a bit niche to me.

[hartytp]

FWIW on Kasli Zync I suspect we may end up needing to use stacked connectors in any case as I think we'll want a SFP routed to the PS MAC for ethernet (as an added bonus, this gives us ethernet on DRTIO slaves, which could be useful). So, we'd probably want 1SFP for ethernet, then 4 SFP to the PL.

[dnadlinger]

The internal cable between two Kasli seems a bit niche to me.

Yep, that – I haven't looked at the signal standard, etc. side of things, but from purely a user's perspective, having only one type of connector/cable/… to worry about is definitely an improvement in modularity.

If just having 4 equivalent (up to gateware) SFPs is an option, that would particularly interesting as adding Ethernet to satellites is still on the back of my mind for throughput reasons, and we could still have a 1:2(+1 local) DRTIO switch this way.

[sbourdeauducq]

this gives us ethernet on DRTIO slaves, which could be useful

Why is it useful? We could already tunnel Ethernet packets on the DRTIO aux channel, which needs no additional wiring.

[sbourdeauducq]

from purely a user's perspective, having only one type of connector/cable/… to worry about is definitely an improvement in modularity.

What I like about the internal cable is that we can ship a more seamless solution where the Kaslis are already connected; otherwise the user has to add the SFP cable themselves (which also takes space on the front of the crate).

[marmeladapk]

What solution would make most sense if we plan to eventually move to cPCI?

[gkasprow]

Standard CPCIS defines controller as 4HP. So we would need another PCB design anyway. CERN backplane will have 6HP spacing.

[jordens]

AFAICT cPCI will invalidate the MMCX connectors and the SATA connector because the backplane is in the way for any internal cabling. Right?

[gkasprow]

CPCIS backplane has dedicated clock distribution on the backplane. About the SATA, it depends on what you would like to use it. We can connect it to one dedicated CPCIS slot.

[jordens]

Sure. That can all be done on cPCI(S). I was just referring the mechanics of internal cabling being mutually exclusive with cPCI(S), at least for standard or "CERN" backplanes.

[marmeladapk]

Okay, was there any resolution to this discussion?

[gkasprow]

Can we leave it as it is? Stacked SFPs are not that common and I already experienced availability issues.

[hartytp]

Stacked SFPs are not that common and I already experienced availability issues.

Yes, I hadn't realised that they are less common. However, there are enough parts that seem widely stocked that I wouldn't have thought it was too much of a concern. e.g.

https://octopart.com/2007492-5-te+connectivity-42631824?r=ap&s=i32uZXbMQPe8xcJPmDmv3Q

My feeling is that it's not essentially, but definitely nice to have. e.g. as @dnadlinger points out, having an ethernet port on DRTIO slaves for diagnostics/streaming data off. Also, on Kasli-Zynq how will we do ethernet? If we use the PS MAC, can we easily switch that between ethernet and DRTIO (transceiver) as we currently do? If not, then slaves only have 1 downstream port, which is a bit obnoxious for growing DRTIO trees.

[gkasprow]

It depends which ZynQ you use. GTR transceivers do not support 1000BaseX, only SGMII. An obvious thing is to use the same PHY we use in Sayma and connect it to RGMII. Such configuration works nicely. Once I had to use GTH transceivers with Ethernet SFP modules, but only 1000Mbit was supported by Xilinx Linux kernel. I didn't manage to make GTH working in SGMII mode for other speeds than 1Gbit. It's probably not a problem here but SFP + ZYnq transceiver + EMAC configuration does not work so nicely as it should. We won't use LInux here so maybe such issues won't appear.

[marmeladapk]

For now I placed 75640-5001 2x1 connector. It's necessary because I need more space for WR-like clocking. I'm still on the verge whether to put 2x2 SFP or leave SATA. @gkasprow is there a difference between 2x2 and 2x1 availability?

On a side note, 2x1 connector is longer, so I had to move EEM connectors.

[gkasprow]

choose the one that is available from more sources and make sure does not have EOL status.

[marmeladapk]

I'll use 2x2 cage and place SATA connector above the cage with 0201 jumpers connecting to SFP by default. Unless @sbourdeauducq is willing to forgo the SATA port. I'll place Amphenol UE78B212700321, it seems to be the cheapest available option.

[marmeladapk]

@jordens @hartytp Additional SFP requires an additional I2C bus. Is it okay to merge Si5324 and Main Si549 I2C buses? I don't think that Si5324 and Si549 will be used in the same time and this gives us a free slot in I2C switch (IC15) for SFP3.

[jordens]

Fine with me. But I think the I2C gateware is different for the Si5324 and the WR oscilators (bit bang GPIO vs proper gateware I2C). If I am not completely wrong, the helper oscillator might be doable with bitbanging in software though. @sbourdeauducq Moving the Si5324 off of the I2C switch is definitely fine. But on that same bus there is the EEPROM which can't be shared with any of the other I2C extender buses and also should be accessible to the FT4232. In that case it might be easiest to skip the I2C bus of the last SFP module.

[sbourdeauducq]

I am already planning to support bit-banging the Si549 bus in software, so that firmware can initialize (and debug) it instead of gateware. Then there is a multiplexer that selects the gateware I2C engine. https://github.com/m-labs/artiq/blob/wrpll/artiq/gateware/wrpll/si549.py So this sounds fine. And anyway, I suppose that there would be different firmware/gateware depending on whether one uses the Si5324 or the Si549?

[marmeladapk]

I'll connect EEPROM to SFP_3 bus with jumpers. Additionally I'll add jumpers to SFP_3 I2C, so we'll have the option to disconnect one or the other with BOM change.

[marmeladapk]

@jordens On a second thought I'll leave Si5324, EEPROM and SFP3 on the same bus but I'll change EEPROM to one that has address pins. Without them it takes to much of address space and collides with SFP (as far as I've read). So now we'll have: Si5324: 0x68 EEPROM: 0x57 SFP3: 0x50 and 0x51 GPIO expanders: 0x20 and 0x21. with jumpers to each device.

[jordens]

That's good.