Kasli specification

[jbqubit]

@jordens is tracking design proposal/specification for Kasli on the wiki. Let's discuss PCB_kasli specification here.

https://github.com/m-labs/sinara/wiki/Kasli

How Kasli might interface to peripherals is discussed in #99.

[jordens]

@dtcallcock Firewire is a pretty atrocious connector. There seem to be systematically bad experiences with stability and reliability e.g. in the SRS shutters. But anyway: Yeah, similar it would be. Modulo the question whether one digitizes at the frontend or later.

Since this is DRTIO, if you can trigger over the bus (I2C/SPI) you can time the sampling so that it meets your window without any additional signals.

@jbqubit Kasli has two SFP, you can therefore do Metlino--Kasli--Kasli--Kasli-- as much as you want. SATA is preferred over SFP for short distances because it is less complicated, more robust, easier to get the right components, smaller (this seems to becoming very relevant for Kasli), and cheaper.

[hartytp]

I was just thinking of starting a discussion of this.

Good idea. Probably worth its own issue though?

[gkasprow]

Guys, we can fit 4 SFPs on Kasli + USB and 2 SMA And for SFPs there are copper cables available and several extensions like 10/100/1000T Ethernet, fibre, SDI, HDMI (cameras). No reason to go for SATA

[gkasprow]

We can use numerous existing FMC extensions in Sayma directly.

Fast ADCs, DACs (up to 16 channels)

Fast IO (32x TTL)

TDC, DTC, programmable delay

LVDS IO

Camera link

In the future we can build deterministic computing module with floating point support

[gkasprow]

For daisy chaining we can use copper sfp patchcords

In addition, SFP has module detection mechanisms.

You can even hook up camera link to SFP using adapter.

[jordens]

@gkasprow Pretty sure that the SMA cables interfere with the SFP modules and the USB cable. That's much too close. Could we please do at most 3 SFP, 1 USB, 2 SMA? I am also worried about PCB real estate for all the IDC connectors, FPGA power supplies, SDRAM, flash, SFP cages, clock fanout. I'd like to focus on what's important and needed now and put all those things that could be possible in addition a bit lower on the topic list.

[gkasprow]

Don't worry about real estate - there is enough room we can try with micro USB

or as you wrote, stay with 3 SFPs and route one to onboard SATA.

[hartytp]

That particular combination of SMA USB SMA looks somewhat alarmed and panicked....

other than the visuals though, looks good to me ;)

[gkasprow]

The only reason is to separate 2 SMAs :)

[sbourdeauducq]

We can also remove one SFP to make some space and potentially ease PCB routing a bit. Kasli was intended as a very simple device, now it has all this stuff...

[dnadlinger]

To entertain the quite probably rather premature front panel discussion for a little longer: Just by eye, the SMAs look rather close to the micro USB jack – are you sure two standard plugs would fit at the same time? I imagine one might want to use both an external clock and UART/JTAG for debugging.

[jordens]

@jbqubit In the current state of things we will be funding Kasli and while we acknowledge and appreciate your recommendation on this topic, we will go a different route. We'd like easy and pain-free access to all front panel element for regular-sized (i.e. fat) fingers. That means reducing the density.

[hartytp]

@jordens To be explicit then, what front panel connections do you plan to include? (2xSMA, 1xUSB, 3xSPF?)

Presumably, at some point in the future we will make a cheap SFP hub board to go with Kasli. IMO, so long as this is planned, there's no need to over-stuff Kasli's front panel with SPFs.

[jordens]

@hartytp yes. 3 SFP, 1 USB, 2 SMA. 3 SFP already gives us more than sufficient scalability with only logarithmic DRTIO tree depth and plenty of unused transcievers at the leaves.

A "SFP hub" would need to be a full DRTIO switch.

I am confused by the apparent worry of not being able to scale due to lack of SFP ports. To me this seems unfounded and I value a good and usable mechanical design very highly. What is the biggest ARTIQ hardware tree that you can reasonably think of and why would one need more that the currently planned number of DRTIO downstream ports in Metlino/Sayma/Kasli? Somebody insisted that Metlino/Sayma would be designed to fill an entire 12-slot crate and the transcievers were routed to the backplane accordingly. That gives you for a single Metlino: 12 Saymas, 27 DRTIO downstream ports from the µTCA crate, enough ports to drive 7 full VHDCI breakout crates, and up to 33 Kaslis with their crates, everything at a maximum DRTIO tree depth of 2.

[gkasprow]

There is an option to use 6HP panel (instead of 4HP) and install stacked SFPs so we would gain plenty of space.

[hartytp]

@hartytp yes. 3 SFP, 1 USB, 2 SMA. 3 SFP already gives us more than sufficient scalability with only logarithmic DRTIO tree depth and plenty of unused transcievers at the leaves.

Works for me.

[hartytp]

There is an option to use 6HP panel (instead of 4HP) and install stacked SFPs so we would gain plenty of space.

IMO, there's no need to do this. 3xSPF should be plenty, and it's nicer to keep Kasli compact and simple.

[jordens]

3 SFP is plenty IMHO, I'd very much appreciate an internal SATA connector (standard, i.e. "computer" pinout, not swapped "disc drive") for the 4th transciever.

[gkasprow]

Add this requirement to spec on wiki.

[jbqubit]

3 SFP, 1 internal SATA, 1 USB, 2 SMA

Sounds good to me. @jordens Please edit wiki.

[jbqubit]

Will DMA streaming (from DRAM on Kasli) be available to Kasli peripherals? eg Zotino

[sbourdeauducq]

Yes, DMA is hardware-independent.

[jbqubit]

@jordens Please update Kasli specification on wiki to reflect details fleshed out in this Issue.

[hartytp]

@jbqubit @gkasprow Did you guys decide how you'd like the Kasli backplane to be wired up? I'm not entirely clear what topology was suggested (agreed on?). If anyone has thoughts, could they put them on the Wiki, please?

[gkasprow]

We specified with @marmeladapk that remaining LVDS lines from FPGA on Kasli will be routed to 96pin connector. The signals will follow signalling available in IDC connectors. We should be able to wire 3 or 4 x8 LVDS pairs + I2C buses. Kasli will also distribute copies of the Si5325 clock in point to point configuration. It may be used for two kinds of applications:

• requiring high speed signalling where IDC ribbon cannot do the job.
• some not yet defined applications that would benefit from B-LVDS or MLVDS signalling where LVDS pair runs across the backplane and several boards can talk to it. One can achieve 100Mb/s operation. In this way we can distribute i.e. high speed multidrop SPI over LVDS and with one Kasli connect several devices to it. We can also use the backplane to synchronize the modules. To answer your question - no, the backplane is not yet defined. We simply lack use cases.

[dtcallcock]

I added my suggested backplane connector allocation from earlier to the wiki.

There is also the issue of wiring the 96pin connector on the VHDCI carrier. My preference would be to route all four IDCs from one VHDCI connector to it (rather than for example two IDCs from each VHDCI). Then you have to option of running two VHDCI carriers and two backplanes off a single Metlino.

[dtcallcock]

Also, should we add an SMA front panel input and clock fanout chip to VHDCI carrier so it can drive the backplane clocks?

[gkasprow]

We can place IDC connector routed to 96pin connector that follows Kasli connectivity. In this way we could either use 8 IDCs or route unused ones to 95pin connector (DIN 41612) using short IDC jumper. To make both boards fully compatible, we would need to addd clock fanout as well. We can implement it in next revision of VHDCI board.

[hartytp]

Thanks for that guys.

Maybe I'm being a little slow here, but I'm still not entirely clear about what exactly we're trying to achieve with this backplane. I can think of three potential reasons to have a backplane:

a. Reducing wiring complexity (no ribbon cables to modules, no external SMAs needed to distribute clocks)? b. For high-speed digital communication that can't be sent over the ribbon cable c. Some kind of multi-point bus link to allow communication between extension modules without going through the carrier

(a) Is potentially quite nice, but IMO probably not worth the effort of designing and maintaining a backplane. Also, as previously discussed, if we use 96 pin connectors then a carrier can only drive 4 extension modules, which isn't convenient: each carrier has 8 expansion headers, so the BP wastes some connectivity; and standard racks will hold far more extension modules than this so we end up with empty slots or multiple backplanes per rack or something.

(b) and (c) are what @gkasprow suggests. But, AFAICT, no one has suggested a use case for them yet. We can't (shouldn't) specify a backplane for undefined purposes. So, if this is our motivation, maybe we should just leave the 96-pin connectors NC until we have a more concrete use case to design the BP around.

Whatever the answer, it would be good to agree on our motivation/goals for this, and to put that on the Wiki...

[gkasprow]

Actually, it's hard to fit more than 32 LVDS pairs (equivalent of 4 IDC connectors). They already would occupy 64pins. Add 6 or 8 clock outputs and we have 80 pins. Add 4* I2C and we have 10 pins left for grounds and supplies which is absolute minimum.

[dtcallcock]

If a) was the priority but only worth doing if we could drive all 8 extensions then we could switch to a 160 pin connector on the Kasli/VHDCI carrier (would also need the I2C switch and/or clock fanout moving to backplane). I think this only makes sense if the BNC/SMA/RJ45 extensions were also made full length to connect to the backplane. As it is those can only be connected via IDC so in most cases I would anticipate a mix of up to four full length cards (DDS/Synth/ADC/DAC etc) connected via the backplane and the rest of the connectivity taken up by the simpler IDC-only extensions.

On the other hand, if we thought a) wasn't worth the trouble but b) and c) were the real worry, we could drop the backplane and stick a bunch of SATA connectors (or similar) on the back of Kasli instead. The hypothetical extension requiring high speed connections would still use the IDC for power/I2C/slowstuff but could also be connected to SATAs as needed for high speed links. SATA would also be used for c).

We could also just wait until we're got our hands on the prototypes before deciding what to do here. I think some hands-on use will inform how much effort we are willing to put in to achieve a). For reference, in moving from the previous to current generation of NIST hardware we did consider it worth the effort to move from IDCs to a backplane. Also, use cases for b) and c) might arise in the meantime.

[hartytp]

Option a

@dtcallcock: I think some hands-on use will inform how much effort we are willing to put in to achieve a). For reference, in moving from the previous to current generation of NIST hardware we did consider it worth the effort to move from IDCs to a backplane.

ack

I think this only makes sense if the BNC/SMA/RJ45 extensions were also made full length to connect to the backplane.

Agreed: IMO, if we're using a backplane to cut down on wiring then all boards should connect directly to it without ribbon cable. This will require a minor revision to some boards.

@gkasprow: Actually, it's hard to fit more than 32 LVDS pairs (equivalent of 4 IDC connectors).

@dtcallcock: If a) was the priority but only worth doing if we could drive all 8 extensions then we could switch to a 160 pin connector on the Kasli/VHDCI carrier

Agreed: if we're planning to use a backplane for this then we will probably want to switch to a higher density connector.

There's also the question of how many extension modules we can fit in the rack. A standard rack is something like 84HP wide. I assume that we'll have "wide" extensions like the BNC breakout, DACs, etc which will be 12HP, and "narrow" extensions like the RJ45 breakout, at 4HP? So, do we want a single carrier per rack (e.g. either VHDCI or Kasli in a rack), and then up to 8 extension cards? A rack can then fit up to 6 wide extensions + carrier. All 8 BP slots can be used if narrow extensions are used as well. Obviously, this will leave a lot of unusable space if one populates the rack with only narrow extensions like the RJ45 breakout, but IMO we can live with that.

---

@dtcallcock: On the other hand, if we thought a) wasn't worth the trouble but b) and c) were the real worry, we could drop the backplane and stick a bunch of SATA connectors (or similar) on the back of Kasli instead. The hypothetical extension requiring high speed connections would still use the IDC for power/I2C/slowstuff but could also be connected to SATAs as needed for high speed links. SATA would also be used for c).

Exactly, I think this was @jordens motivation for putting some SATA connectors on Kasli (inside the rack, rather than on the front-panel).

---

@dtcallcock: We could also just wait until we're got our hands on the prototypes before deciding what to do here. I think some hands-on use will inform how much effort we are willing to put in to achieve

Agreed, it's probably better to wait until we've had a play with the hardware than to rush into specifying a backplane interface without due thought.

However, obviously the longer we put this decision off, the more hardware we're likely to design in the mean time (ADCs DACs SPI I2C etc). That means that we'll have to alter more boards to fit the backplane when a standard is agreed. IMO, so long as we can agree on what we're trying to achieve now then we should be able to specify it now.

It sounds to me like:

• Out chief motivation for a backplane is (a): cutting down on wiring clutter
• Any BP that can works for (a) will also work for (b) and (c), so we get those for free, even though they're not our main aim with the BP.
• High-speed interfaces (b) will additionally be supported by SATA on Kasli (not VHDCI)

Thoughts?

[gkasprow]

We decided to go for IDCs to enable out of rack operation of Kasli extensions. If we switch to backplane connectivity we would have to develop dedicated custom backplane for every combination of extension modules. Keep in mind that these modules may have different size so there will always be wasted some slots on the backplane. IDCs can be low or high performance. You can easily run 2.5Gbit (PCIe) data over such cable so there should be no issues with DDS modules.

[sbourdeauducq]

Also Kasli was intended as a simple device; any backplane support is more secondary (if not feature creep).

[hartytp]

@gkasprow: We decided to go for IDCs to enable out of rack operation of Kasli extensions.

Agreed, we definitely want to keep the IDCs. The questions was whether we have the backplane as well and, if so, what we are trying to achieve with it.

@gkasprow: IDCs can be low or high performance. You can easily run 2.5Gbit (PCIe) data over such cable so there should be no issues with DDS modules.

Agreed, signal integrity for LVDS over ribbon cable should not be an issue.

If we switch to backplane connectivity we would have to develop dedicated custom backplane for every combination of extension modules. Keep in mind that these modules may have different size so there will always be wasted some slots on the backplane.

Agreed, if we don't constrain the module sizes better then the backplane topology gets to be a mess. This is something that would need to be thought about

@sbourdeauducq: Also Kasli was intended as a simple device; any backplane support is more secondary (if not feature creep).

Agreed, backplane support is secondary and definitely not required. But, as @dtcallcock points out, without it the systems can start to descend into a rats' nest of ribbon cable.

FWIW, I don't feel strongly about whether we have a backplane or not. I'm mainly just trying to clarify the various suggestions that have been put forward and ensure that people aren't talking cross purposes.

I would like to make sure that if we do have a BP, that it's properly thought through and has a well-documented motivation. If we're not going to do that then I don't see any point in putting the 96-pin DIN connector on the Kasli/VHDCI boards in the first place.

[hartytp]

@jbqubit: Please update Kasli specification on wiki to reflect details fleshed out in this Issue.

Done. Although, let's not worry too much about the details of this (e.g. 3 v 4 SPF) until someone actually gets funding for it.

[jbqubit]

@hartytp Thanks for the cleanup of the wiki and update of Kasli specification.

Comments on recent discussion:

• The EEM peripherals are about doing simple things simply. The IDC connector for EEM is simple and should be used for our initial round of 3U peripherals. The "rat's nest" is unlikely to be unruly by physicist standards since EEM consolidates IO, I2C, and power.
• The Kasli Carrier and VHDCI Carrier should be able to interchangeably drive EEM peripherals.
• Extend EEM specification to add an extra pin pair for differential clock. 2x15 -> 2x16
• I don't see a strong use case for a backplane. I advocate that we drop the backplane from the specification entirely. Let future users define this. This also permits the use of a smaller FGPA -- see below.

I've taken a quick look at the number of IO used by the current design. Looks like at least XC7A75T is required if we support 8 IDC and 4 backplane EEM. A XC7A75T may suffice if we forgo the back plane. Google doc is open to edit.

[jordens]

To repeat what @sbourdeauducq said: If we do this (we plan to do so), Kasli will be simple and cheap. Backplane support will be there but probably just secondary. The backplane might -- for example --- end up being just an alternative to using the first four IDC connectors. I.e. you can still only connect 8 extensions. But you can use the backplane to conveniently (mechanically and electrically) wire the first four extensions. We see no problem with VHDCI breakout compatibility but where adequate we may choose to give that a low priority. @jbqubit, please undo your sprinkling of "EEM" everywhere. Inventing new and wrong acronyms is not helpful. None of the extensions are Eurocards at this moment. And currently I see no reason for more than a 50T. Your pin count is off and not how we prioritize of even approach this. I don't understand what you are trying to achieve. Instead it would great if you could contribute to publication quality datasheets for Sayma and Metlino (actual numbers, features, configuration options, high quality block diagrams) that detail the specification and convince other potential users of your design decisions and of the hardware quality.

[jordens]

Well. The EEM may have been @hartytp. Anyway, AFAICS most of them are not 100+n*60 mm long...

[gkasprow]

@filipswit started seriously working on Sayma and Metlino documentation. Filip just finished schemtic review of all boards and found several issues. He will use tex templates from AFCK.

[hartytp]

@gkasprow: @filipswit started seriously working on Sayma and Metlino documentation. Filip just finished schemtic review of all boards and found several issues. He will use tex templates from AFCK.

What do people think about the idea of using the Wiki as the main source of documentation? This could have a few advantages:

• Easy for people to edit later on, for example to post measurement results or comment on hardware bugs we find
• Allows us to keep the entire Sinara ecosystem documented in a single place, rather than having a tex file for each component
• It's already a git repo, which makes version control easy

[jbqubit]

@hartytp @gkasprow Please move discussion on documentation to #133.

[hartytp]

@jordens The EEM terminology and pinout were mine. Apologies if I've made mistakes here, or if you feel this was unhelpful.

To be clear about my motivation for the updates I've made to the Wiki: we intend to fund the DAC and ADC boards imminently (hopefully in the next couple of weeks). The plan is to make something that we can run from the VHDCI carrier in the short term, but which will ultimately work with Kasli. From reading the issues, I get the impression that there are various conflicting ideas about what this hardware will do, and how it should fit together. I'd like to get a coherent, well documented plan before we start building hardware -- for example, if there's going to be a backplane then we will make our DACs compatible with it, but we can't do that until the specification is written down somewhere. If you have strong opinions, feel free to put them on the Wiki and we'll be happy to accommodate them.

Turning to your more specific points:

None of the extensions are Eurocards at this moment.

You're correct, my language was imprecise. I mean that the boards should fit within the Eurocard form-factor, so that all boards are 100mm wide and no deeper than 160mm. AFAIK all currently planned boards do fit this. I'll amend the Wiki.

please undo your sprinkling of "EEM" everywhere. Inventing new and wrong acronyms is not helpful.

When I've had conversations with people about this, I found that I needed a convenient short-hand way of referring to this hardware (and to differentiate it from the uTCA sub-ecosystem). EEM emerged as an obvious convenient choice. I think it's important (and certainly not unhelpful) to have some name for them. If you have a better name then please introduce it.

Your pin count is off and not how we prioritize of even approach this. I don't understand what you are trying to achieve.

Which pin count? Do you mean the 2x15 pin header pin count? That was taken from the current VHDCI/3U_BNC designs. What mistake have I made? My goal is to have a pinout that I can design our DAC/ADC boards around.

Backplane support will be there but probably just secondary. The backplane might -- for example --- end up being just an alternative to using the first four IDC connectors. ... We see no problem with VHDCI breakout compatibility but where adequate we may choose to give that a low priority.

Can we at least try to agree on specifications (pinout etc) for the headers + backplane now? We plan to start building hardware based on the current VHDCI carrier design soon. It seems perverse to start building a Kasli carrier in a few months that won't work with DAC/ADC extensions we start building now.

I.e. you can still only connect 8 extensions. But you can use the backplane to conveniently (mechanically and electrically) wire the first four extensions.

The backplane might -- for example --- end up being just an alternative to using the first four IDC connectors. I.e. you can still only connect 8 extensions. But you can use the backplane to conveniently (mechanically and electrically) wire the first four extensions.

If you have strong opinions about this, please add them to the Wiki, and we'll design our hardware around them.

[hartytp]

Also, a small bugbear: in a few places, people are calling the 100mil male pin header on these boards "IDC". IDC (insulation-displacement contact) is a method of crimping ribbon cable onto housings. These housings mate with the pin headers on the board, but are not the connectors used on the boards themselves. Can we stick to "pin header" or similar please to avoid confusing things?

[gkasprow]

The 3U boards have such pin headers surrounded by walls which are mating parts for crimped ribbon cable IDC connectors. So the pin headers are like this: while IDC headers are like that:

[hartytp]

@gkasprow Okay, I'm used to seeing that referred to a "shrouded" or "boxed" pin header, rather than an "IDC" header. But, if this is standard terminology then I'm wrong and I rescind my objections. Live and learn ;)

[gkasprow]

Whatever you google it gives the same pics :) But indeed, shrouded is official name in many producers catalogues. But since IDC header means that you plug IDC jumper we can leave this for clarity.

[hartytp]

ack

[gkasprow]

To define backplane structure we first have to think of communication channel for backplane enabled devices. IMHO we should reserve backplane option for special use cases. Individual IO channels like SMA, BNC, RJ45 need dedicated LVDS links and IDCs do the job nicely. Backplane enabled modules will benefit from clock distribution so we can install there DDS, RF generators, DACs, ADCs that need external synchronization. For backplane devices we can make dedicated VHDCI adapter that supports high speed MLVDS. So we can define backplane as MLVDS bus with limited number of supported protocols. In this way all modules may have identical pinout and routed i.e. 16 LVDS lines + one dedicated selection line. The interface that fits best both simple and complex devices is SPI over LVDS with multiple incarnations like QSPI, double QSPI. It resembles MMC interface where clock, chip select and 8 data lines are used. With such interface running across the backplane, we can insert as many devices as the rack fits and obtain scalable transfer rate. It has one more benefit - we can mix 4HP and 8HP modules without loosing too many Kasli IO lines - only single chip select is not used. For simple DACs we simply use standard SPI and ignore remaining data lines. For applications that require hundreds of MB/s transfer rate (high speed ADC, DAC modules) we simply use 8 or 16 bit interface. To make all deterministic we can define time slots for every device. SPI clock speed can also vary depending on currently selected module. MLVDS supports up to 100MHz. Some simpler modules (slow ADC, slow DAC) can be designed as dual function with both backplane connectivity and IDC connector to enable stand-alone operation without the backplane.

[jordens]

Rough Kasli design looks settled. Backplane discussion moved to #168.