[RFC] New EEM: Fast Kasli Servo

[dtcallcock]

A fresh issue for one of the two boards proposed in [RFC] Sinara Servo sinara-hw/meta#16.

This would be a high performance board using the FPGA on a Kasli to run the feedback loop. Board would have ADCs, DACs and all the required analog electronics.

• Roughly equivalent capabilities to the [NIST servo] (https://arxiv.org/abs/1508.06319) (100MSPS parallel DACs/ADCs, few 100ns latency).
• Able to do direct digital modulation/demodulation for things like PDH locks.
• Would need quite a few EEMs.
• Need to consider timing issues if using ribbon cables. Could be made as a daughterboard that stacks onto Kasli and uses the EEMs as board to board connectors to avoid this but that would limit us to 1 per Kasli.
• High cost, approaching $1k/ch

[dtcallcock]

Relevant comment from previous thread by @gkasprow :

There are fast 100MS/s ADCs with LVDS output. For example LTC2174 family. With Spartan6 and Artix chips you need 10 LVDS to get data from 4 channel ADC (8 data lines, bit clock and frame clock) It adds some latency but it is negligible.. We use them in this design. Then 2 EEM sockets would serve 4 ADC channels. 2 channel versions are also available and would consume 6 LVDS lines. One could use single EEM then. But there are no serial LVDS DACs. Probably due to issues with lane alignment. So they would need to be driven directly. Dual channel LVDS DAC would need 14 + 1 LVDS lines and would consume another 2 EEMs. With 3 EEMs occupied we would be able to fit 2x 125MS/s ADC and 2x DAC on same 3U board. In this way Kasli would be able to run 3 such boards and 6 servo channels.

[gkasprow]

@hartytp what EEM has higher priority for you? Fast Kasli Servo or Fastino? I have a student who want to do sth related with fast control using FPGAs.

[hartytp]

@gkasprow right now, neither Fast Kasli Servo or Fastino are particularly important for me (the MSPS ADC+DAC+uProcessor board is very important for me right now however). Others may feel differently.

[dtcallcock]

I think the microcontroller servo will end up serving the 'fast control' role I had envisaged for Fastino (though I think Fastino is so simple and potentially useful for other things that it is still worth making at some point).

Therefore the Fast Kasli Servo would probably be the most useful project this student could do if they want a fast control FPGA project.

[hartytp]

It would be really useful to have a list of potential use cases for this card that aren't served by the microprocessor servo. That way we can target the design a bit.

FWIW, I'm personally not convinced that this design is the right way to go for PDH locks. That seems like a specialist enough application that it's worth making a Sinara card to do it (2-channel DDS, mixer, etc).

[dtcallcock]

I think having that a servo that can do direct digital up/down conversion would be a really powerful tool. Being able to do PDH locks in gateware is just a subset of this. I know Jeff Sherman has been thinking about it for optical timescale work. I'll ask him to see if he wants to comment on this.

Btw. the NIST servo can already do this but IIRC the phase detector is limited to something like 25MHz so we still end up having to use external analog electronics for PDH locks (most of our optical cavity locks are ~50MHz modulation frequency).

use cases for this card that aren't served by the microprocessor servo

I think to a large extent this will depend on:

• How powerful the microprocessor servo is. There are many orders of magnitude in microcontroller power and features (DSPs, dedicated realtime cores, etc.) so until it's really specced out it's hard to say whether it'll be able to do most things or if it'll just be a dinky slow PI lock.
• Whether we make a PDH lock card for the microcontroller servo and it ends up being much cheaper than a fast Kasli servo (very likely). If we do, that takes reduces the market a lot I imagine.

[hartytp]

I think having that a servo that can do direct digital up/down conversion would be a really powerful tool.

I think it probably would -- and I'm definitely not against it as an idea for a Sinara board -- I just don't know what it would be used for and I'm curious...

I know Jeff Sherman has been thinking about it for optical timescale work. I'll ask him to see if he wants to comment on this.

Cool. It would be interesting to hear from him.

How powerful the microprocessor servo is. There are many orders of magnitude in microcontroller power and features (DSPs, dedicated realtime cores, etc.) so until it's really specced out it's hard to say whether it'll be able to do most things or if it'll just be a dinky slow PI lock.

That's a good point. I'd definitely like to spec out a reasonably high-end microprocessor for that servo to maximise its usefulness.

AFAICT the DSPs etc don't buy one that much extra on top of a good microprocessor running well-designed code.

Whether we make a PDH lock card for the microcontroller servo and it ends up being much cheaper than a fast Kasli servo (very likely). If we do, that takes reduces the market a lot I imagine.

While it's not something I have any plans to do myself, I think it would be really cool to make a decent Sinara PDH modulator/demodulator eurocard. We're using an analog solution from a well-known laser vendor and it's just awful. The batch of units we have all have parasitic oscillations in the GHz regime (probably a MMIC amplifier + sloppy layout/inadequate decoupling), which alias down in weird ways. Turning the gain above a certain level causes the whole thing to crap out (looks like a lower frequency rail-rail oscillation). Huge offsets, low bandwidth, etc.

The manufacturer wasn't even that surprised/bothered when this was reported. Just awful crap. The newer boxes they sell may be better though.

[hartytp]

Another potential application for this would be locking lasers with decent BW without having to buy an expensive lock box (the ADCs + DACs in the DL Pro controllers are surprisingly slow).

[sbourdeauducq]

Could be made as a daughterboard that stacks onto Kasli and uses the EEMs as board to board connectors to avoid this but that would limit us to 1 per Kasli.

That would also cause nasty mechanical issues, as the mating tolerance for the IDC-type connectors is pretty tight and front panels from both cards then have to fit into the enclosure, with complex and poorly controlled tolerances. You'd potentially get poor contacts, and perhaps a blown board from +12V leaking into inputs, if the card guides of your Eurocard enclosure aren't just right. Very frustrating! That could be avoided by having cables for the analog signals, but analog degradation isn't necessarily better than digital. And actually LVDS on ribbon cables works surprisingly well...

[gkasprow]

ACK, IDCs are not foreseen for board-to-board connectivity. They do not have alignment keys so assembly tolerance is not very high because relies on manual soldering. We can fit such high-speed ADC and DAC on standard EEM with a tiny 5$ ICE40 FPGA serving as a deserializer for parallel DAC. 2 ADC + 2 DAC channels sampling at 125MS/s are feasible for dual channel EEM. Let's not use more than 2 IDCs because it would prevent us from using CPCICS backplane.

[sbourdeauducq]

If we put an FPGA on that board we might as well run the servo on it. ECP5 also has free software tools.

[gkasprow]

The idea behind simple FPGA was that it is a part of the design that does not need modification. Once we move the whole servo to dedicated FPGA, we need to add a communication channel, setting registers, and configuration interface. Both approaches have pros and cons. In Banker/Humpback I added I2C path that enables FLASH update. For the fast servo, we may have troubles with FPGA resources and timings when setting up multiple servo channels so such a dedicated FPGA approach makes sense.

[gkasprow]

With dedicated FPGA we can put more channels to the board making i.e. quad servo with MCX connectors that would fit 3U panel easily.

[jordens]

From my experience the free tools are not in a state that would make me nearly as productive as ISE/Vivado.

[sbourdeauducq]

What did you try and what were the problems?

[jordens]

Banker and Humpback. Broken LVDS support, broken handling of platform pins (more broken than xilinx/ise), erroneous extraction and insertion of clock buffers that lead to false combinatorial loops, immature packaging and distro support, segfaults (more than vivado), assertion errors, missing analysis tools, less readable signal names to name a few. I am generally very much in favor of open tools and I'd really like to be able to invest time and money in improving them so that I can use them. But depending on them just to make an ideological point and not a technical one is not a good idea. To put the burden of proof that your proposal is a good one with you: What did you try that lead you to advocate for it?

[sbourdeauducq]

I haven't tried to push them myself (just compiled trivial designs), but I see many other people's projects and a very active community.

As for installation, just stop using Debian and switch to a better distro like Arch Linux or Nixos, then those tools are way easier to install (it's a single command) than Vivado on any platform. They don't bloat your HDD with dozens of GB of junk too. Bonus: you can leave a system untouched for months and then it will reboot properly after a fully-automatic upgrade. IME this systematically tickles Debian bugs :)

[jordens]

I'm not going to get involved in a my-distro-is-better than-yours peeing contest. I fought in distro wars 20 years ago. It's a waste of time.

[sbourdeauducq]

Fact remains that installing those tools on nixos or arch is a single command and not painful at all, unlike ISE or Vivado on any system. Those tools are very much in active development. Have you tried using the latest versions (yep, using a git-friendly distro helps), and have you tried reporting the bugs you were seeing?

[sbourdeauducq]

And you can also install nix on debian and install those tools with it just like on a nixos system. Then it's like a conda environment with more packages, more features and more speed, and without the constant stream of bugs and chaotic behavior.

[gkasprow]

I've selected CPLD instead of FPGA due to much lower and easier to control delays which are critical in this case. There are not many CPLDs on the market. New Altera CPLDs are in fact FPGAs with FLASH

[jordens]

@sbourdeauducq You have again managed to derail this to an airing of your well-trodden grievances. I'll leave it there.

@gkasprow That's what I noticed comparing the designs in Urukul and Banker as well. I would still stick with both choices (CPLD for Urukul et al. and cheap FPGA for Banker et al.).

[hartytp]

One possibility we've discussed here would be to consider unifying this with the "Sayma on a EEM" proposal. Because (a) once we Sayma v2.0 working well, that would allow us to largely reuse working hw/gw designs rather than beginning a new complex project from scratch (b) would provide a convenient path to users who need lots of EEMs, with only a small number of SAWG channels an no other uTCA hw (e.g. people who are happy using Fastino instead of Shuttler).

Comments:

• 220mm Eurocard
• DRTIO slave, running DRTIO over LVDS (needs rate conversion in gateware)
• Kintex-7 FPGA, size TBD once we've finshed Sayma v2.0/SAWG v2.0
• either 2 or 4 DAC channels depending on space, cost, power, etc
• DAC AFE copied from Urukul, no mezzanines, no upconversion (we'd make a simple EEM for that and run short external coax leads between boards)
• maybe use the 5MSPS LTC LVDS ADCs with a simpler faster AFE than Sampler
• copy clock tree from Sayma/Urukul (reference clock from WR/internal MCX/external SMA, on-board DAC clock PLL)
• in the long run, one could imagine replacing the JESD DAC + LVDS ADCs with an RF soc (lower power consumption and feedback latency). But, that might be better for the second version of the design.

Questions:

• what would channel cost be? (and, how would it compare to Sayma)?
• what would current consumption be?
• how tight would the space/routing constraints be?
• would it be sufficient to add PCB-mount heatsinks on the DAC + FPGA and add temperature sensors to cut off board power if either of these components got too hot (to protect the circuit in case of fan failure)
• what would the feedback latency be? Would it be low enough for the uses people have in mind? What analog feedback bandwidth do people actually require? 1MHz?

[gkasprow]

For such case one can use Sayma in a stand-alone uTCA box. Cost of such box will be <1k. I'm just producing 10 pcs.

[gkasprow]

In case of shuttler or RFSOC, one can design simple 6U CPCIS adapter that allows using such uTCA module in 6U crate. it would fit mechanically. 6U crate allows mixture of 6U and 3U CPCIS cards.

[gkasprow]

I was thinking about impementing DRTIO channel on CPCIS backplane. For this purpose we could use controller diff channels originally foreseen for slot 2&3 PCIe x8

[gkasprow]

Yet another idea that came to my mind recently. CERN wants to develop CPCIS controller based on ZynQ with FMC. The idea behind this approach is to provide support for multiple use cases and upstream interfaces:

We can combine Metlino and Kasli functionality in one module. FMC would be fitted with SFP cages or Ethernet PHYs. Some slots could be dedicated to EEM, some would be dedicated to DRTIO only. We plan to develop custom backplane for 19" rack with 8x8HP slots. Another idea would be a backplane with i.e. 4x 8HP EEM slots + 4 * 4HP EEM slots + 4 HP DRTIO slots.

[hartytp]

I see two main ways of implementing this:

1. An EEM with 2 fast LVDS DACs and 2 fast LVDS ADCs (maybe in the form of a SDR chip)
2. a standalone board, basically fast stabilizer. PoE as well as an EEM for real-time control from Kasli. The EEM interface could be e.g. a fast-ish SPI bus that exposes some FPGA registers.

I'd lean towards (2) but would be interested to hear others' views. One big consideration I can see is how much real-time control people are likely to want. If people want to do a lot of fast modulation of the lock params (e.g. sweep the set-point around quickly) then it might make more sense to go for (1) since then the lock params can be directly controlled over DRTIO without having to go through an SPI bus.

[hartytp]

IME a really strong argument for the fast-Stabilizer route is the AFE mezzanines. I think it's really nice that students can build a little AFE mezzanine that plugs into either servo, without having to re-implement all the features provided by Stabilizer: PoE; power supplies; microprocessor/FPGA to control simple peripherals; ADC front-end; CPU ADCs and DACs for logging, biassing things; high-quality firmware; etc. This makes it really easy for a student to make an AFE mezzanine for one of those cards to do a domain-specific servo (e.g. current servo, PDH etc).

[gkasprow]

What if we take Stabilizer and replace CPU with small ZynQ, ADCs, and DACs with 100MS/s ones? Of curse, we keep compatibility with mezzanines. ZynQ has also slow ADC. To make it easier to design and more flexible, we can use SOM instead of CPU. There are low-cost Trenz 4x5cm modules with Artix, SPartan, Kintex, ZynQ, ZynQUS that share the same pinout. They have Ethernet PHY as well. Trenz has also some open source ZynQ boards that could be directly integrated to the design.

[gkasprow]

It fits nicely

[hartytp]

What if we take Stabilizer and replace CPU with small ZynQ, ADCs, and DACs with 100MS/s ones?

Yes, that's what I had in mind.

The only thing I don't know is if we'd want to route more DIO to the mezzanine for the faster version?

ZynQ has also slow ADC.

That's useful. Otherwise, I'd add a slow I2C/SPI ADC + DAC to the PCB to replace the microprocessor ADCs + DACs.

[hartytp]

The things I don't have a clear idea on for this project yet -- and I'd welcome thoughts from other on -- are things like:

• What are the use-cases? Do people just want few MHz PID loops, or are there more complex use-cases out there?
• e.g. do people want to do PDH locks with this, using the FPGA to generate/demodulate the RF waveforms? Is there a strong reason to prefer that over a DDS-based solution like Pounder?
• What input/output levels do we need? +1V or +-10V? Do we need a PGIA front-end or can we use use a 2-channel DAC/ADC SDR chip with maybe a passive EMI filter and protection diodes as the only external front-end?

[gkasprow]

About the fast signals - what about adding one high-speed signal connector? We could do it for Stabilizer as well to push 200Mbit QSPI link over it

[hartytp]

What kind of connector do you have in mind? Will it be compatible with the current stackup? If so then this could be a nice idea, I guess....although 200MHz over pin header doesn't sound that scary to me, so are we sure this is actually necessary?

[gkasprow]

I meant that would work in 1GHz range. Some Samtec hermaphroditic connector. For Stabilizer it is not an issue, but for FPGA it is.

[hartytp]

Those are fine, but I think that making them work in the current stackup could be problematic. Anyway, let's open a new issue to discuss...

[hartytp]

@gkasprow what nice SDR chips do you know, e.g. with: compact LVDS interface; >=100MSPS; ideally 2xDAC and 2xADC (or, just 1xDAC and 1xADC fine too); ideally 16-bit; DC-precise?

[gkasprow]

I'm using AD9364 and chips from Lime Microsystem.They were not designed to be DC precise. There is some discussion about DC. These chips have RF path with IQ mixer so wouldnt espect any DC characteristics.

[gkasprow]

They also have minimum operating RF frequency so we can forget about DC

[hartytp]

ok

[jordens]

The last DC capable sdr adc dac combo chip that I remember was the one they used in the first usrp, the ad9866 iirc. But that's more like 12 bit.

[gkasprow]

Since I got funding, I can accelerate this design as well. Can we close the specification? My assumption is

• 100 or 125MHz 16bit ADC. Does it need to be low latency (parallel CMOS) or low pin count (serial LVDS)?
• 100 or 125MHz 16bit DAC. Here we have only parallel busses. Do we need some interpolation?
• Hardware compatibility with Stabilizer. The ADC range would be limited to 1V
• 10/100/1000T Ethernet with PoE and USB interfaces
• full compatibility with Stabilizer mezzanines (3V3 signaling)
• the utilisation of low cost Trenz FPGA 4x5cm modules. One can choose Artix, ZynQ and Kintex devices
• power supply, PoE, low-speed ADC and DAC with similar performance as in Stabilizer
• optionally some high-speed connector to mezzanines with routed LVDS + gigabit transceivers
• SATA?

Would there be any use cases for this board providing that we have Stabilizer, Fastino and Pounder? Who is going to fund ARTIQ support?

[jordens]

• Design for overall low latency (AFE+ADC+bus+bus+DAC+AFE). Choose digital signalling based on that.
• No interpolation needed.
• There is also Enclustra for some nice FPGA modules. Put them on your list of things to look at. Especially for Zynq US+ and for small numbers where FPGA sourcing is a pain.
• AFE need to be significantly faster and wider bandwidth. Goes up as the sample rate does. Otherwise the latency game (or direct sampling and digital downconversion) is pointless.
• One use case: phase metrology where low jitter ADC and high sampling rates gives you low noise.
• The fact that we are trying to do both DC accuracy and stability and very wide bandwidth is a bit worrying to me though.

[gkasprow]

Maybe it's a time for general purpose FPGA FMC card?

[jordens]

Sure. Trenz or Enclustra should do that. But if you don't need front panel real estate isn't a so-dimm or something custom as the existing modules much simpler?

[gkasprow]

I'd use something that gives the user migration between families. With 4x5 you can use Microsemi and Xilinx FPGAs and SoCs. And there are a lot of them, some are available in DIgikey. SO-DIMM is usually custom design since there is no de-facto standard. In the case of ARM/Intel CPUs, there are nice standards like Q7, SMARC or COM, but for FPGAs, the only standard that has more than one vendor is Trenz 4x5cm. The advantage of SO-DIMM is a very low-cost connector. Enclustra has also Altera FPGAs and SoCs. Maybe other vendors also have Enclustra-compatible modules, I didn't look at that.

[gkasprow]

We are building a Space-VPX module with 4x5 Microsemi FPGA and FMC. It could be quick work to convert it to 3U. Of course, if there is such use case :)

[gkasprow]

There is also open source design that uses Enclustra SO-DIMM modules.

[jordens]

Yes. My point was just that FMC seems to be designed to offer you front panel real estate, i.e. is more suited to "front ends" while an FPGA module can live somewhere in the back and doesn't need what FMC gives.