[RFC] New AMC: Shuttler (high-speed multi-channel DAC)

[marmeladapk]

From @hartytp on 2017-08-16 05:24

I've started sketching out requirements for a fast DAC to be used for ion shuttling/splitting. It's on the Wiki.

All comments welcome...

[marmeladapk]

From @gkasprow on 2018-04-25 12:33

µTCA power is definitely not nice, as demonstrated by Sayma not being

powered up when placed into the crate.

@gkasprow https://github.com/gkasprow please can you confirm the absolute minimum that needs to be done to power up an AMC.

Absolutely nothing. One can force the MTCA power supply to deliver 12V even without MMC!. @sbourdeauducq did you try our recent MMC ? It works well with NAT MCH. And it fixes 1V8 problem. Anyway, this 1V8 problem was solved long time ago. Exar chip can be upgraded by a programmer which needs Windows machine. Since it is so serious limitation, we implemented automatic upgrade via MMC. My initial MMC code was never supposed to get power from MTCA crate. It was designed as a simple code to init power and PHY chip. We ported OpenMMC so now it does get the power from MTCA. There may be some issues still, but that's why Joe sent us all the HW to solve them at once. It's hard to solve all possible issues when you have only one set of HW.

Let's say you build Eurorack system which has hundreds of power hungry channels. And at certain moment a fan dies. Or the power supply starts delivering too high voltage.Or the dust blocks the air flow. You won't even notice when expensive electronics gets overheated. Without proper thermal management iut's very difficult to build reliable system. A few generations of complex electronics before, I developed my own custom systems. Because I thought that MTCA or VME is expensive. They are still installed at CERN, GSI, JET and a few polish universities and I spend really plenty of time maintaining them. I mainly replace PSUs, fans, overheated boards, broken panels, connectors that went off the PCB. Now I'm smarted and use only standard platforms because they are much cheaper in longer perspective.

[marmeladapk]

From @hartytp on 2018-04-25 13:05

Well said @gkasprow. In my experience, there is a very large (near complete) overlap between the set of people who think that designing their own chassis standard/using an exotic, non-standard chassis is the right solution for their project, and the set of people who have never tried to do that.

EuroCard works for EEMs/Kasli because it is explicitly low BW (SPI), low complexity, low power consumption. I would not extrapolate from that to assume that it will work for complex systems.

[marmeladapk]

From @hartytp on 2018-04-25 13:14

@dhslichter to be completely honest, my impression is that the scale of complaints about uTCA/MMC are way way out of proportion to the scale of the problems. The major issue has been communication/coordination between the various parties and the fact that people are working on many projects at once.

As @gkasprow says, once he received a board that demonstrated the 1V8 problem (which, for reasons I still don't fully understand was months after the problem was known about) he had it diagnosed and had found optimal loop parameters within about a day. After that, a solution was available if one wanted to update the regulator directly. It took an few weeks after that for a version of the MMC firmware with the update to arrive. I've installed that updated MMC on my board with no problems (took 15 min).

[marmeladapk]

From @hartytp on 2018-04-25 13:17

Anyway, no Greg's team are working hard to solve all remaining issues with MMC etc and I think we should give them the benefit of the doubt while they work on that. If they find a really good solution then I think we should seriously consider keeping Sayma AMC as is, instead of starting again from scratch and beginning a whole new round of debugging.

If M-Labs really feel that they cannot work with Ultrascale FPGAs, then we may need to consider moving away from them, but that is something that would have to be done on Sayma as well, so it's not shuttler specific.

[marmeladapk]

From @vdirksen on 2018-04-25 13:29

People in the field love it, if developer spend some extra money in the hardware and software and thoughts of life time maintenance. You save only a little in the BOM (build of material) compared by the costs for unexpected service costs. Yes, life for developers gets harder for one year, but life for service people gets easier for several years up to decades. ;-) We have VME, cPCI, PMC, PCI, PCIe and custom boards beside AMC/MTCA in the field. We enjoy the support cost savings with MicroTCA systems. We have more than 12000 MCHs in the field. We save a lot of money and get a better reputation due to better and quicker support.

[marmeladapk]

From @dhslichter on 2018-04-25 16:16

I was hoping to wake up this morning to find 50,000 additional comments on this thread, thanks @hartytp @sbourdeauducq for not disappointing me ;)

Based on all of these discussions, I think it would make sense to make Shuttler an AMC or RTM card for uTCA. I am operating on the assumption that NIST will fund this, but obviously if someone else does then they can make it whatever they want.

I stand by the notion that I don't want to involve gigabit transceivers in the design if at all humanly possible. My current notion for maximum simplicity would be to have Shuttler be an AMC card with a large Artix-7 or a Kintex-7 (same as KC705 perhaps) and all the DACs on the same card, with a single AFE card connected by reverse-polarity FMC. This also means we have one board instead of two, and we save the cost of two Artix-7 FPGAs (plus that of having two boards with all the attendant expenses) if we put it all one one board.

[marmeladapk]

From @gkasprow on 2018-04-25 17:05

@dhslichter why not to keep same mechanical and electrical layout of the AFE board as on Sayma RTM. We decided to go for longer version of FMC, so can unify it a bit. Since the Shuttler is going to have 16 channels, we can make unified designs of AFE boards and reserve 16 diff pairs in the connector. In this way, when needed we could use 4-channel MixMod in Shuttler (with liited channel count) or Wide-band, DC-coupled Shuttler AFE in Sayma (with half of assembled channels)

[marmeladapk]

From @hartytp on 2018-04-25 19:14

@sbourdeauducq I know that in the past there has been a feeling that ideas aren't properly listened to. I don't want that to be the case here.

So, to be clear, I'm not saying that the points you make aren't valid, just that I don't see most of them as criticisms of uTCA as much as being criticisms of Sayma (e.g. uTCA does not need MMC, etc). As such, this thread isn't the right place to air them. And, yes, I agree that uTCA is a bit crufty and, in many ways, overkill for what we want to do. But, as Greg and @vdirksen have pointed out, it's actually really hard to find a better alternative that's really reliable and well suited to the kinds of complex systems we want to build. Obviously, if you can develop something better then that's great, but I'll be skeptical until I see it.

[marmeladapk]

From @hartytp on 2018-04-25 19:22

I stand by the notion that I don't want to involve gigabit transceivers in the design if at all humanly possible. My current notion for maximum simplicity would be to have Shuttler be an AMC card with a large Artix-7 or a Kintex-7 (same as KC705 perhaps) and all the DACs on the same card, with a single AFE card connected by reverse-polarity FMC. This also means we have one board instead of two, and we save the cost of two Artix-7 FPGAs (plus that of having two boards with all the attendant expenses) if we put it all one one board.

@dhslichter That's a perfectly respectable way of doing this.

FWIW, now that we're committed to using RTMs for Sayma, my personal preference would still be to do it the way outlined above with an RTM + SERDES FPGA. My thoughts are that:

• the thermal management (the high-power FPGA is not on the RTM) and EMI benefits of the RTM are quite nice
• Even if the new Shuttler AMC is simple, it's quite likely to take a lot of time to get all the bugs out of if. Even if Sayma is a bit over complicated, once it's working it still makes more sense to use it than to start again
• I'm not convinced that the all-on-an-AMC approach is actually cheaper. Yes, one board is cheaper than two, but two boards will give more channels per board. So, if you need a lot of channels, they come out to the same cost per channel (which, in any case, should be dominated by silicon costs anyway)
• Generally, I'm pro homogeneity in the Sinara ecosystem. Introducing another complex FPGA board that has to be produced, debugged, supported and stocked is a lot of extra work, reduces bulk-buy discounts etc.

Having said all that, if you're really going to fund this and do all the work involved in project management, bug-finding/fixing etc then great! We'll use whatever you produce gladly, even if it's not how we would have done it.

[marmeladapk]

From @hartytp on 2018-04-25 19:25

@gkasprow I'm not sure that proposal works. @dhslichter wants to have all 16 DAC channels on a single AFE, with the output on a single SCSI connector. For mix-mod we want to have 4 DAC channels (2 RF channels) on the board, with the outputs on SMAs. Also, mix-mod has ADC inputs. I think the differences here are big enough to warrant a new design (although, there should be a lot of overlap, like using the same form-factor).

[marmeladapk]

From @gkasprow on 2018-04-25 21:17

At lest we should make the pin assignment in such way that plugging AFE to wrong carrier board, won't damage it.

[marmeladapk]

From @gkasprow on 2018-04-25 21:18

btw, my grad student is working on this RTM board since some time and expect the schematics for review very soon.

[marmeladapk]

From @hartytp on 2018-04-25 21:25

btw, my grad student is working on this RTM board since some time and expect the schematics for review very soon.

Amazing!! Can't wait to see it. :)

Is that an RTM to work with Sayma AMC? Is that wired up as discussed, with the BP GTX lines connected to the RTM FPGA?

[marmeladapk]

From @gkasprow on 2018-04-25 21:34

Yes. We can make the development cost much lower and build it on ADI JESD204b core. Then we can treat it as a kind of ASIC chip from ARTIQ point of view :)

[marmeladapk]

From @tprzywoz on 2018-04-25 21:46

@hartytp yes I'm designing Shuttler compatible with Sayma AMC. Two Artix-7 - so 2 x 8 GTP lines connected to GTX on AMC. I need about a week of work but we have holidays now. In the worst case I think all schematics will be ready on 8-9 May.

[marmeladapk]

From @sbourdeauducq on 2018-04-26 02:38

Let's say you build Eurorack system which has hundreds of power hungry channels. And at certain moment a fan dies. Or the power supply starts delivering too high voltage.Or the dust blocks the air flow. You won't even notice when expensive electronics gets overheated. Without proper thermal management iut's very difficult to build reliable system.

Considering how much of an annoyance it is just to get a Sayma powered (fact remains, this has been a problem and it dragged on for months) I suspect this requires a monumental amount of yak-shaving and careful testing before all the µTCA monitoring and protection systems work in a useful and reliable manner.

Isn't this effort better spent designing a cruft-free alternative?

[marmeladapk]

From @sbourdeauducq on 2018-04-26 02:59

So, yeah, they're great for a simple system with a few EuroCards, but not what I'd want to build a system of 100+ channels of high-speed DACs out of, which is what we're talking about for a complex ion trap like a HOA2.

You are limited by the front panel connector density anyway (unless you're using expensive connectors like MDCX/MDHC). With SMA connectors, the FPGA/DAC density cannot be that high and I believe a Kasli/Eurorack type of design is likely OK and much less expense and hassle.

[marmeladapk]

From @sbourdeauducq on 2018-04-26 03:08

Even if the new Shuttler AMC is simple, it's quite likely to take a lot of time to get all the bugs out of if.

Putting in a 7-series FPGA plus designing the SERDES gateware doesn't sound simpler than putting in a bigger 7-series FPGA and DDR memory and doing away with the RTM cruft, while reducing gateware latency and complexity at the same time.

Is DDR memory and a bigger FPGA producing that much EMI that it has to be put across the AMC/RTM boundary?

And don't expect ADI JESD "IP", or any "IP" for that matter, to work without hassle. Plus the code of "IP" is typically unmaintainable trash, so if you have any bug with it, good luck! (especially if you have no support contract with the vendor). Note: I have not carefully reviewed ADI IP, it may be OK, but typically IP is not OK. (did you know IP actually stands for "Intellectual Poverty"?)

Then we can treat it as a kind of ASIC chip from ARTIQ point of view :)

The poster child for such an ASIC chip would be the Sayma Ethernet PHY.

[marmeladapk]

From @sbourdeauducq on 2018-04-26 03:41

Desy Techlab https://techlab.desy.de. Talk to the people maintaining the MTCA

Are the people behind this website actually developing and maintaining µTCA systems, or are they marketing and "technology transfer" bureaucrats? I've actually talked to one DESY engineer and we somewhat agree about µTCA.

[marmeladapk]

From @gkasprow on 2018-04-26 09:15

Guys from Creotech are using ADI JESD on Sayma and are not complaining. So my student will try to implement it and add DAC interface. I'm pretty sure he will succeed. Did you read what I wrote about the MMC? The code I developed was not MMC! It was only some test firmware. Open MMC is the right one. Creotech sold hundreds of AMC boards with MMC to tens of customers. They use Open MMC. We ported the firmware to very similar CPU. There are some issues for sure, but once they are solved, this will work. Creotech will maintain this MMC later on.

[marmeladapk]

From @vdirksen on 2018-04-26 09:16

The Desy MicroTCA Techlab have a team of FPGA, BSP, MMC firmware and Linux design engineers. Otherwise I would not recommend it in this round.

[marmeladapk]

From @gkasprow on 2018-04-26 09:17

@sbourdeauducq As far as I remember, Joe started testing our OpenMMC a few weeks ago in MTCA crate. It did not work because he burned Flash image in wrong way. It was not a problem with MTCA nor code itself.

[marmeladapk]

From @gkasprow on 2018-04-26 09:19

@sbourdeauducq I know many of these DESY people personally. They are very closely cooperating with WUT and two guys are already involved in Sinara.

[marmeladapk]

From @hartytp on 2018-04-26 09:19

That all sounds great Greg! If you can supply the RTM FPGA pre-programmed with working JESD then that removes a huge amount of software development!

You'll probably also need a few things like SPI buses that ARTIQ can control to configure clock chips etc. What about having one RTM FPGA that is a dedicated JSED->LVDS converter which, as you say, is essentially an ASIC with gateware that you develop, and then having a second FPGA which ARTIQ controls (this would be the similar gateware that is on the Sayma RTM FPGA)?

[marmeladapk]

From @hartytp on 2018-04-26 09:20

Also, @gkasprow do you guys have enough experience with ARTIQ to try to reconfigure the Sayma gateware to work with Shuttler, at least for a demonstration?

[marmeladapk]

From @jordens on 2018-04-26 09:46

My 2¢:

µTCA is good for this once we are talking about more than 10 W per card or more than 16 channels in total. The idea that µTCA is overly expensive and unreliable is unsubstantiated. As long as e.g. a basic MCH is as cheap/expensive than Kasli I fail to see how that can be called too expensive. And faced with the overhead required to ensure proper power supplies, power distribution, mechanics, shielding, cooling when looking at using classic Baugruppenträger for Shuttler, I'd pay for a µTCA crate any day. AFAICT all problems with Sayma are in fact problems with Sayma (specification, design, hardware, gateware, software, toolchain) and not with µTCA. Sure, bad MMC code is bad. But so is a bad networking stack. Neither implies that µTCA is bad.

I'd recommend trying very hard to avoid complexities like serwb, or splitting into AMC/RTM (at least until the AMC side is rock solid), or having mezzanines (unless absolutely necessary), or radical ideas like using a "passive" AMC to loop power and GT links from the AMC backplane to the RTM connector. IMHO a lot of thought should go into the front panel, connectors, and whether mezzanines are really the way to go. Are the specifications settled well enough to lock in the DAC, interface, and FPGA choice?

[marmeladapk]

From @gkasprow on 2018-04-26 09:47

@hartytp I wrote that we can try. I don't know if we succeed with limited resources. There is no funding and we do it as a hobby project :)

[marmeladapk]

From @hartytp on 2018-04-26 09:51

@jordens I'm in agreement with all of that.

@gkasprow ACK. Well, it's a fun hobbyist project! I'll be very interested to hear how you get on.

[marmeladapk]

From @dhslichter on 2018-04-26 20:08

@jordens I agree with all you've said -- my comment here https://github.com/sinara-hw/sinara/issues/253#issuecomment-384345342 seems to be in line with most of it. It seems that cutting things down to maximize simplicity is the best way to go. AMC/RTM communications, gigabit transceivers, make-your-own-SERDES code with necessary timing, all are complications, and my comments above have been about trying to remove these.

I would also be fine doing away with the AFE for Shuttler, upon further consideration. If we are shooting for 125 MSPS output from the DACs, we wouldn't constrain ourselves that much by just choosing an on-board output buffer stage and not having a detachable/configurable AFE. The thinking previously had been that one might want to do trap voltage step-up (e.g. from +/- 1 V to +/- 10V or thereabouts) on an AFE, but this is going to add a lot of power dissipation and complexity. In this case, one loses all of the headaches with AFE mechanical design and layout as well.

@gkasprow I didn't realize you had someone actually developing a Shuttler design -- had I known that, I would have been interfacing with you to discuss these sorts of things earlier. I am eager to see what has been put together when it comes out, but perhaps much of it might have to be changed if we decide not to do RTM, for example...

[marmeladapk]

From @hartytp on 2018-04-26 20:18

It seems that cutting things down to maximize simplicity is the best way to go. AMC/RTM communications, gigabit transceivers, make-your-own-SERDES code with necessary timing, all are complications, and my comments above have been about trying to remove these.

So is reinventing the wheel for every new project. There is a huge advantage to reusing Sayma AMC hardware and gateware for this if we can.

And, as Greg points out, it's not make-your-own-SERDES, it's sticking an ADI/Xilinx JSED204B core onto an FPGA to provide a JESD204B interface to the DACs. Once that's done, we can reuse all the gateware that's already being carefully debugged on Sayma.

@gkasprow I didn't realize you had someone actually developing a Shuttler design -- had I known that, I would have been interfacing with you to discuss these sorts of things earlier. I am eager to see what has been put together when it comes out, but perhaps much of it might have to be changed if we decide not to do RTM, for example...

If Greg wants to do it this way, let's give him a chance. If the board works nicely, what's the point in changing it all and starting again because of gigabitaphobia?

[marmeladapk]

From @dhslichter on 2018-04-26 22:06

Proposed specification for Shuttler Here I have tried to collect/synthesize the things that I think make sense from above, and where people seem to have come to some agreement. Please let's try to keep this thread focused on Shuttler design.

• AMC card form factor
  • Some have expressed the opinion that being able to use an existing, debugged AMC card design (Sayma AMC) is valuable, and I agree. However, I think the difficulties with roll-your-own-SERDES over AMC/RTM, and other points elaborated above, mean that there are still substantial costs to using an existing AMC card. Also, this means we do NOT need to use gigabit transceivers for communications other than DRTIO tasks.
• Kintex-7 FPGA XC7K325T-2FFG900C (same as KC705)
• 16 channels of DAC: LTC2000-16
• Gateware to be 16 channels of phaser running at RTIO clock frequency (125 MSPS or 150 MSPS)
• DDR RAM: the purpose here would be to enable more frames/deeper splines for phaser waveforms.
  • @jordens what sort of requirements would one want for this? Would 32-bit be sufficient? I feel like we don't need a ton of RAM, but perhaps there is not enough available in the FPGA itself?
• Analog outputs of DACs to be buffered with unity or low gain (up to G=5) by ADA4927 (provisional, although I haven't been able to find a superior option for differential-to-differential buffering in terms of noise and bandwidth).
  • If we run at some small finite gain, then we improve the input-referred noise of the buffer amplifier (less SNR degradation of DAC output) and we also make the output signal even more noise-tolerant once it reaches the outside world. Gain of 5 would give us differential voltages swinging between +/- 5V (max voltage on any output line +/-2.5V). Have to consider the power dissipation ramifications.
• Buffered differential outputs go to SCSI connector on the front panel -- 16 differential pairs.
  • Alternative connectors can be considered, but should have controlled-impedance differential pairs with shielding.
• SFP cage to enable running as a standalone DRTIO slave over fiber
• Front panel SMA for external DAC clock option?
  • This enables operation outside of an ARTIQ context, as well as risk mitigation/fast startup in the absence of stable DRTIO clock source.
  • Do we want to just make it WR compatible?

[marmeladapk]

From @dhslichter on 2018-04-26 22:15

If Greg wants to do it this way, let's give him a chance. If the board works nicely, what's the point in changing it all and starting again because of gigabitahpobia?

If it all is going to hang together, then I am OK with it. Note that I didn't say "stop working and scrap everything".

So is reinventing the wheel for every new project. There is a huge advantage to reusing Sayma AMC hardware and gateware for this if we can.

That's why my proposal involves using phaser code on KC705's model of FPGA directly outputting to slow parallel buses. Seems like that reuses a lot of already-existing stuff. We can duplicate KC705's RAM as well if desired because that is also already figured out. Yes, there will be board routing, but that has to happen no matter what (AMC or RTM).

[marmeladapk]

From @gkasprow on 2018-04-26 22:31

At the moment the schematics are nearly done. PCB is not started yet. The design is based on two biggest Artix devices (200T) with small SDRAM attached to each of them. Since Kintex 325T and Artix 200T have nearly same banks layout and number of pins, we can at this moment replace one with other. Some time ago I replaced Artix with KIntex and it was straightforward. If we get rid of mezzanines and complex power supply, we could try to fit it on AMC. But we loose flexibility of mezzanines of course. And Kintex chips are far more expensive than Artix. At the moment the clock distribution and supply was copied directly from Sayma RTM. The design @tprzywoz does was discussed a few months ago and I thought that we found a consensus :) Another issue is that @tprzywoz needs to finally finish his master thesis and he already seriously modified schematics because we changed the concept. I don't want him to do it once again...

[marmeladapk]

From @dhslichter on 2018-04-26 23:12

@gkasprow thanks for the update. Does the board need to be produced for @tprzywoz to finish his thesis? Or is design/layout/simulation sufficient? All I saw in the old comments was indications that @tprzywoz was thinking about this, but not that this was a thesis project or that design had started, so I'm sorry if I missed it somewhere else.

What do you mean by "complex power supply"? We would need analog and digital 1.8V and 3.3V rails for the DACs, and +/- 5V rails for the buffer amplifiers, plus FPGA supplies.

I was thinking with the Kintex chip that one could try your two-to-one or three-to-one "hack" multiplexing of the FPGA outputs. Another option (quite possibly crazy) might be to use 1.8 LVCMOS and translate to LVDS with external chips, which would let you put up to 3 DACs per bank. If we are running this at max 125 MSPS then the SI should still be OK, but obviously one has more EMI issues.

[marmeladapk]

From @dhslichter on 2018-04-26 23:27

gigabitaphobia

@hartytp that's rather a snarky remark -- it's pretty clear to me that a design that only involves slow parallel LVDS, with no gigabit transceivers, is going to be less complex than a design that involves both. I am trying to simplify and reduce the number of moving parts and things that have to come together for the design to work. The gigabit transceiver stuff has been a demonstrated pain in the neck so far with Sinara, and because the receiver is not implemented, we would be using someone else's IP cores. I don't know how well those will perform, what corner cases they won't handle well, what level of synchronization one gets, etc. -- in general, most applications that use gigabit transceivers (communications/digital video) don't care at all about microsecond latency variability, brief link losses, bit alignments to an absolute time, etc, and so there may be design choices that have been made which won't work for us. It is entirely possible that God's Perfect Gigabit IP Core With No Bugs exists out there, and will work perfectly for us, but based on everything I have seen so far, there are various things we might care about where the spec is either not good enough or not quoted at all.

Now, it appears that the ADI JESD203B cores would be able to do what we want to do, but how much yak shaving is involved in getting them to work, integrating them with everything else, and so on? What happens if they are doing something wrong/unexpected? I would rather just not have to deal with any of this hassle whatsoever, by just generating the waveforms on the same FPGA that will then output them to the DACs. We are going to have to have these large, slow parallel LVDS buses one way or another. Why add another layer of complexity on top?

[marmeladapk]

From @sbourdeauducq on 2018-04-27 00:40

While I dislike this SERDES RTM for the reasons Daniel mentioned, I also don't see why it would need any SDRAM. Another thing I dislike about µTCA is that it encourages a culture where it is OK cram a lot of unwarranted complexity and components into boards. When I look at Sayma I see that we at least managed to avoid Zynq Ultrascale+ and that proprietary SERDES switch chip that would have gotten in the way of DRTIO, but there is still a long way to go...

[marmeladapk]

From @dhslichter on 2018-04-27 01:24

The only reason for SDRAM would be to extend the complexity of possible waveforms for playback by the phaser gateware. I am not sure what the exact RAM consumption on the FPGA itself would be for the phaser gateware engine (I think this consumes minimal RAM, though) vs waveform storage, but for example on the PDQ boards we are very limited in what we can play back because all the spline coefficients need to be stored in block RAM on the FPGA, and there is not much of it. The PDQ FPGA (XCS3500E) has 360 Kb of block RAM; the XC7A200T has 13,140 Kb of block RAM (for reference, the XC7K325 has a smidgen more at 16,020 Kb). We have determined that for our near-term purposes, the PDQ design does not have enough memory storage to run 3 channels, so we have allocated all of that to one channel. If each XC7A200T is running 8 channels of DAC, and it has about 40x the block RAM of the PDQ, then we expect that just with block RAM we could have up to ~5x as much waveform data as there is currently available in the one-channel PDQ design, or 15x as much as in the (insufficient) three-channel PDQ design. This is not accounting for the fact that phaser actually runs more splines (two DDS channels instead of one for PDQ), and so you may well need more memory for a given waveform duration because the waveforms are a bit more complex. This is all a task for @jordens to make definitive pronouncements, because he is the designer of both the current PDQ gateware and the phaser gateware. However, it seems to me that it is likely that without some additional external RAM, we would run up against resource limitations fairly soon if we wanted longer waveforms, more complex waveforms, and/or more waveform branches. Adding an external SDRAM would provide essentially infinite storage, so we wouldn't have to worry. We would need to ensure that the memory bus is sufficiently wide/fast to serve the required data to 8 channels of phaser. Again, here @jordens should comment on the throughput requirements.

[marmeladapk]

From @dhslichter on 2018-04-27 01:37

it encourages a culture where it is OK cram a lot of unwarranted complexity and components into boards

Beyond uTCA, we have been fighting this battle more broadly as well. No less a personage than @hartytp has inveighed against mission creep, overly complex design, and trying to be everything to everyone, and his points are well taken. My desire to avoid gigabit transceivers and Sayma AMC cards when running Shuttler has a lot to do with this philosophy, and the discussion about possibly going to Eurocard format above (an option which I have since thought better of, thanks to various insightful comments in this thread) was an attempt to ask "what complexities can we strip off this thing and still have it run?" The idea of generating waveforms on the same FPGAs that output them directly to DACs over slow LVDS parallel buses is another example of trying to strip complexities that are not necessary. One certainly could run this as an RTM card off a Sayma AMC using roll-your-own gigabit data links, but if there is a less complex way of achieving the goal (e.g. just make an AMC, perhaps with a Kintex-7 or perhaps with two Artix-7, and put the DACs right on there with no AFE mezzanine either), it seems that we need to a have a strong, compelling reason why the added complexity is necessary for the function of the device. So far, I haven't seen that -- yes, much of the gateware is already written for Sayma AMC, but this simpler solution already has the gateware written for it too (namely, phaser).

Things that could be showstoppers for a Shuttler AMC card:

• too much EMI from the FPGA(s)? I don't see that this is likely to be the case, and we can shield the FPGA and SDRAM if needed under a perforated lid, but worth understanding.
• too much power dissipation? The DACs will be about 8 W, and the buffer amplifiers considerably less. The FPGAs will be putting out roughly as much heat as Sayma AMC if not less, so this extra dissipated power represents a small increase in dissipation over the current Sayma AMC, which we say is "good enough" (current budget for Sayma RTM is ~100 W, Sayma AMC is ???)
• not enough physical space to run as many DACs as desired? We are currently planning for 16 DACs, and could revise that down a little (maybe to 14 or 12), but less than that would suggest we should look for a denser solution. It sounds like this is manageable, though.
• not enough pins to run the number of DACs we want along with any SDRAM we feel is needed? If we use a single Kintex-7, we would have to resort to some hacks. If we use two Artix-7, this seems not to be a problem.
• it's just too hard to make/debug a new AMC card design: I would hope that @gkasprow's previous track record, plus the lessons we have learned from Sayma AMC, would mean that this isn't the case, especially if we are not trying to do anything exotic or wildly different from existing/previous designs. Yes, we will probably have to fix some silly issues after the prototype; this is true for all boards of this complexity, though.

If any of the above things are truly showstoppers for making a Shuttler AMC, but are solved with high probability for a Shuttler RTM coupled to Sayma AMC where the Sayma AMC streams the data over gigabit pipes, then one should seriously consider the latter solution. Otherwise, it seems hard to defend the more complex solution.

I would note that one can run a Sayma AMC coupled to a Shuttler RTM and still have all the waveform computation done on the RTM card -- this was my original proposal and still seems like it would give us a lot of the simplicity we want. The gigabit links between Sayma AMC and Shuttler RTM would just be used to pass DRTIO through to the RTM.

[marmeladapk]

From @sbourdeauducq on 2018-04-27 02:38

I would note that one can run a Sayma AMC coupled to a Shuttler RTM and still have all the waveform computation done on the RTM card -- this was my original proposal and still seems like it would give us a lot of the simplicity we want. The gigabit links between Sayma AMC and Shuttler RTM would just be used to pass DRTIO through to the RTM.

That's a very expensive backplane connector.

Note: my remark was about the SDRAM on the "one RTM FPGA is causing problems so let's have two" SERDES/DAC RTM; I see why it can be useful to have SDRAM somewhere on Shuttler.

IMO µTCA is vaguely acceptable if we don't have a RTM, everything is on one card, there are zero MMC shenanigans, complexity is controlled in a totalitarian manner throughout the whole project, and any complex new chip (e.g. FPGA, Ethernet PHY, DAC, programmable voltage regulator, PLL, MCU) is treated with the utmost skepticism, i.e. assumed to be a trash fire of obscure bugs until proven otherwise (e.g. using a development kit very carefully). Also, it's better if we can do away with a maximum of MCH hardware. We don't need an Altera FPGA, an ARM SoC, and a gigabit Ethernet switch to monitor a few voltages and temperatures and spin up a couple fans. And those "tongue" connectors look very expensive.

[marmeladapk]

From @dhslichter on 2018-04-27 05:57

That's a very expensive backplane connector.

Yes, which is yet another reason why I think we should make Shuttler an AMC card :)

[marmeladapk]

From @gkasprow on 2018-04-27 12:57

@sbourdeauducq we used this PHY chip only because you refused to make Ethernet working over FPGA serdes (exactly the same as Kasli does). This PHY makes Ethernet 1000Base-X avaialble over RGMII. And I thought that it also does speed translation which was not true. So in next revision we will remove the mux. SCANSTA is also only to make JTAG switching easier. uTCA has nothing to do with it. The only chip that is required by uTCA is MMC processor and i2C connectivity. Other ports are optional.

[marmeladapk]

From @gkasprow on 2018-04-27 13:09

And programmable power IC seem has one advantage - many hardware issues can be fixed by single firmware update. At certain point, when ceramic capacitors used in the design degrade with time (they always do, can be reset by heating only), standard regulator may become unstable. Programmable one can be upgraded, loop bandwidth can be modified and the boards starts working again with firmware upgrade. I switched to such programmable ICs due to bad experience with standard switching regulators in large scale projects. If you have just a few modules, you don't care. But if you have plenty of them distributed across the world, this really saves a lot of maintenance. Every regulator on your computer motherboard is programmable due to various reasons and there were strong arguments why manufacturer use ones. And it saves plenty of components related with i.e. power sequencing. Look at the power supply of KC705 and Sayma. The second has a few times less components.

[marmeladapk]

From @sbourdeauducq on 2018-04-27 13:46

I'm not assigning personal blame here, only devising part of an approach that would hopefully prevent the Sayma problems from happening again. All I'm advocating for is a very careful test of each new complex chip before committing to them. I am certainly guilty of not having done that myself. e.g. I should have vetoed DDR3 (since it could not be tested on KCU105) and tested more parts of the design on the KCU105. I do remember the KC705 power supply design and its ARM processors and finding it ridiculous. Unsurprisingly, it is also somewhat buggy. I'm not convinced a few dumb SMPS controllers would not have done a better job.

[marmeladapk]

From @hartytp on 2018-04-27 14:15

that would hopefully prevent the Sayma problems from happening again.

Well, as an example, it's starting to look like a not insignificant fraction of our Sayma problems were due to the HMC7043 randomly injecting noise in to the AMC FPGA clock tree. No changes anyone has suggested would have prevented that. These problems always come up on designs of this complexity, and would still have happened if Sayma were an AMC, EruoCard or whatever.

[marmeladapk]

From @gkasprow on 2018-04-27 14:16

Moreover, we would not detect it with two or three prototypes produced. Only because we produced 10 pcs, such problem emerged.

[marmeladapk]

From @gkasprow on 2018-04-27 14:19

OK guys. I'm not against AMC approach. I wanted to reuse as much as possible the RTM design. We can go for AMC side, with 2 Artix chips and simplest possible connection between FPGA and DACs. Then we get rid of mezanines, +/-12, +/-6V supply which simplifies things a lot and makes it feasible on AMC side. We add simplest possible amplifiers and SCSI connector. We can leave the HMC7043 clock tree, but stripped to minimum.

[marmeladapk]

From @gkasprow on 2018-04-27 14:21

Each FPGA has 16 GTPs, two can be routed to PORT0/1 where we reuse Kasli Ethernet controller Then another one or two would go to SFP cage, next four would be connected to the FatPipe. What about remaining GTPs? Any idea how to use them? Maybe connect to the PTP links?

[marmeladapk]

From @gkasprow on 2018-04-27 14:26

Yet another issue with 2 FPGAs is how we connect them together and provide the data flow :)

[marmeladapk]

From @gkasprow on 2018-04-27 14:27

@tprzywoz must build physically the board and make it running, so let's simplify his job as much as possible.

[marmeladapk]

From @dhslichter on 2018-04-27 16:21

@gkasprow ouch, that is a tall order. Timescale?