Sayma ADC

[hartytp]

Summary of conversation:

• [ ] DNP Sayma ADCs and all components they are using (regs, SMPs, etc, etc)
• [ ] Make sure we have enough IO on the AFE digital mezzanine to run a 1MSPS SPI ADC over the mezzanine.
• [ ] Decide how we want to route the IO from the digital mezzanine to Sayma AMC. The routing should have deterministic and low latency (i.e. try to avoid transceivers).
• If removing the ADCs completely will help in any way (e.g. if we need the board space for something else) then we should reconsider doing it.

---

Okay, I'm going to make a somewhat radical proposal here, but: should we remove the Sayma ADC?

• It's quite expensive and consumes quite a bit of power
• It's untested and if we do test it, it makes the test suite more expensive
• Removing it allows us to simplify the routing a lot
• Having a fast JESD ADC is not that useful anyway, since the latency is so high. e.g. the feedback bandwidth one can get is not significantly higher than with a 1MSPS SPI ADC
• There aren't that many applications that really need that kind of BW.
  • I don't realistically see Sayma being used for PDH locking (it's way too expensive and not optimised for that job).
  • Thinking about the details of how demodulating a signal would work (in the context of Nanonga AFE) I can't see a good way of demodulating signals using this ADC. (it's not fast enough to directly digitize the the RF produced by the DAC). So, while demodulating signals seems like a nice idea in practice, I don't think we can do it using the ADC on Sayma.
  • To use this bandwidth e.g. for demodulating signals requires some quite fancy gateware. This is completely unfunded and unspecified, and would be extremely resource intensive on the FPGA.
• The only really likely use for a Sayma ADC that I can see is things like laser intensity stabilization or reading a microwave power meter value. However, these uses don't require a high-bandwidth ADC. For this case, by far the best solution IMO is to put the ADC directly on Allaki. Doing this allows us to remove a pair of SMPs, saving costs and reducing all the mechanical issues we've seen.

[hartytp]

Other comments:

• If we remove the JESD lines associated with the fast ADC, can we go to fewer layers? That would be a big cost saving.
• If we remove the ADC JESD then can we use a cheaper FPGA on Sayma AMC? That would also be a huge cost saving!

It just seems like these ADCs are a massive cost and complexity inflation, while not clearly provinding any functionality that anyone will actually use.

Thoughts?

[gkasprow]

@hartytp We would have to add some high speed connector if we want to move ADC to Allaki. We can also not mount them which would cut the cost drastically. So there would be simple DAC version and full featured ADC+DAC+RF backplane version. Since routing is already done, we won't save much. Number of layers cannot be reduced neither.

[gkasprow]

@hartytp We need 16 GTHs for DACs, so this does not influence the FPGA selection. For ADCs we spent not used Rx parts of GTHs. Replacement of FPGA is complex and error prone task.

[hartytp]

We can also not mount them which would cut the cost drastically.

That's not an option on its own, since we do need an ADC for intensity stabilization. So, if we DNP them, we still need an ADC on allaki. It's just a much slower one.

DNP-ing them is fine. However, I do wonder if it's better to just grit our teeth and remove them:

• Frees up board space, which can be used later if needed. e.g. we can design mezzanines without having to leave space for ADC SMPs
• I don't like the idea of leaving DNP pads for complex circuitry which has never been tested and has no software written for it
• Removing them simplifies the boards, reducing the chance of issues during manufacture

@hartytp We would have to add some high speed connector if we want to move ADC to Allaki. We can also not mount them which would cut the cost drastically.

Is the connector already there not fast enough to run a 50MHz SPI bus?

We do also have to think about how we get these SPI buses from Sayma AMC to Sayma RTM without too much latency.

[dhslichter]

@gkasprow I think the idea would be to ditch the current ADCs entirely, and just use a slow ADC on Allaki with SPI output (~50 MHz max or so) -- the current Allaki connectors should be fine for that.

[hartytp]

Number of layers cannot be reduced neither. hartytp We need 16 GTHs for DACs, so this does not influence the FPGA selection. For ADCs we spent not used Rx parts of GTHs. Replacement of FPGA is complex and error prone task.

ACK.

[hartytp]

@gkasprow I think the idea would be to ditch the current ADCs entirely, and just use a slow ADC on Allaki with SPI output (~50 MHz max or so) -- the current Allaki connectors should be fine for that.

Right.

[hartytp]

So, perhaps the most important point about putting a SPI ADC on Allaki: it's hackable. If I want to hack in a quick feedback loop into my gateware using the ADC, that's basically a non-starter with the JSED ADC. Too complex, not worth the time, I'd rather just attach a Novo/Sampler via the FMC. If you put an SPI ADC on Allaki then it becomes trivial to write code to use it without long expensive gateware contracts.

[dhslichter]

Silicon cost for the current AD9656 ADCs is $700 per Sayma. If we went for 8 channels of ~1 MSPS SPI ADC, we're looking at a cost of $100-200, depending on component choices.

[hartytp]

Silicon cost for the current AD9656 ADCs is $700 per Sayma. If we went for 8 channels of ~1 MSPS SPI ADC, we're looking at a cost of $100-200, depending on component choices.

Right! Scrapping the ADCs and SMPs it looks like we can shave something approaching $2k off the Sayma cost. That's huge!

Plus, the improvements in power dissipation, which should improve thermal stability significantly.

Plus, we can use a slower ADC driver OpAmp on Allaki, which saves more money and current.

[dhslichter]

For example: http://www.analog.com/en/products/analog-to-digital-converters/ad7903.html#product-overview

6 mW per channel!!

Gives you 2 channels of 1 MSPS 16-bit ADC on an Allaki card (thus 8 channels per Sayma) for just over $30 per chip ($140 per Sayma).

Going to SPI ADC on Allaki also means that we can cut the SMPs in half for the Allaki boards anyway, so if the alternate connectors don't work out mechanically we still achieve cost reductions there.

[hartytp]

6 mW per channel!!

The downside, you now need to buy a light and can't just use Sayma to light the lab...

[hartytp]

Going to SPI ADC on Allaki also means that we can cut the SMPs in half for the Allaki boards anyway, so if the alternate connectors don't work out mechanically we still achieve cost reductions there.

And reduce the insertion force.

[dhslichter]

Heater, @hartytp , heater.

[gkasprow]

@hartytp with 4 SMPs @jbqubit had troubles with insertion... I'd love to switch to these Samtec connectors and FMC-like panels.

[dhslichter]

I'd love to switch to these Samtec connectors and FMC-like panels.

If they work mechanically I am all for it, but if they don't...then we have a fallback with smooth bore SMP and lower connector count is my point.

[hartytp]

@hartytp with 4 SMPs @jbqubit had troubles with insertion... I'd love to switch to these Samtec connectors and FMC-like panels.

Agreed! But, even then, the fewer RF lines we have to send to our mezzanines the better, right?

[jbqubit]

@hartytp Thanks for raising this point for discussion. Here's a recap of proposed trajectories and my own comments.

option: Sayma_AFE_baseband slow ADC

Adding a simple, slow ADC to Sayma_AFE_baseband sounds like a great path. Simple, low cost, hackable. Nice. Lets do this.

option: DNP_Fast_ADC

The Sayma concept was conceived well before Zotino+Novo emerged on the scene. Indeed Sayma is a poor choice for a 100 kHz BW servo. The Sayma ADC is also expensive and power hungry. I support the option of not populating the fast ADCs in future builds. Is M-Labs OK with this variant?

Since we must rejig Sayma_RTM - Sayma_AFE RF interconnects anyway (#386), consider using a pair of RF headers.

• one for DACs and LOs
• one for Fast ADCs The DNP_Fast_ADC option could then include DNP of the Sayma_AFE Fast ADC headers.

option: remove ADC from design, recover board space

@gkasprow articulates why this is a poor choice.

@hartytp We need 16 GTHs for DACs, so this does not influence the FPGA selection. For ADCs we spent not used Rx parts of GTHs. Replacement of FPGA is complex and error prone task.

Since routing is already done, we won't save much. Number of layers cannot be reduced neither.

what are use cases for fast Sayma ADC anyway?

With the DNP_Fast_ADC approach we don't have to answer this question right now. In the medium term somebody there is a strong case to use Sayma_RTM with the ADC installed that path is not blocked. Putting off development of JESD204B input also saves time and money that could be directed elsewhere.

[hartytp]

@gkasprow articulates why this is a poor choice.

I'm not sure that's fair. He didn't say it was a poor choice, just he just pointed out a couple things that removing the ADC completely won't solve. That doesn't imply that there aren't other good reasons to remove it!

To recap, I think there are strong reasons to remove it, rather than just DNP it:

1. No one has articulated a good use-case for it. Since we're designing without a use-case, I'll bet you a case of beer that if a use-case ever does emerge, this will turn to be the wrong ADC for the job, so it won't work anyway.
2. If we keep it, we have to reserve the space for the SMPs, which makes the AFE design more complex (these are very space constrained)
3. If we remove it, we can simplify signal routing and power distribution. This gives us more options if we have PI or SI issues later on
4. I think it's very poor practice to keep these kinds of high-speed high-spec components on the PCB without ever testing them, even if they are DNPd. However, testing them would be a huge resource cost, which will take a lot of time and slow things down. I just don't want to get into debugging issues to do with an ADC that no one will ever use
5. This leaves more fine-pitched pads which can introduce shorts etc and reduce yield of the pcbs.

So IMO if no one has a concrete use for these then we should just acknowledge that while the ADC was a nice idea, we should just remove it.

[dhslichter]

The board design work for the ADCs is a sunk cost, and as long as the DACs work currently I don't see a big problem with just DNP ADCs. Perhaps we might just put soldermask over all the ADC pads so they don't present a tempting opportunity for shorting or other shenanigans, and then if people want to fund/use ADCs in future runs it's relatively quick and easy to add them back in. We would use connectors for the AFE cards that only send the DAC signals, which again could be changed relatively easily in the future if people decide they want the ADCs on.

My feeling is that this ADC design will be left behind by future Sinara uTCA boards with parallel ADCs and DACs on an RTM card for fast feedback. The main problem facing Sinara right now is getting boards out and working in labs to demonstrate that this is a viable method for experimental control, and at a reasonable price point. I'd rather focus M-Labs' bandwidth accordingly (and not on JESD RX cores), since there remains a lot to be done to get this all up and running.

[jbqubit]

@hartytp I appreciate your enthusiasm for thinking about future revisions of Sayma_RTM. But I wholeheartedly agree with @dhslichter that it's better to to focus in the near term on getting Sayma and Kasli working.

[hartytp]

Re ADCs you mean? Well, DNP-ing them and deleting them take the same amount of time, so I don't think that's a good argument.

My feeling is that removing the fast ADCs frees up some board space and will simplify routing for other components which is likely to speed things up rather than slow them down. But, if you really want to keep the fast ADCs then I'm okay with that so long as we don't end up compromising on other things.

In any case, the main thing is that we all agree not to waste time testing, supporting or developing a test harness for the fast ADCs. That really would slow things down.

[gkasprow]

@hartytp @dhslichter The ADC path is currently being tested by Creotech from HW point of view. The automated testbench is in final stage of preparation to be ready for production of second release. So please don't reinvent the board completely because they would need to adopt the test-benches which is costly and time consuming. We can change connectors, mezzanines mechanical format because this does not influence the board functions. We can get rid of some LDOs. Automated test bench also includes various PCB adapters that supply signals to the RF connectors, RTM connectors etc. Their role is to test all possible functionalities. It's far easier to DNP certain components then to remove them from board design. To make variant I simply select which components are mounted. When I remove components from schematic, I have to regenerate bus contents, symbols and manually delete all hanging traces so it is quite a lot of work. So if we modify the AFE layout, I'd need to produce and design AFE tester again which also adds cost. And please don't remove Rx lines from RTM connector - the board has already other users so we would need to support both versions of it. One application of it is RTM board with SFP cages where Rx/Rx lines are needed. As @jordens stated, we can support serial ADCs over mezzanine connector easily. Since their latency is already high - need tens of SPI clock cycles, you won't be able to use them in low latency applications anyway.

[dhslichter]

@gkasprow this makes a lot of sense. I do not want to rip out all the ADC stuff, just DNP is suitable for me, as I stated above. In particular, I definitely don't want to start making some dirty/non-extensible hacks to the board like direct lines from analog mezzanine connectors to AMC cards, single-ended logic over RTM connector, etc. I would point out that there are plenty of potential customers who might be interested in JESD ADCs from the current design and don't care about the latency, so leaving the traces in place (which is the cheapest option as well) seems sensible. You can shave $700 off an RTM card just by DNP the ADCs.

If there is room on the RTM connector and there are pins available on the AMC and RTM FPGAs, it would be nice to route some additional differential pairs between RTM and AMC FPGAs - regular LVDS. Serial data from ADCs on an AFE card would go into the RTM FPGA and then be sent on (potentially with 4:1 SERDES if needed, explicitly NO gigabit transceivers) to the AMC FPGA with low additional latency. These traces could also be used for other purposes by other board users.

If Creotech has already got the ADC test harness to the testing phase, we should let that finish. The cost savings for our end will be board cost reduced by 2 ADCs, and then the time/cost savings of not having M-Labs thinking about/working on JESD RX cores.

[gkasprow]

@dhslichter AMC FPGA has some unused pins in HP bank that can extended number of IO lines that go to RTM connector. But there are no free pins in RTM and we wuld have to share Rx part of gigabit ones which causes SI issues. We already have a 2 LVDS links that are not connected on the RTM side and could be routed to mezzanine connectors. But again, there are no free pins. The RTM FPGA does not have any free pins but we could use 4 LVDS pairs that are routed to goldpin array. The easiest path is as you said - implement simple transfer protocol using IOSERDES and existin LVDS pairs. We have at the moment 3 such pairs RTM_FPGA_LVDS1/2 and RTM_FPGA_USR_IO and can connect another two, not used by DAC chips but already routed to the AMC FPGA In this way we get 5 LVDS links that can be used to transfer 5x 1Gbit/s of data easily which would satisfy our needs. Modification of PCB is minor.

[hartytp]

@hartytp @dhslichter The ADC path is currently being tested by Creotech from HW point of view. The automated testbench is in final stage of preparation to be ready for production of second release. So please don't reinvent the board completely because they would need to adopt the test-benches which is costly and time consuming

@gkasprow I understand the reluctance to make changes after putting all that work in. However:

• The proposal isn't to put a new ADC on Sayma, which would require a new testbench, it's to remove/DNP the ADC entirely. We can then put a cheap, easy to test ADC on the AFE Mezzanine later on. This would be trivial to test separately from Sayma. So, I don't believe that this change will lead to more costly work on the test bench.
• I agree that it's a shame to waste all the work that has been done on developing the ADC test bench. But, so far all users have agreed that they want to DNP the ADC. Since this will not be populated on any boards in the near future, the test bench will not be used in any case.

It's far easier to DNP certain components then to remove them from board design. To make variant I simply select which components are mounted. When I remove components from schematic, I have to regenerate bus contents, symbols and manually delete all hanging traces so it is quite a lot of work. So if we modify the AFE layout, I'd need to produce and design AFE tester again which also adds cost. And please don't remove Rx lines from RTM connector - the board has already other users so we would need to support both versions of it. One application of it is RTM board with SFP cages where Rx/Rx lines are needed.

Okay, if that's easier for you then I'm okay to leave the fast ADCs on, but have them DNP-d. So long as there is a decent way to connect the SPI busses to the AMC FPGA then I'm happy.

I would point out that there are plenty of potential customers who might be interested in JESD ADCs from the current design and don't care about the latency, so leaving the traces in place (which is the cheapest option as well) seems sensible. You can shave $700 off an RTM card just by DNP the ADCs.

Who? What for? I don't like this kind of argument, as one can justify essentially any design decision by appealing to as-yet-unknown users who might benefit. And, generally, if one makes design decisions without an exact use in mind, the design doesn't work well.

I really can't see who would use it:

• AFAICT, the superconducting community is not interested in Sayma at all since the JESD latency makes it a non starter
• The ADCs aren't very useful for downconversion, since for that once really wants the ADC to be as fast as the DAC. Otherwise, one has to do an intermediate downconversion using a DDS/PLL, which is quite inconvenient
• Ion trappers don't want the cost, power dissipation of the fast JSED ADC and don't need anything like this bandwidth.

So, what is the potential use case? I'm really struggling to see it. AFAICT, it was a nice idea that didn't work out.

[hartytp]

We already have a 2 LVDS links that are not connected on the RTM side and could be routed to mezzanine connectors. But again, there are no free pins.

Presumably switching to a connector with a few more pins is quite easy? Otherwise, we can re-look at that connector and see if we can reallocate a few pins.

The easiest path is as you said - implement simple transfer protocol using IOSERDES and existin LVDS pairs. We have at the moment 3 such pairs RTM_FPGA_LVDS1/2 and RTM_FPGA_USR_IO and can connect another two, not used by DAC chips but already routed to the AMC FPGA In this way we get 5 LVDS links that can be used to transfer 5x 1Gbit/s of data easily which would satisfy our needs. Modification of PCB is minor.

@jordens what do you think about all of this?

My feeling is that while that's nice and easy from a hardware side, it really complicates things from the gateware side. As I said, I'd really like these ADCs to be something that users can hack at easily. Having to set them up using a shared bus in this way introduces a few nasty issues in the gateware that will be a real pain for users.

[gkasprow]

@hartytp anyway, if you want to route 8 SPI ADCs, you'd need at least 32 lines to pass via RTM connector. Even if you use single-ended signalling, we still won't have enough pins.

[gkasprow]

One can share CS, clock and DIN lines, flexibility would be reduced and buffers would have to be added.

[hartytp]

If I seem a bit frustrated with the ADC situation, here's why:

• Basically all planned users of Sayma need 1 ADC per DAC channel for some form of feedback, either laser intensity stabilisation or microwave stabilisation
• We have the best part of $1000 worth of high-spec ADCs on Sayma
• Despite this, our plan is to do our feedback by attaching a Sampler (used to be Novo) to Sayma AMC via a FMC->VHDCI adapter and VHDCI EEM carrier. Other users I've spoken to are either planning to do the same or to use an external ADC (e.g. the NIST lock box) and an external modulator for the feedback.

I can't think of a clearer indication than that that there is something fundamentally wrong with our current approach to the ADC on Sayma.

[hartytp]

And, note that the above isn't meant to imply that I'm not mainly happy with Sayma, which I think is generally excellent. The core functionality of Sayma is the high-speed DACs, which seem to work perfectly. The ADCs are a relatively minor (but still important) add on, which we just need to get right.

[hartytp]

@hartytp anyway, if you want to route 8 SPI ADCs, you'd need at least 32 lines to pass via RTM connector. Even if you use single-ended signalling, we still won't have enough pins. One can share CS, clock and DIN lines, flexibility would be reduced and buffers would have to be added.

Agreed, there isn't enough IO to do this the way one would ideally like with a separate SPI bus per ADC, so we're going to have to do some bus sharing. @gkasprow Can you clarify how many unused pins we have on: the RTM FPGA; the AMC FPGA; the AMC<->RTM connector; and, the mezzanine DIO connector, please?

If the mezzanine IO connector is the limitation, is it a problem to use a connector with a few more pins? If the AMC <-> RTM connector is the limitation and we really don't want to remove the ADC, would it be possible to add solder jumpers to allow one to switch between the JESD and LVDS busses? Or is that too much of a SI nightmare for gigabit lines?

[gkasprow]

@hartytp There are several double ADC with SPI. For txample this. So let's do following: ADC configuration goes via RTM FPGA as it was before Data are transferred directly to the AMC FPGA via 4 or 5 LVDS links - 4 data lines and one sync. The RTM FPGA is used only as LVCMOS to LVDS converter. If we want to go for individual ADCs, then RTM FPGA would serve as DDR buffer only multiplexing 2 MISO channels into one. In this way we leave Sayma as it is, only 2 LVDS links are added, RTM firmware is trivial.

[hartytp]

@gkasprow Something along those lines sounds ideal to me (although, I'd need to put a bit more though in before making a final decision to check that I'm not missing anything). I'd still like to hear from @jordens on this.

[dhslichter]

@hartytp other potential users of JESD ADCs include:

• ion trappers doing feedforward on fast pulse shaping (e.g. the Magtrap experiment)
• superconducting people not interested in feedback with their ADCs, just digitizing their readout data (many groups)
• people looking for a general high-speed DAC/ADC board with wide bandwidth (e.g. wireless test)
• others we don't even know about

We have a design that nominally works from a hardware standpoint for the JESD ADCs. The test bench is basically ready, so that once hardware is made with JESD ADCs it could be tested to make sure that the board design/layout is correct. If we just go with DNP for the ADCs for right now, without ripping out anything, then it's straightforward to make use of the option for people as listed above.

What benefit is gained by ripping out the ADCs instead of just DNP? It's additional work, and @gkasprow is happy with just DNP from his end. There are no gains to be made in terms of reducing board complexity or fabrication cost, based on what @gkasprow has said.

[dhslichter]

ADC configuration goes via RTM FPGA as it was before Data are transferred directly to the AMC FPGA via 4 or 5 LVDS links - 4 data lines and one sync. The RTM FPGA is used only as LVCMOS to LVDS converter. If we want to go for individual ADCs, then RTM FPGA would serve as DDR buffer only multiplexing 2 MISO channels into one. In this way we leave Sayma as it is, only 2 LVDS links are added, RTM firmware is trivial.

I think this is basically what I was proposing earlier, fleshed out a bit, and it seems like a reasonable solution to me. I really, really want to avoid descending into complete hackery by shipping raw communications between AMC FPGA and SPI ADCs on an AFE card.

[hartytp]

I really, really want to avoid descending into complete hackery by shipping raw communications between AMC FPGA and SPI ADCs on an AFE card.

I think we're all agreed on this, so long as we have the FPGA pins available (paying attention to where the pins are located on the FPGA, and which pins are cc). If we don't then we may have to make a compromise...