[RFC] new EEM: humpback

[jordens]

Despite the general purpose of the Sinara ecosystem hardware and the wide range of features, lot of development in labs is still based on random "toy"/"development"/"evaluation"/"hobbyist" boards (called "boards" throughout) with µCs and FPGAs. They are dirt cheap, fit for the many applications that lie outside the typical use cases for Sinara EEMs, get the job done just fine, and have a shallow learning curve suitable for physicists. That won't change and there seems no point in trying to replace those "boards". Instead let's try to integrate them.

Idea

I'd like to propose a simple modification to the TTL tester EEM to give it a much wider applicability and open it up for more use cases.

The basic idea is to

1. get connectivity between boards and the Sinara EEM connector
2. build a easy way of mounting those boards in a crate
3. to supply them with power.

Use cases

• Ethernet connectivity of all kinds. Translate EEM to ethernet.
• Can be get plenty of power from PoE independent of the board (e.g. 21W)
• Wire up microcontrollers that run simple control tasks and loops (temperature control, laser power stabilization, slower modulation/demodulation frequency locks) when (a) where the µC is connected to EEMs and controlling them and (b) when the µC is connected as an EEM to a core device.
• Enable usage and testing of other EEMs with a simple adapter using a µC or FPGA. Also for acceptance testing at the factory.

Implementations

There seem to be two approaches:

A EEM adapter: Shield

Build a "shield" that just implements the connectivity aspect (1) where a remote board is connected to the core device.

• To be used outside a crate
• 2xRJ45 connected to a DIO-RJ45 EEM in a crate and thus to the core device
• Should use the same LVDS-to-CMOS converters and DIP switches as Tester. Or maybe just a cheap (ICE40) FPGA (and DIP switches)
• Should be powered from the board
• Does not give I2C connectivity.

A EEM board carrier: Humpback

Build a EEM that achieves (1) both ways, (2), and (3).

• Call it e.g. "Humpback"
• Mount the board on the EEM
• Since there will likely be a need for external connectivity, while the panel pattern for that connectivity is unknown, let's open up the panel and also build a gap into the board so that only a panel frame (or even just a panel-U) remains.
• Make the EEM short so that small boards can be used and large boards just stick out of the crate.
• Two EEM connectors
• Power the board from the EEM
• Add a barrel connector so that the EEM can power MP3V3, the board and attached EEMs or the EEM and board can be powered from the upstream core device (I.e. dual use just like Tester).
• Maybe a 5V SMPS for VIN to feed the more sensitive boards.
• Use a cheap FPGA (ICE40 maybe) to convert LVDS. Add DIP switches and implement an I2C direction control shift register in FPGA.
• Expose 8xCMOS or LVDS raw (for FPGA boards) and I2C to board.
• Maybe expose MP3V3 to board.

Potential layouts and pinouts

The choice for the connector layout (power, i2c, digital) is a bit tricky. For both shield and EEM there could be:

Arduino Uno V3

The "Arduino Uno v3" pattern for the shield and the EEM plus a common denominator of mounting holes and standoffs for the EEM would cover large subset of boards. These are typically stacked and the EEM would probably the "top" of the stack while the board is the "bottom". Best position and orientation unclear.

µC/CPU:

• Arduino (but not with 5V IO!)
• ST Nucleo 144, ST Nucleo 64
• ~Intel Galileo~

FPGA:

• Xilinx Artix: Arty
• Intel Max 10 (fka Altera) XLR8
• Xilinx Spartan 6: Papilio (Pro)
• Lattice ICE40:

26 pin Raspberry Pi+

Has 5V, I2C, 3V CMOS, mounting hole pattern.

• Raspberry Pi B+ etc
• Orange Pi (Zero)

Digilent PMOD

Probably only as a shield.

ST Morpho

• Nucleo

ST ZIO

• Nucleo 144

beagle bone

• Beagle bone black

[marmeladapk]

Just a quick note: Intel Galileo (Edison and Joule also for that matter) has been EOLed.

[gkasprow]

There is new Arduino with FPGA, so this could be interesting idea. If we want to keep the cost down, maybe ICE40 FPGA family could be an option? It is very low cost (~1$) and has completely open source toolchain. Tiny FPGA between uPC and EEM could be benefit and help build relations between Arduino and EEM. ICE40 would replace LVDS drivers. FPGA is loaded from FLASH or processor.

[sbourdeauducq]

f we want to keep the cost down, maybe ICE40 FPGA family could be an option? It is very low cost (~1$) and has completely open source toolchain. Tiny FPGA between uPC and EEM could be benefit and help build relations between Arduino and EEM. ICE40 would replace LVDS drivers.

Sounds good.

[sbourdeauducq]

If we have a FPGA flash, the boards can come shipped with a bitstream that turns the FPGA into a dumb LVDS/TTL converter.

[gkasprow]

Exactly, and we would cut the cost. Arduino has nice feature - standarisaton of the connector. So we can plug variety of clones even with Linux machines on it. We can make double Arduino/RPI footprint.

[gkasprow]

There are ICE40 shields for RPI with tools that configure the FPGA from Linux, this could be powerful tool.

[dtcallcock]

Would this be a way of creating an Ethernet carrier, as discussed in #473?

[gkasprow]

This is my understanding of this task. Recently I use tiny Orange Pi board. It has essentially all what is needed, quad core CPU and several IOs. The cost is 8$. But RPI has better support and probably would be optimal choice for such purpose. So ability to plug either Arduino Leonardo (with Ethernet) or RPI would satisfy most use cases.

[gkasprow]

ICE40 FPGAs has nice features : hardened SPI and I2C cores :)

[sbourdeauducq]

Those aren't nice features, those are a waste of time; a harbor for frustrating bugs and having to learn SPI/I2C again with the proprietary interface du jour. What is nice is high-quality, portable, open-source HDL for SPI and I2C.

[gkasprow]

@sbourdeauducq I didn't use them, can be useful in case of lack of resources when somebody desperately needs to get some logic cells:)

[gkasprow]

@jordens do you think that it makes sense to provide support for 2 EEMs? I.e. to run Sampler or Urukul with Rapsberry pi :) If so, then we have to use ICE40 FPGA in BGA package (8$) instead of QFN one for 2..3$. But then 3 EEMs can be installed.

[jordens]

@gkasprow For Humpback (but not Shield) that makes a lot of sense to me. The added cost of one more IDC header seems ok. And most µC/FPGA mezzanine boards seem to be able to handle 16 IO. Then for the FPGA on Humpback, 21 I2C pairs and 28 LVDS pairs, plus 2*8+2 CMOS IO to the mezzanine? That would be 54 pins. Are we talking something like the ICE40HX1K-TQ144?

[gkasprow]

@jordens The ICE40 family has LVDS lines only in one bank. With TQFP140 package you can connect only one EEM, BGA256 enables 3 EEMs. I want to use Beaglebone as default with corresponding panel cutouts.

[gkasprow]

Some sketch

[gkasprow]

why beaglebone?

• has real Ethernet with low latency
• has fast IOs with PRU
• has external bus interface (GPMC) which gives fast IO channel to the FPGA

Of course place for Arduino Leonardo Ethernet will also be available

[jordens]

@gkasprow Ack the ICE40 BGA256 and the three EEMs.

The Beagle bone is fine. But I'd like to support a simpler/cheaper µC board as well. The BB needs male headers on Humpback. It looks like we can add female headers for the ST Nucleo 64 boards. And I think they will not collide with the beaglebone. They are wider apart and the underside of the nucleo is empty.

Why Nucelo 64/STM32:

• STM32 are one of the dominating embedded chips with Rust support with the most active community.
• Are used in labs, even have flight herritage.
• Are really cheap and have a wide range of chips/boards readily and easily available.
• Don't collide mechanically with the Beaglebone.
• Have boards with Ethernet

[gkasprow]

OK, so maybe let's put Ethernet and USB couplers on the panel and provide several footprints on the EEM. In this way the panel is fixed and we can install Arduino, BB, STM32, etc.

[jordens]

A couple open questions:

• Add DIP switches to the FPGA to configure the LVDS direction or is that only done by bitstream/I2C?
• Leave the panel cutout wide open so that different boards can be accessed through it? I think there are two problems with matching ethernet+usb cutouts in the panel. They are not mechanically compatible between boards and they are a mechanical constraint that conflicts with the pin headers (see Allaki). I'd rather solve that like FMC (or the next iteration of Allaki) with a wide cutout and a panel on the mezzanine/board.
• Maybe ship the pin headers for the different boards unsoldered so that the user can choose and solder the ones he wants.

[gkasprow]

DIP switches will be routed to the FPGA so can be used for any purpose. Defualt configuration will provide LVTTL-> LVDS buffers. I'd use low cost Ethernet and USB couplers on the panel. We can also place the USB and RJ45 connectors on the PCB and make such couplers ourselves.

[jordens]

Ah. Then there would be a short patch cable to the board. That's OK by me. The other approach with connectors on Humpback is OK as well. I don't know which is more compact/cheaper. Since the front panel is now used and closed, I'd expect users to add another front panel next to humpback for their own non-Ethernet/USB connectivity, i.e. ADCs, DACs, GPIO, etc.

[hartytp]

I'd use low cost Ethernet and USB couplers on the panel. We can also place the USB and RJ45 connectors on the PCB and make such couplers ourselves.

I'm never a fan of having USB/ethernet feedthroughs on the FP + internal cables. They tend to be bulky and annoying and the cables get caught on things. Plus, it's more parts that need to be sourced.

If we're thinking of going down the route of having the customer solder on pin header and supply the microprocessor board, then why not have front panels sold separately to the PCB? That way the user buys the right panel for their microprocessor board and installs it when fitting the microprocessor module.

[jordens]

Yes. There is the pigtail mess with inflexible but short RJ45 cables and the added parts. Thinking about it, I'd also prefer either a big empty cutout in the panel large enough to accommodate the target boards or different panels for different boards.

[jordens]

And I suspect this will need to be 6 HP or 8 HP, certainly for the Nucleo, but looking at the rendering also the BB.

[jordens]

After the Banker design a few additional notes and opinions.

• Let's do 16 dip switches. To the fpga.
• Let's keep it simple and have the IOs all 3.3 V only. No need for translation or drivers.
• 3 EEM like Banker.
• I'd like a big cutout or maybe even a deep wide slot in the front panel. So that the remaining front panel is just a U shape and the boards can get a matching rectangular (like FMC/new allaki) or even T shape panel insert. And 4HP. The front panel on the boards can then be T shape and extend the width to 6/8/12 HP.
• A common set of mounting holes for the boards.
• Maybe consider mounting a solderless breadboard on Humpback so that there is prototyping flexibility. There are small ones around with mounting holes.
• I can't come up with a common set of header placement for the boards. This is difficult. Like the mounting holes. My favorite list of boards ist currently: some SIL/DIL40/32 style nucleo/generic, BBB, orangepi, wiznet lan module, esp12

[gkasprow]

What about using 3D printed panel inserts? I will make and releases stl files so everybody will be able to print them at home. 3D printers are so popular these days. For 200$ one can get one with sufficient print quality. We have such one in the lab. Recently I bought Prusa MK3 for home use and it is decent print quality printer for less than 1k EUR. Why 3D printed panels? Various board have different distance between connectors and board edge. Some have 5V supply connectors that it's safer to cover entirely. Some have mini USB that require very thin panel. 3D printed one would solve all the issues easily. Due to small area print time / cost would be low.

[gkasprow]

@jordens do you want one of EEM ports to be upstream? I.e. to use one of SBC as a target for Kasli ?

[jordens]

3d printed panel inserts sounds good. Then you can easily do recessed panels or none at all for quick testing. Yes. I'd stick pretty close to the banker design for the eem connectors. Using Humpback as a Target for kasli is definitely a nice use case.

[gkasprow]

I want to produce Humpback, Stabilizer and Banker at the same time using shared documentation to limit NRE costs. @jordens @hartytp please check schematics and initial component placement for Humpback. The board is 99% routed but no DRC and cosmetics were done. PDF is here, sources are in repo Humpback.PDF

I provided support for: Nucleo 144 (compatible with Arduino), Beaglebone Black, Orange Pi Zero, Wiznet (2x UART only) and ESP32 which uses same code and tools as ESP8266 but has 2 CPU cores and much more memory. I use it for my projects since some time. And it has ufl antenna connector for WiFi and Bluetooth so one can attach SMA pigtail to the 3D printed panel and gain wifi access. It is supported by Arduino so remote logging or control using MQTT with i.e. Cayenne/Blynk and mobile App is trivial - just a few lines of code. I have such projects in my private repo. All common ports like I2C, UARTs, SPI were routed together to save pins. All SBCs have access to FPGA configuration pins so they can load the FPGA directly without supporting FLASH. There are open source codes that do it for Beaglebone and STM32. Beaglebone pin naming comes from LOGI-BONE project where bus access to FPGA is used. I also use same FPGA config pins so changes to the code would be minimal. Open source HDL and Linux code are available. Rest of Beaglebone pins are named after the GPIOs All dedicated pins were connected to individual FPGA pins. In this way we will be able to provide and support same HDL code for all SBC platforms. FPGA supports up to 4 images in the FLASH, selectable using DIP switch. In this way user can choose which functionality to load depending on installed SBC.

[hartytp]

@gkasprow I'm not really the best person to review this kind of design. I had a quick look over it and it looks really nice, but I can't give you detailed feedback (not enough experience using these kind of microprocessor boards).

One question: did we plan to make this a PoE module? Or, is it intended to always be powered from a 12V supply?

[hartytp]

I want to produce Humpback, Stabilizer and Banker at the same time using shared documentation to limit NRE costs.

Can we sneak Thermostat in there as well?

[gkasprow]

Thermostat is 2 layer board, I'd rather produce in one run: 2 thermostats, 4 banker extension modules and EEM protection board

[gkasprow]

PoE would be hard - none of the boards support PoE. There are some plans for PoE for OrangePie but it is not real POE, just a way to send 12V over Ethernet lines.

[jordens]

A quick first set of comments:

• Great work. Congrats to fitting all devboards onto Humpback! I didn't expect that.
• I assume this would come with all pin headers not soldered but still included.
• It's a bit risky to do all three boards at the same time especially with Banker and Humpback being so similar. Fine if the NRE savings outweigh that risk.
• Testpoints should be on the top if at all possible (TP, B)
• Nice solution on the I2C master/slave issue! I also extended the kasli-i2c tools to automatically scan any tree configuration of those I2C switches recursively. I.e. EEPROM detection is fine at any depth.
• The Nucleo 144 boards seem to have one of four different pinouts. But I couldn't identify the actual difference quickly. Which pinout is this?
• Is that distance between the capacitor/resistor combo and the front panel screws/washers large enough?
• Doesn't the ESP32 need a specific ground/keep-out area/pattern near the PCB antenna?
• From the PCB renders with the 12V copper areas, the EEM headers are still the wrong assignment. The upstream one should be the rightmost.
• Is 10µF as the biggest cap going to be sufficient for the FPGA VCC/VCCIO? Comparing that with Kasli it sounds tiny. But I guess you took a devboard as the reference.
• I didn't check any pinouts, or pins vs datasheet consistency. Would you like someone to do that?
• No PoE is fine by me. There are cheap external PoE+ splitters for barrel.

[gkasprow]

Great work. Congrats to fitting all devboards onto Humpback! I didn't expect that.
I assume this would come with all pin headers not soldered but still included.

yes

It's a bit risky to do all three boards at the same time especially with Banker and Humpback being so similar. Fine if the NRE savings outweigh that risk.

these are relatively simple boards, easy to rework

Testpoints should be on the top if at all possible (TP*, B*)

Aren't they? I'll check

The Nucleo 144 boards seem to have one of four different pinouts. But I couldn't identify the actual difference quickly. Which pinout is this?

I took the 144 series because they have embedded Ethernet

Is that distance between the capacitor/resistor combo and the front panel screws/washers large enough?

I'll check

Doesn't the ESP32 need a specific ground/keep-out area/pattern near the PCB antenna?

I used one with ufl. An antenna inside of the rack do not make a lot of sense.

From the PCB renders with the 12V copper areas, the EEM headers are still the wrong assignment. The upstream one should be the rightmost.

that's right, the copper is not finished, it's copied from previous version of Banker

Is 10µF as the biggest cap going to be sufficient for the FPGA VCC/VCCIO? Comparing that with Kasli it sounds tiny. But I guess you took a devboard as the reference.

This FPGA consumes really tiny amount of power, I'll check with datasneet. I copied it from another design.

I didn't check any pinouts, or pins vs datasheet consistency. Would you like someone to do that?

Yes, mainly power supplies and dedicated pins. Rest of the are general purpose IOs which can go to any location

No PoE is fine by me. There are cheap external PoE+ splitters for barrel.

right

[gkasprow]

I took 144 series devkit with CPU we use for Stabilizer as a reference.

[gkasprow]

The devkit I've chosen supports many CPUs with assembly options: NUCLEO-F207ZG, NUCLEO-F303ZE, NUCLEO-F412ZG, NUCLEO-F413ZH, NUCLEO-F429ZI, NUCLEO-F439ZI, NUCLEO-F446ZE, NUCLEO-F722ZE, NUCLEO-F746ZG, NUCL EO-F756ZG, NUCLEO-F767ZI and NUCLEO-H743ZI

So this gives a lot of options.

[jordens]

I'll try to go through the power supplies and dedicated pins over the next days. Ack the other points.

[jordens]

• I'd suggest connecting GBIN0 to SPI_SCK (clock FPGA from CPU SPI) and GBIN1 to STM32_PB10 or STM32_PB11 (clock FPGA from CPU timer), GBIN4 and 5 are good.
• On the beaglebone I think you have CFG_MOSI and CFG_MISO swapped. On STM32 MISO/MOSI can be swapped. I don't know about BB.
• If the I2C tree gets large, multiple 2k2 resistors are in parallel (up to eight counting Humpback/Banker alone if the switch has all ports enabled, more if there are EEMs connected to it). Is this really robust? How many I2C bus segments can you enable until you get a latchup? I think that note on the schematic only applies to banker where the FPGA bank is at 2.5V.
• GBIN7 must be the first LVDS pair of the upstream EEM (both here and on banker)
• GBIN6 can be the first downstream EEM
• Probably worth running the Lattice tools to estimate dynamic power consumption.
• The 1.2V LDO talks about a 2 mA minimum current draw. The static draw from the FPGA is 1.1 mA.
• There is a GND pin on the ESP32 not connected.
• Is T8A/T8B just two AND gates with open collector outputs? What's the story behind it?
• What's with SW0 and SW1? And separate from that, why are FLASH_SDI/SCK on the dip switch? How would you use that?
• Does it make sense to put populated 0R resistors into at least a few of the ADC and DAC pins of the STM32 to optionally disconnect them from the FPGA (FPGA pulling the ADC inputs, or coupling noise)? Similar for BB.
• There seems to be an opportunity to remove a lot of wire crossings by pin swapping, e.g. around the DIP switches. I guess that's what you meant by cosmetics.

[gkasprow]

I'd suggest connecting GBIN0 to SPI_SCK (clock FPGA from CPU SPI) and GBIN1 to STM32_PB10 or STM32_PB11 (clock FPGA from CPU timer), GBIN4 and 5 are good.

OK

On the beaglebone I think you have CFG_MOSI and CFG_MISO swapped. On STM32 MISO/MOSI can be swapped. I don't know about BB.

will check it

If the I2C tree gets large, multiple 2k2 resistors are in parallel (up to eight counting Humpback/Banker alone if the switch has all ports enabled, more if there are EEMs connected to it). Is this really robust? How many I2C bus segments can you enable until you get a latchup? I think that note on the schematic only applies to banker where the FPGA bank is at 2.5V.

True, actually I don't think that somebody will enable all ports, but when one does, may lock the bus.

GBIN7 must be the first LVDS pair of the upstream EEM (both here and on banker)

OK

GBIN6 can be the first downstream EEM
Probably worth running the Lattice tools to estimate dynamic power consumption.

The design must contain some useful logic to make sense doing it.

The 1.2V LDO talks about a 2 mA minimum current draw. The static draw from the FPGA is 1.1 mA.

Good catch, anyway in real life it works and voltage is 1.2V. I can add some load to be sure.

There is a GND pin on the ESP32 not connected.

ok

Is T8A/T8B just two AND gates with open collector outputs? What's the story behind it?

simplicity. It is used on all ESP32 devkits so I don't want to invent something else.

What's with SW0 and SW1? And separate from that, why are FLASH_SDI/SCK on the dip switch? How would you use that?

After FPGA gets configured, these are regular IOs and can be used as inputs. In this way you spare 2 pins.

Does it make sense to put populated 0R resistors into at least a few of the ADC and DAC pins of the STM32 to optionally disconnect them from the FPGA (FPGA pulling the ADC inputs, or coupling noise)? Similar for BB.

Do we plan to use thes ADC and DAC pins for something analog?

There seems to be an opportunity to remove a lot of wire crossings by pin swapping, e.g. around the DIP switches. I guess that's what you meant by cosmetics.

Yes, but vias cost nothing. I meant curly and dangling traces :)

[jordens]

All ACK. I'd like to be able to use the STM32 ADC/DAC without worrying about the FPGA pins connected to them. Let's put 0R (or a DIP switch array? but I am fine with 0R here) on PA3 PC0 PC3 PC1 PC4 PC5 and PB1 PC2 PA2 for ADC and PA4 PA5 for DAC.

[jordens]

And it would be great if the various switch and pin arrays could get lots of silkscreen labels, even for individual pins on the PCB top side.

[jordens]

Looking at the ice40hx break out board and the ice40 hardware checklist. Probably also applies to Banker.

• The FPGA gets a 100n for each supply pin
• 10u for VCCPLL1 and 0
• You decided that LDOs to get from 3.3V to 1.2 and 2.5V is efficient enough and not switchers are needed?
• They have 10n and 1u on a couple of rails but I don't know whether that makes much sense. They don't have your 100p though.
• N14 on the FPGA is TRST_B on that devboard and pulled low. Also mentioned explicitly in the HW checklist.

[gkasprow]

I copied it from working design. And that working design was copied from somewhere else. So errors could propagate. I can add switcher for 1.2V, but for such small FPGAs I expect core current less than 0.5A. We can use same low cost AUR9705 chip as for 3.3V

[jordens]

ACK, I am not particularly worried about LDO vs switcher and my guess is significantly less than 0.5A but that would be 1W burned in the LDO.

[jordens]

• The beaglebone (but not the orange pi) top side is facing Humpback, and the RJ45 connector is between the two boards. Does that work with the spacing and the header separation or do we need cutouts in Humpback or are there special long pin headers for this?
• There should be a way to either beef up or bypass the fuses to the downstream EEM connectors. So that e.g. two Urukuls and a Sampler can be used without problems. Maybe just bypass jumpers?

[gkasprow]

I didn't finish the mechanical side yet. True, we will need Ethernet cutout to use stadard goldpin connectors.

[gkasprow]

Lattice is using 300mA LDO (LT3030EFE#TRPBF) on their ICESTICK devkit which uses HX4K FPGA They supply it from 5V directly.

[jordens]

Ack. Your choice. If consumption is linear in size that would be 600 mA.

[gkasprow]

OK, I used switcher. It costs 0.5EUR.