sinara-hw / Fastino

Fast 32-channel, 3MS/s per channel, 16bit DAC EEM card compatible with Zotino
11 stars 2 forks source link

Intermittent failure to start up #90

Open McSherry opened 11 months ago

McSherry commented 11 months ago

Summary

We've observed that Fastino occasionally fails to initialise properly on power-up.

Our suspicion is that the bitstream is not being read correctly because the power-up sequencing for operating the FPGA in SPI master mode is not adhered to. This requirement is that $V{CC}$ and $V{CC\_SPI}$ are energised at least 0.25 ms before $V_{PP\_2V5}$.[^1]

[^1]: Lattice DS02029-4.0, table 4.3

Measurements

We first suspected a failure of the PLLs to lock, so we measured the clock output from two Kasli-SoC masters. This was perfectly fine (N-side as at the Fastino end of the ribbon cable, while under failure, is the blue trace in image below):[^2]

image

[^2]: The top four logic traces aren't great representations of the true waveform. The logic probe was attached to vacant EEM connectors on Kasli-SoC mainly to verify that there was clock output on all expected connectors under failure. The probe didn't use a local ground and likely there was some mistriggering from the LVDS signal levels.

What we also tested was adding three registers to the FPGA design. These were all set up to toggle every cycle and their outputs are shown as traces D0, D1, and D2. They were:

Under a PLL misconfiguration/misuse (or a PLL-specific failure-to-initialise that Greg has said is a known issue with early-revision iCE40HXs), I'd expect D0 to be toggling while D1 and D2 remain steady. Because D0 also remains steady, this looks a lot more like a complete failure to initialise the FPGA. Since we know the SPI master power supply sequencing isn't being adhered to, it seems likely that this is a failure to read the bitstream.

gkasprow commented 11 months ago

@marmeladapk isn't it related to the issues we observed on some Fastinos? W thought that it was PLL issue..

marmeladapk commented 11 months ago

What we observed was connected to lack of proper initialization of some structures inside the FPGA. AFAIK since fixes in ARTIQ we haven't observed these issues.

jordens commented 11 months ago

IIRC I've seen some Fastinos with very early "engineering sample" silicon from mid 2012 that may have behaved differently.

gkasprow commented 11 months ago

That's why I asked about datecode

McSherry commented 11 months ago

The datecodes are A1461R81 and A5271R15. Apologies, forgot to copy them over from our chat.

gkasprow commented 3 months ago

image

gkasprow commented 3 months ago

Now 3V3 starts first, VCC/VCCPLL and VPP_2V5 wake up later.

gkasprow commented 3 months ago

The plan is to add:

gkasprow commented 3 months ago

Here is the fixed power supply; update: 100nF->1uF; 10k -> 33k image