Sayam: startup issues

[hartytp]

Building with current ARTIQ master (AMC=4.0.dev+1108.gbb87976d, RTM=4.0.dev+1108.gbb87976d) using vivado 2018.1. Full build environment: https://hastebin.com/etuyayozug.pas

Edit: OpenOCD tells me that my RTM is at 42.6C and my AMC is at 33.5C (https://hastebin.com/zumuqenibi.pas)

Clean 100MHz reference (+6dB) provided from a synth.

I've seen one of two issues on startup:

• SERDES PLL lock timeout: https://hastebin.com/wuyosexibu.php
• UART output freezes after HMC7043 configuration: https://hastebin.com/ohowipavey.php

Currently, I haven't got beyond that. I'll have a look at the HMC830 output to see if I can find a reason for the PLL not locking...

[hartytp]

My 100MHz reference, measured with a 20dB coupler

150MHz FPGA clock, measured on J61

1.26GHz HMC830 output measured on J58

All in all, I'd say there are no large noise issues on my clocks once the HMC830/7043 are correctly configured...

[enjoy-digital]

@hartytp: good. Are you measuring that when PLL fails to lock or on a after a crash during the HMC7043 configuration?

[hartytp]

both. I didn't see any difference between the two cases.

[enjoy-digital]

While measuring, are you also able to see the broadband noise @gkasprow noticed before the hmc7043 initialization?

[hartytp]

just about to post that...sec

[hartytp]

J61 during boot:

after boot:

Which explains some of the issues we had...

[hartytp]

Rebuilt without sawg, and seeing the same two issues on startup.

[hartytp]

Actually, without sawg, I've also seen a few instances of it freezing on AMC to RTM link test (no errors reported).

[enjoy-digital]

Indeed... thanks for the measures. So now to understand the freeze during/after hmc7043 configuration, can you add some prints/delays to the hmc7043 configuration (maybe in the write access) to be able to see if it always fails at the same access?

[hartytp]

note that the error is always after the HMC7043 configuration: it's after "....done" but before the next line printed to the UART... Remind me where you want me to add the debug prints? (remote debugging here would be very useful)

[hartytp]

Comments:

• The rework on my board that connects HMC7043 RESET to GND had come loose. This pin was floating. Tying it to GND again didn't make any obvious difference.
• Tying this pin to 3V3 does indeed stop the noise (and all other output from the 7043!)
• I've written some code that modifies the init sequence to make use of the HMC7043 reset https://github.com/hartytp/artiq/commit/c31e9de7b6469d0d93aed6758e21827fe9e4b585 however, I haven't tested it yet because the HMC7043 reset line has been cut on my board (https://github.com/sinara-hw/sinara/issues/216)

[hartytp]

I'm beginning to wonder if some of the intermittent/board-board variations we have are due to some piece of rework failing.

In any case, my board just isn't working at all now: either serwb fails to init or the HMC830 identifies as 0x00.

So, there doesn't seem to be much I can do on any front right now. I could send my board back to @gkasprow for investigation, but it might be better to just focus on the boards that work and push towards v2.0.

[jordens]

The robust solution would seem to be to put a reasonably strong pullup on that hmc7043 reset (and only that one) that won't get lowered accidentally during any other state and only drive that low when the firmware is certain the input is fine.

[hartytp]

@jordens yes, I was about to suggest that as well. I think there should also be one on the HMC830 CS line to prevent it from getting stuck in the wrong SPI mode.

[hartytp]

So, I did the following

• Asked @klickverbot to undo @gkasprow's rework and reconnect the HMC7043 reset pin
• Apply my new init sequence

After more than 10 restarts, I haven't seen a single crash (but many serwb init failures).

[hartytp]

I also haven't seen any SERDES PLL lock failures with the new init sequence.

[jbqubit]

I've not seen serwb init failures in the last week at UMD. Spanning > 100 reboots.

[hartytp]

After fixing the HMC7043 init, I only see serwb init issues.

[hartytp]

After the latest round of fixes and using @enjoy-digital's fixed 1gsps line rate serwb, I've had 22 successful inits in a row and counting (but, not counting JESD init failures).

So, Sayma not boots properly! Whoo!