hartytp
commented
6 years ago Closed hartytp closed 6 years ago
hartytp
commented
6 years ago hartytp
commented
6 years ago @marmeladapk did you test the AMC + RTM you sent me? If so, please could you send me the binaries you used?
hartytp
commented
6 years ago Anyway, this is currently stopping me from looking at Sayma synchronization...
gkasprow
commented
6 years ago The cards we shipped to you and Sebastien were the two the best behaving. No issues with DAC config. @marmeladapk is on a way to USA right now.
hartytp
commented
6 years ago Thanks Greg.
@marmeladapk when you have email access again, please can you tell me which tests you did on this board and which binaries you used. I'll reproduce your tests and see if I can get the board to work.
@gkasprow do you still have my AMC + RTM? My RTM is identifiable because I removed the screening cans. My AMC is identifiable because I put a solder blob acress the jumper to fix the JTAG issues. If you can find those two boards can you ship them back to me so I can test synchronisation.
All we need to do for Sayma v2.0 is the synchronisation tests and to check the SI for the new AFE test board you made.
gkasprow
commented
6 years ago @marmeladapk took all the RTMs to USA. I have only single AMC in the lab.
hartytp
commented
6 years ago :( @jbqubit @marmeladapk if you can find my RTM + AMC in the batch that @marmeladapk took with him then please can you post it to me asap. Until I have a working board I cannot test the new synchronisation scheme.
hartytp
commented
6 years ago @gkasprow the P2V0 LD4 and P1V2 LD7 LEDs are off on this RTM
gkasprow
commented
6 years ago the transistor nearby was probably broken mechanically during insertion to the crate.
hartytp
commented
6 years ago Okay, yes, someone has smashed a transistor near those LEDs (my board was never put into a crate, so didn't have this particular issue).
Does that affect the supplies or just the LEDs?
gkasprow
commented
6 years ago just LEDs. It happened on many boards. We shipped you one of the two boards that fully worked and didn't have any sync issues.
marmeladapk
commented
6 years ago @hartytp AMC should have a sticker with rxcclock phase. Please tell me this value.
Did you test the synchronization with the gateware and firmware that were flashed on the board when you got it?
hartytp
commented
6 years ago AMC should have a sticker with rxcclock phase. Please tell me this value.
I don't remember seeing that, but I'll have another look next time I'm in the lab.
Did you test the synchronization with the gateware and firmware that were flashed on the board when you got it?
No, I didn't. Can you send me the binaries you used and I'll re-test. Also, let me know what your setup (e.g. clocks etc)
Did you manage to track down where my AMC + RTM? I've used the same boards for all testing so far, and did some rework on them myself, so I'd really like to finish testing on that same pair of boards instead of switching to some other boards, whose history I know nothing about.
hartytp
commented
6 years ago No red stickers. Did you post me the wrong board? To help you track down my AMC, IIRC it was missing one of the thumbscres on the FP.
marmeladapk
commented
6 years ago @hartytp As far as I can remember I used this gateware and firmware:
I didn't encounter problems you are describing with DACs.
hartytp
commented
6 years ago Thanks, I'll give that a spin on Monday. Will let you know how I get on.
hartytp
commented
6 years ago Thanks @marmeladapk. Strange, with the binaries you sent me, this works fine:
__ __ _ ____ ____
| \/ (_) ___| ___ / ___|
| |\/| | \___ \ / _ \| |
| | | | |___) | (_) | |___
|_| |_|_|____/ \___/ \____|
MiSoC Bootloader
Copyright (c) 2017-2018 M-Labs Limited
Bootloader CRC passed
Gateware ident 4.0.dev+1335.gfab6e5cd.dirty;standalone
Initializing SDRAM...
DQS initial delay: 110 taps
Write leveling scan:
Module 3:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100000000000000000000000000000000000000000000000000
Module 2:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100000000000000000000000000
Module 1:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000
Module 0:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000
DQS initial delay: 110 taps
Write leveling: 102 112 138 128 done
Read leveling scan:
Module 3:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000101010111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100101001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Module 2:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111010001010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Module 1:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000101011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111101010100010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Module 0:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000001011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111010100100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Read leveling: 222+-84 212+-93 191+-84 178+-92 done
SDRAM initialized
Memory test passed
Booting from flash...
Starting firmware.
[ 0.000005s] INFO(runtime): ARTIQ runtime starting...
[ 0.003867s] INFO(runtime): software ident 4.0.dev+1359.g0e741945.dirty;standalone
[ 0.011521s] INFO(runtime): gateware ident 4.0.dev+1335.gfab6e5cd.dirty;standalone
[ 0.019525s] INFO(runtime): log level set to INFO by default
[ 0.025085s] INFO(runtime): UART log level set to INFO by default
[ 0.031044s] INFO(board_artiq::slave_fpga): Loading slave FPGA gateware...
[ 0.037994s] INFO(board_artiq::slave_fpga): magic: 0x5352544d, length: 0x000b4c90
[ 0.986799s] INFO(board_artiq::slave_fpga): ...done
[ 0.990609s] INFO(board_artiq::serwb): waiting for AMC/RTM serwb bridge to be ready...
[ 1.025338s] INFO(board_artiq::serwb): ...done.
[ 1.028708s] INFO(board_artiq::serwb): RTM to AMC link test...
[ 2.511011s] INFO(board_artiq::serwb): ...passed
[ 2.514556s] INFO(board_artiq::serwb): AMC to RTM link test...
[ 3.996865s] INFO(board_artiq::serwb): ...passed
[ 4.000418s] INFO(board_artiq::serwb): Wishbone test...
[ 5.932451s] INFO(board_artiq::serwb): ...passed
[ 5.936307s] INFO(board_artiq::serwb): RTM gateware version 4.0.dev+1359.g0e741945.dirty
[ 5.944161s] INFO(runtime): press 'e' to erase startup and idle kernels...
[ 6.944005s] INFO(runtime): continuing boot
[ 7.206235s] INFO(board_artiq::si5324): waiting for Si5324 lock...
[ 13.455774s] INFO(board_artiq::si5324): ...locked
[ 13.459540s] INFO(board_artiq::hmc830_7043::hmc830): loading HMC830 configuration...
[ 13.467474s] INFO(board_artiq::hmc830_7043::hmc830): ...done
[ 13.473128s] INFO(board_artiq::hmc830_7043::hmc830): setting HMC830 dividers...
[ 13.480591s] INFO(board_artiq::hmc830_7043::hmc830): ...done
[ 13.486413s] INFO(board_artiq::hmc830_7043::hmc830): waiting for HMC830 lock...
[ 13.493833s] INFO(board_artiq::hmc830_7043::hmc830): ...locked
[ 13.500062s] INFO(board_artiq::hmc830_7043::hmc7043): enabling HMC7043
[ 13.516763s] INFO(board_artiq::hmc830_7043::hmc7043): loading configuration...
[ 13.534249s] INFO(board_artiq::hmc830_7043::hmc7043): ...done
[ 13.538929s] INFO(board_artiq::hmc830_7043::hmc7043): testing GPO...
[ 13.546042s] INFO(board_artiq::hmc830_7043::hmc7043): ...passed
[ 13.562256s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 13.574133s] INFO(board_artiq::ad9154): ...done
[ 13.648009s] INFO(board_artiq::ad9154): AD9154-0 running PRBS test...
[ 14.654372s] INFO(board_artiq::ad9154): ...passed
[ 14.658022s] INFO(board_artiq::ad9154): AD9154-0 running STPL test...
[ 14.664810s] INFO(board_artiq::ad9154): c0 errors: 0
[ 14.670013s] INFO(board_artiq::ad9154): c1 errors: 0
[ 14.675223s] INFO(board_artiq::ad9154): c2 errors: 0
[ 14.680433s] INFO(board_artiq::ad9154): c3 errors: 0
[ 14.685359s] INFO(board_artiq::ad9154): ...passed
[ 14.700341s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.707757s] INFO(board_artiq::ad9154): ...done
[ 14.860963s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 14.872686s] INFO(board_artiq::ad9154): ...done
[ 14.946550s] INFO(board_artiq::ad9154): AD9154-1 running PRBS test...
[ 15.952904s] INFO(board_artiq::ad9154): ...passed
[ 15.956552s] INFO(board_artiq::ad9154): AD9154-1 running STPL test...
[ 15.963337s] INFO(board_artiq::ad9154): c0 errors: 0
[ 15.968545s] INFO(board_artiq::ad9154): c1 errors: 0
[ 15.973755s] INFO(board_artiq::ad9154): c2 errors: 0
[ 15.978965s] INFO(board_artiq::ad9154): c3 errors: 0
[ 15.983888s] INFO(board_artiq::ad9154): ...passed
[ 15.998873s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 16.006289s] INFO(board_artiq::ad9154): ...done
[ 16.080521s] INFO(board_artiq::jesd204sync): verifying SYSREF margins at DAC-0...
[ 16.176805s] INFO(board_artiq::jesd204sync): margins: -34 +35
[ 16.182729s] INFO(board_artiq::jesd204sync): verifying SYSREF margins at DAC-1...
[ 16.279063s] INFO(board_artiq::jesd204sync): margins: -34 +35
[ 16.284991s] INFO(board_artiq::hmc542): card 0 channel 0 set to 4 dB
[ 16.292263s] INFO(board_artiq::hmc542): card 0 channel 1 set to 4 dB
[ 16.299478s] INFO(board_artiq::hmc542): card 1 channel 0 set to 4 dB
[ 16.306692s] INFO(board_artiq::hmc542): card 1 channel 1 set to 4 dB
[ 16.313908s] INFO(board_artiq::hmc542): card 2 channel 0 set to 4 dB
[ 16.321122s] INFO(board_artiq::hmc542): card 2 channel 1 set to 4 dB
[ 16.328336s] INFO(board_artiq::hmc542): card 3 channel 0 set to 4 dB
[ 16.335550s] INFO(board_artiq::hmc542): card 3 channel 1 set to 4 dB
[ 16.342994s] INFO(runtime): using MAC address 34-45-32-12-ff-22
[ 16.347952s] INFO(runtime): using IP address 192.168.95.176
[ 16.355122s] INFO(runtime::mgmt): management interface active
[ 16.368270s] INFO(runtime::session): accepting network sessions
[ 16.382221s] INFO(runtime::session): running startup kernel
[ 16.386962s] INFO(runtime::session): no startup kernel found
[ 16.392395s] INFO(runtime::session): no connection, starting idle kernel
[ 16.399537s] INFO(runtime::session): no idle kernel found
@sbourdeauducq can you think of any changes since that build that could account for the issues I'm seeing? (Particularly anything related to the clocking?)
hartytp
commented
6 years ago gateware version: https://github.com/m-labs/artiq/commit/fab6e5cdff1541d4d96f462ce584a867ceffe5e8 firmware version: https://github.com/m-labs/artiq/commit/0e7419450ec926227cc577f1b8b322ad9686f42c (quite a bit newer)
hartytp
commented
6 years ago For good measure, I rebuilt the current artiq master in a fresh conda environment. I'm actually mainly seeing bad SYNC errors. Any ideas which change to artiq could be responsible for this?
__ __ _ ____ ____
| \/ (_) ___| ___ / ___|
| |\/| | \___ \ / _ \| |
| | | | |___) | (_) | |___
|_| |_|_|____/ \___/ \____|
MiSoC Bootloader
Copyright (c) 2017-2018 M-Labs Limited
Bootloader CRC passed
Gateware ident 4.0.dev+1400.g69d060b6;standalone.without-sawg
Initializing SDRAM...
DQS initial delay: 110 taps
Write leveling scan:
Module 3:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111101000000000000000000000000000000000000000000
Module 2:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110110110010000000000000000000000000
Module 1:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111101101000000000000000000000000000000000000000000000000000000000000000
Module 0:
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111101111000000000000000000000000000000000000000000000000000000000000000
DQS initial delay: 110 taps
Write leveling: 102 112 139 128 done
Read leveling scan:
Module 3:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110100010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Module 2:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Module 1:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110101010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Module 0:
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000101011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111101010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Read leveling: 222+-85 211+-92 193+-83 179+-91 done
SDRAM initialized
Memory test passed
Booting from flash...
Starting firmware.
[ 0.000006s] INFO(runtime): ARTIQ runtime starting...
[ 0.003867s] INFO(runtime): software ident 4.0.dev+1400.g69d060b6;standalone.without-sawg
[ 0.012130s] INFO(runtime): gateware ident 4.0.dev+1400.g69d060b6;standalone.without-sawg
[ 0.020778s] INFO(runtime): log level set to INFO by default
[ 0.026302s] INFO(runtime): UART log level set to INFO by default
[ 0.032260s] INFO(board_artiq::slave_fpga): Loading slave FPGA gateware...
[ 0.039210s] INFO(board_artiq::slave_fpga): magic: 0x5352544d, length: 0x000b5ce8
[ 0.046936s] INFO(board_artiq::slave_fpga): DONE before loading
[ 0.999431s] INFO(board_artiq::slave_fpga): ...done
[ 1.003243s] INFO(board_artiq::serwb): waiting for AMC/RTM serwb bridge to be ready...
[ 1.037973s] INFO(board_artiq::serwb): ...done.
[ 1.041346s] INFO(board_artiq::serwb): RTM to AMC link test...
[ 2.523648s] INFO(board_artiq::serwb): ...passed
[ 2.527195s] INFO(board_artiq::serwb): AMC to RTM link test...
[ 4.009504s] INFO(board_artiq::serwb): ...passed
[ 4.013057s] INFO(board_artiq::serwb): Wishbone test...
[ 5.944908s] INFO(board_artiq::serwb): ...passed
[ 5.948704s] INFO(board_artiq::serwb): RTM gateware version 4.0.dev+1400.g69d060b6
[ 5.956098s] INFO(runtime): press 'e' to erase startup and idle kernels...
[ 6.956006s] INFO(runtime): continuing boot
[ 7.218249s] INFO(board_artiq::si5324): waiting for Si5324 lock...
[ 13.517078s] INFO(board_artiq::si5324): ...locked
[ 13.520847s] INFO(board_artiq::hmc830_7043::hmc830): loading HMC830 configuration...
[ 13.528781s] INFO(board_artiq::hmc830_7043::hmc830): ...done
[ 13.534435s] INFO(board_artiq::hmc830_7043::hmc830): setting HMC830 dividers...
[ 13.541899s] INFO(board_artiq::hmc830_7043::hmc830): ...done
[ 13.547720s] INFO(board_artiq::hmc830_7043::hmc830): waiting for HMC830 lock...
[ 13.555140s] INFO(board_artiq::hmc830_7043::hmc830): ...locked
[ 13.561370s] INFO(board_artiq::hmc830_7043::hmc7043): enabling HMC7043
[ 13.578071s] INFO(board_artiq::hmc830_7043::hmc7043): loading configuration...
[ 13.595559s] INFO(board_artiq::hmc830_7043::hmc7043): ...done
[ 13.600242s] INFO(board_artiq::hmc830_7043::hmc7043): testing GPO...
[ 13.607355s] INFO(board_artiq::hmc830_7043::hmc7043): ...passed
[ 13.623570s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 13.635449s] INFO(board_artiq::ad9154): ...done
[ 13.720706s] WARN(board_artiq::ad9154): AD9154-0 config attempt #1 failed (bad SYNC)
[ 13.737523s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 13.749358s] INFO(board_artiq::ad9154): ...done
[ 13.828895s] WARN(board_artiq::ad9154): AD9154-0 config attempt #2 failed (bad CODEGRPSYNCFLG)
[ 13.846581s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 13.858417s] INFO(board_artiq::ad9154): ...done
[ 13.939425s] WARN(board_artiq::ad9154): AD9154-0 config attempt #3 failed (bad SYNC)
[ 13.956238s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 13.968073s] INFO(board_artiq::ad9154): ...done
[ 14.044498s] WARN(board_artiq::ad9154): AD9154-0 config attempt #4 failed (bad CODEGRPSYNCFLG)
[ 14.062185s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.074023s] INFO(board_artiq::ad9154): ...done
[ 14.153298s] WARN(board_artiq::ad9154): AD9154-0 config attempt #5 failed (bad SYNC)
[ 14.170112s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.181949s] INFO(board_artiq::ad9154): ...done
[ 14.261051s] WARN(board_artiq::ad9154): AD9154-0 config attempt #6 failed (bad SYNC)
[ 14.277864s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.289700s] INFO(board_artiq::ad9154): ...done
[ 14.367508s] WARN(board_artiq::ad9154): AD9154-0 config attempt #7 failed (bad CODEGRPSYNCFLG)
[ 14.385193s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.397029s] INFO(board_artiq::ad9154): ...done
[ 14.482046s] WARN(board_artiq::ad9154): AD9154-0 config attempt #8 failed (bad CODEGRPSYNCFLG)
[ 14.499732s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.511569s] INFO(board_artiq::ad9154): ...done
[ 14.589154s] WARN(board_artiq::ad9154): AD9154-0 config attempt #9 failed (bad CODEGRPSYNCFLG)
[ 14.606840s] INFO(board_artiq::ad9154): AD9154-0 initializing...
[ 14.618675s] INFO(board_artiq::ad9154): ...done
[ 14.711075s] WARN(board_artiq::ad9154): AD9154-0 config attempt #10 failed (bad SYNC)
[ 14.717692s] ERROR(board_artiq::ad9154): failed to initialize AD9154-0: bad SYNC
[ 14.735706s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 14.747432s] INFO(board_artiq::ad9154): ...done
[ 14.823268s] WARN(board_artiq::ad9154): AD9154-1 config attempt #1 failed (bad CODEGRPSYNCFLG)
[ 14.840953s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 14.852682s] INFO(board_artiq::ad9154): ...done
[ 14.929847s] WARN(board_artiq::ad9154): AD9154-1 config attempt #2 failed (bad SYNC)
[ 14.946660s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 14.958386s] INFO(board_artiq::ad9154): ...done
[ 15.035174s] WARN(board_artiq::ad9154): AD9154-1 config attempt #3 failed (bad CODEGRPSYNCFLG)
[ 15.052861s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.064590s] INFO(board_artiq::ad9154): ...done
[ 15.138864s] WARN(board_artiq::ad9154): AD9154-1 config attempt #4 failed (bad SYNC)
[ 15.155678s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.167405s] INFO(board_artiq::ad9154): ...done
[ 15.244469s] WARN(board_artiq::ad9154): AD9154-1 config attempt #5 failed (bad SYNC)
[ 15.261282s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.273009s] INFO(board_artiq::ad9154): ...done
[ 15.360116s] WARN(board_artiq::ad9154): AD9154-1 config attempt #6 failed (bad SYNC)
[ 15.376929s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.388657s] INFO(board_artiq::ad9154): ...done
[ 15.463190s] WARN(board_artiq::ad9154): AD9154-1 config attempt #7 failed (bad SYNC)
[ 15.480003s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.491731s] INFO(board_artiq::ad9154): ...done
[ 15.579841s] WARN(board_artiq::ad9154): AD9154-1 config attempt #8 failed (bad CODEGRPSYNCFLG)
[ 15.597527s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.609254s] INFO(board_artiq::ad9154): ...done
[ 15.684615s] WARN(board_artiq::ad9154): AD9154-1 config attempt #9 failed (bad SYNC)
[ 15.701428s] INFO(board_artiq::ad9154): AD9154-1 initializing...
[ 15.713154s] INFO(board_artiq::ad9154): ...done
[ 15.793104s] WARN(board_artiq::ad9154): AD9154-1 config attempt #10 failed (bad SYNC)
[ 15.799719s] ERROR(board_artiq::ad9154): failed to initialize AD9154-1: bad SYNC
[ 15.807458s] INFO(board_artiq::jesd204sync): aligning SYSREF with RTIO...
[ 15.831430s] INFO(board_artiq::jesd204sync): ...done (45/64 slips)
[ 15.839572s] INFO(board_artiq::jesd204sync): margins at FPGA: -17 +17
[ 15.845248s] INFO(board_artiq::jesd204sync): verifying SYSREF margins at DAC-0...
[ 15.858703s] INFO(board_artiq::jesd204sync): margins: -0 +1
[ 15.863220s] ERROR(runtime): failed to align SYSREF at DAC: SYSREF margins at DAC are too small, board needs recalibration
[ 15.874250s] INFO(board_artiq::hmc542): card 0 channel 0 set to 4 dB
[ 15.882762s] INFO(board_artiq::hmc542): card 0 channel 1 set to 4 dB
[ 15.889979s] INFO(board_artiq::hmc542): card 1 channel 0 set to 4 dB
[ 15.897196s] INFO(board_artiq::hmc542): card 1 channel 1 set to 4 dB
[ 15.904414s] INFO(board_artiq::hmc542): card 2 channel 0 set to 4 dB
[ 15.911631s] INFO(board_artiq::hmc542): card 2 channel 1 set to 4 dB
[ 15.918849s] INFO(board_artiq::hmc542): card 3 channel 0 set to 4 dB
[ 15.926066s] INFO(board_artiq::hmc542): card 3 channel 1 set to 4 dB
[ 15.933525s] INFO(runtime): using MAC address 34-45-32-12-ff-22
[ 15.938493s] INFO(runtime): using IP address 192.168.95.176
[ 15.945599s] INFO(runtime::mgmt): management interface active
[ 15.958751s] INFO(runtime::session): accepting network sessions
[ 15.972694s] INFO(runtime::session): running startup kernel
[ 15.977428s] INFO(runtime::session): no startup kernel found
[ 15.982863s] INFO(runtime::session): no connection, starting idle kernel
[ 15.990011s] INFO(runtime::session): no idle kernel found
sbourdeauducq
commented
6 years ago Is your JESD core up-to-date?
hartytp
commented
6 years ago (artiq-dev) tph@buildsvr:~/scratch$ conda list
# packages in environment at /home/tph/anaconda3/envs/artiq-dev:
#
# Name Version Build Channel
aiohttp 3.1.3 py35_0 conda-forge/label/main
alabaster 0.7.11 py_3 conda-forge/label/main
artiq 4.0.dev0+1400.g69d060b6 <pip>
artiq-dev 4.0.dev 1403+git92be9324 m-labs/label/dev
asn1crypto 0.24.0 py35_3 conda-forge/label/main
async-timeout 2.0.1 py35_0 conda-forge/label/main
asyncserial 0.1 py_13+git340e430 m-labs/label/main
attrs 18.2.0 py_0 conda-forge/label/main
babel 2.6.0 py_1 conda-forge/label/main
binutils-or1k-linux 2.30 7 m-labs/label/main
blas 1.0 mkl
bscan-spi-bitstreams 0.10.0 2 m-labs/label/main
bzip2 1.0.6 h470a237_2 conda-forge/label/main
ca-certificates 2018.8.24 ha4d7672_0 conda-forge/label/main
certifi 2018.8.24 py35_1001 conda-forge/label/main
cffi 1.11.5 py35h5e8e0c9_1 conda-forge/label/main
chardet 3.0.4 py35_3 conda-forge/label/main
colorama 0.3.9 py_1 conda-forge/label/main
coverage 4.5.1 py35h470a237_1 conda-forge/label/main
cryptography 2.3.1 py35hdffb7b8_0 conda-forge/label/main
cryptography-vectors 2.3.1 py35_0 conda-forge/label/main
dbus 1.13.0 h3a4f0e9_0 conda-forge/label/main
docutils 0.14 py35_1 conda-forge/label/main
expat 2.2.5 hfc679d8_2 conda-forge/label/main
fontconfig 2.13.1 h65d0f4c_0 conda-forge/label/main
freetype 2.9.1 h6debe1e_4 conda-forge/label/main
gettext 0.19.8.1 h5e8e0c9_1 conda-forge/label/main
glib 2.55.0 h464dc38_2 conda-forge/label/main
gst-plugins-base 1.12.5 hde13a9d_0 conda-forge/label/main
gstreamer 1.12.5 h61a6719_0 conda-forge/label/main
h5py 2.8.0 py35hb794570_1 conda-forge/label/main
hdf5 1.10.2 hc401514_2 conda-forge/label/main
icu 58.2 hfc679d8_0 conda-forge/label/main
idna 2.7 py35_2 conda-forge/label/main
idna_ssl 1.0.0 0 conda-forge/label/main
imagesize 1.1.0 py_0 conda-forge/label/main
intel-openmp 2019.0 118
jesd204b 0.10 py_1 m-labs/label/main
jinja2 2.10 py_1 conda-forge/label/main
jpeg 9c h470a237_1 conda-forge/label/main
levenshtein 0.12.0 py35_1 m-labs/label/main
libcurl 7.61.0 h1ad7b7a_0
libffi 3.2.1 hfc679d8_5 conda-forge/label/main
libgcc-ng 7.2.0 hdf63c60_3 conda-forge/label/main
libgfortran 3.0.0 1 conda-forge/label/main
libgfortran-ng 7.2.0 hdf63c60_3 conda-forge/label/main
libgit2 0.24.1 7 m-labs/label/main
libiconv 1.15 h470a237_3 conda-forge/label/main
libpng 1.6.35 ha92aebf_2 conda-forge/label/main
libssh2 1.8.0 h5b517e9_2 conda-forge/label/main
libstdcxx-ng 7.2.0 hdf63c60_3 conda-forge/label/main
libusb 1.0.20 0 m-labs/label/main
libuuid 2.32.1 h470a237_2 conda-forge/label/main
libxcb 1.13 h470a237_2 conda-forge/label/main
libxml2 2.9.8 h422b904_5 conda-forge/label/main
lit 0.4.1 py_9 m-labs/label/main
llvm-or1k 6.0.0 25 m-labs/label/main
llvmlite-artiq 0.23.0.dev py35_4 m-labs/label/main
markupsafe 1.0 py35h470a237_1 conda-forge/label/main
microscope 1.3 py_1 m-labs/label/main
migen 0.7 py35_73+gitbef9dea m-labs/label/dev
misoc 0.11 py35_31+git5ce139dd m-labs/label/dev
mkl 2019.0 118
mkl_fft 1.0.6 py35_0 conda-forge/label/main
mkl_random 1.0.1 py35_0 conda-forge/label/main
msgpack-python 0.5.6 py35h2d50403_2 conda-forge/label/main
multidict 4.4.0 py35h470a237_0 conda-forge/label/main
ncurses 6.1 hfc679d8_1 conda-forge/label/main
numpy 1.15.0 py35h1b885b7_0
numpy-base 1.15.0 py35h3dfced4_0
openocd 0.10.0 6 m-labs/label/main
openssl 1.0.2p h470a237_0 conda-forge/label/main
outputcheck 0.4.2 py_7 m-labs/label/main
pcre 8.41 hfc679d8_3 conda-forge/label/main
pip 18.0 py35_1 conda-forge/label/main
prettytable 0.7.2 py_2 conda-forge/label/main
pthread-stubs 0.4 h470a237_1 conda-forge/label/main
pycparser 2.19 py_0 conda-forge/label/main
pygit2 0.24.0 py35_4 m-labs/label/main
pygments 2.2.0 py_1 conda-forge/label/main
pyopenssl 18.0.0 py35_0 conda-forge/label/main
pyqt 5.6.0 py35h8210e8a_7 conda-forge/label/main
pyqtgraph 0.10.0 py_3 conda-forge/label/main
pyserial 3.4 py35_0 conda-forge/label/main
pysocks 1.6.8 py35_2 conda-forge/label/main
python 3.5.5 h5001a0f_2 conda-forge/label/main
python-dateutil 2.7.3 py_0 conda-forge/label/main
pythonparser 1.1 py_8 m-labs/label/main
pytz 2018.5 py_0 conda-forge/label/main
qt 5.6.2 hf70d934_9 conda-forge/label/main
quamash 0.5.5 py_4 m-labs/label/main
readline 7.0 haf1bffa_1 conda-forge/label/main
regex 2015.11.22 py35_1 m-labs/label/main
requests 2.19.1 py35_1 conda-forge/label/main
rust-core-or1k 1.28.0 21 m-labs/label/main
rustc 1.28.0 21 m-labs/label/main
scipy 1.1.0 py35hfc37229_0
setuptools 33.1.1 py35_0 conda-forge/label/main
sip 4.18.1 py35hfc679d8_0 conda-forge/label/main
six 1.11.0 py35_1 conda-forge/label/main
snowballstemmer 1.2.1 py_1 conda-forge/label/main
sphinx 1.4.8 py35_0 conda-forge/label/main
sphinx-argparse 0.1.13 py_4 m-labs/label/main
sphinx_rtd_theme 0.4.1 py_0 conda-forge/label/main
sphinxcontrib-wavedrom 1.1.0 <pip>
sphinxcontrib-wavedrom 1.1.0 py_1 m-labs/label/main
sqlite 3.25.1 hb1c47c0_0 conda-forge/label/main
system 5.8 2
tk 8.6.8 ha92aebf_0 conda-forge/label/main
urllib3 1.23 py35_1 conda-forge/label/main
wheel 0.31.1 py35_1001 conda-forge/label/main
xorg-libxau 1.0.8 h470a237_6 conda-forge/label/main
xorg-libxdmcp 1.1.2 h470a237_7 conda-forge/label/main
xz 5.2.4 h470a237_1 conda-forge/label/main
yarl 1.2.6 py35h470a237_0 conda-forge/label/main
zlib 1.2.11 h470a237_3 conda-forge/label/main(so, yes)
hartytp
commented
6 years ago But, @marmeladapk's build that works predates the new JESD core, so I was wondering if the update was the issue...
sbourdeauducq
commented
6 years ago Ok, then I have no better idea than building the version @marmeladapk was using, checking if the problem is still there, and then bisecting
hartytp
commented
6 years ago Okay, I'll do that. On the off-chance though, can you remind me where the best place to measure vccint is? I'd like to check that rework was done on the board I have, otherwise we'll be back into the bad old days of builds breaking with the phase of the moon
sbourdeauducq
commented
6 years ago BTW, note that the new version of the JESD core is not compatible with the old - you need to have the matching ARTIQ version. The conda file in ARTIQ should tell you which JESD version to use. VCCINT can be measured on the two large 0R resistors that are on the back side of the PCB and connected to the large coil.
hartytp
commented
6 years ago 0.96V so fine
hartytp
commented
6 years ago @sbourdeauducq building with artiq fab6e5cdff1541d4d96f462ce584a867ceffe5e8 and misoc 57ebe119d80beabad30d232fc3c9229882042807 I'm getting errors like
error[E0259]: the name `compiler_builtins` is defined multiple times
--> libboard_misoc/lib.rs:4:1
|
4 | extern crate compiler_builtins;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `compiler_builtins` reimported hereany idea why?
sbourdeauducq
commented
6 years ago You need to track the rustc version as well.
hartytp
commented
6 years ago what's the easiest way of doing that?
Maybe it's easier to stick with the latest artiq and just manually revert individual commits.
hartytp
commented
6 years ago Relevant looking artiq commits
Let's start by reverting all of those as well as the jesd core updates
sbourdeauducq
commented
6 years ago I would first double-check that you can reproduce @marmeladapk's results and build working gateware at all (with his version).
hartytp
commented
6 years ago Reverting those commits does indeed fix the issue.
sbourdeauducq
commented
6 years ago Can you pinpoint which is causing the issue? I did test the new JESD core so this is surprising. Could it be vivado/P&R/timing problems?
hartytp
commented
6 years ago Can you pinpoint which is causing the issue
Yes, but it will take some time.
I did test the new JESD core so this is surprising. Could it be vivado/P&R/timing problems?
My build scripts flag any timing issues as well as any vivado warnings that I haven't explicitly white listed. So far I haven't see anything suspicious.
hartytp
commented
6 years ago Okay, building with everything reverted apart from the update to the new JESD core.
hartytp
commented
6 years ago @sbourdeauducq what Sayma build did you test the new JESD core with? Standalone or master?
sbourdeauducq
commented
6 years ago Standalone.
hartytp
commented
6 years ago @sbourdeauducq remind me: are the commit strings in the artiq logs based on the actual git version it was built with or information from conda? ie do we trust that @marmeladapk's builds actually use the code revision in those logs?
sbourdeauducq
commented
6 years ago I'm not aware of any bug with the version string.
hartytp
commented
6 years ago ok
hartytp
commented
6 years ago odd...still works with the new jesd core.
marmeladapk
commented
6 years ago Commit ids:
artiq: 0e74194
misoc: 0a77fd4
migen: a6082d5
jesd204b: 03d3280That's what I used + changed rx clock phase in misoc and commented out rtio sysref alignment in jesd204b.
hartytp
commented
6 years ago artiq: 0e74194
Your gateware is older than that according to the log I posted.
hartytp
commented
6 years ago and, that uses the new version of the jesd204b core, which is compatible with the artiq revision you posted above, but not with the gateware revision reported in the logs I posted.
hartytp
commented
6 years ago Okay, will have to finish this some other time.
I've started going through the list I posted above and reintroducing commits. So far I haven't been able to reproduce this issue.
hartytp
commented
6 years ago I have no idea. Can't reproduce this any more. Will repoen the issue if I see it again, but for now this board seems to work.
marmeladapk
commented
6 years ago I noticed the same error + failed prbs tests on MasterDAC variant.
hartytp
commented
6 years ago Board damage? At least one board has these symptoms because of damaged coupling capacitors on the clock line (I'm beginning to think that these boards have become rather ragged from repeated rework and global shipping, and trying to get them to work might be a loosing battle -- time to spin some new ones up with the fixes properly implemented)
marmeladapk
commented
6 years ago It worked just fine in Warsaw and here with the same gateware. Only thing that changed was gateware, I didn't even move it from crate.
marmeladapk
commented
6 years ago It dissapeared. That's frustrating. I think I'm more conviced, that it's board fatigue.