Allaki in slots 3 and 4 are not working

[sbourdeauducq]

• DAC signal is present at the SMP connectors
• No signal visible at the Allaki SMA connectors
• The problem is on the Sayma side (as can be demonstrated by swapping Allakis)
• Could be a problem with the hardware, or with the RF switch and attenuator programming in gateware/firmware

[gkasprow]

@sbourdeauducq could be also a problem with FPGA pin assignment in some of the slots. I tested Allaki operation on all boards except for very first 2 prototypes where Allaki were not available. Can you test if the signal is missing after the amplifier? You need to solder a piece of wire to access it. Another issue could be with migen - I observed that one signal declared LOW was high on some boards. When it was register controlled, the problem disappeared. Looks like Vivado optimised it somehow.

[sbourdeauducq]

Another issue could be with migen - I observed that one signal declared LOW was high on some boards. When it was register controlled, the problem disappeared. Looks like Vivado optimised it somehow.

You have interesting ideas of what to blame and what to praise. In my book, if Vivado generates a logic low from assign allaki3_rfsw1 = 1'd1; (which is what Migen generates), it is called a "Vivado bug", not a "Vivado optimization" or an "issue with Migen". What signal was it, exactly?

[gkasprow]

@sbourdeauducq I didn't look where the problem happened. I tested binaries only. Could be either migen or Vivado. That's true, my statement was not precise :) In vivado you can have mixture of settings that treat differently such statements so it may not be a bug. I described the issue here

[sbourdeauducq]

In vivado you can have mixture of settings that treat differently such statements

Where? Anyway all the statements generated by Migen to set outputs to fixed levels are the same, and they are compiled with the same Vivado options.

[gkasprow]

I didn't play with them, my colleague (@wzab) told me that once had similar case in the past. And it was some option to change the compiler behaviour. But anyway, there is certainly an issue with this Artix chip, at least with one of the pins. To check it, route the pins that you want to bind to vcc/gnd to the register and write the value you want to have. This solved the issue in my case.

[jbqubit]

there is certainly an issue with this Artix chip

Is this reproducible in a stand alone bit of VHDL? Does @gkasprow approach work?

[whitequark]

I've compared migen pin assignment with schematics and everything matches.

[jbqubit]

@sbourdeauducq Did you try to trick that @gkasprow suggested on Feb 9? What is scope of IO error for Slots 3 and 4? Single IO line or all the IO lines?

[whitequark]

@jbqubit I'm working on this. I will check that tomorrow or the next day; so far I've not been able to get any output from DACs whatsoever because of serwb hangs.

[sbourdeauducq]

I've not been able to get any output from DACs whatsoever because of serwb hangs.

Even with the AMC/RTM combination that I had working? (Florent's AMC, and RTM with the resistor in DAC_CLK_N) As with everything in Sayma, there is a lot of board-to-board variation...

[whitequark]

Even with the AMC/RTM combination that I had working? (Florent's AMC, and RTM with the resistor in DAC_CLK_N)

On that board I got past JESD initialization (unlike with the other AMC), but I think it still hung somewhere.

[sbourdeauducq]

Also I wonder if the RTM connector sometimes make poor contact.

[whitequark]

What's the point in that complicated expensive connector if it doesn't even work well?

[sbourdeauducq]

I'm not sure if it's really a source of problems, just something to check - sometimes, problems disappeared when reconnecting the rtm, but it could have been just chance.

[hartytp]

Although that reminds me that we should fit the guide pun to the boards. @gkasprow can you post one for each Sayma?

[gkasprow]

I assembled them on most of the boards. I still have one or two pieces in my lab. Who else is missing the guiding pun?

[jbqubit]

@gkasprow I'm missing the AMC-side (female) pun on one board.

[sbourdeauducq]

@whitequark I suspect your problems are due to incompatible or outdated RTM gateware. I just updated everything and it worked. The RTM gateware that you had loaded into the board was quite old.

[     0.000006s]  INFO(runtime): ARTIQ runtime starting...
[     0.003887s]  INFO(runtime): software version 4.0.dev+803.gf0771765
[     0.010154s]  INFO(runtime): gateware version 4.0.dev+803.gf0771765
[     0.016424s]  INFO(runtime): log level set to INFO by default
[     0.022134s]  INFO(runtime): UART log level set to INFO by default
[     0.028286s]  INFO(board_artiq::serwb): waiting for AMC/RTM serwb bridge to be ready...
[     0.504989s]  INFO(board_artiq::serwb): done.
[     0.508113s]  INFO(board_artiq::serwb): RTM gateware version 4.0.dev+803.gf0771765
[     0.515581s]  INFO(runtime): press 'e' to erase startup and idle kernels...
[     1.515007s]  INFO(runtime): continuing boot
[     1.518022s]  INFO(board_artiq::hmc830_7043::hmc7043): HMC7043 found
[     1.524288s]  INFO(board_artiq::hmc830_7043::hmc7043): HMC7043 configuration...
[     1.542751s]  INFO(board_artiq::ad9154): AD9154-0 found
[     1.546670s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[     1.629690s]  INFO(board_artiq::ad9154): AD9154-0 PRBS test
[     2.644680s]  INFO(board_artiq::ad9154): AD9154-0 found
[     2.648590s]  INFO(board_artiq::ad9154): AD9154-0 configuration...
[     2.741913s]  INFO(board_artiq::ad9154): AD9154-1 found
[     2.745830s]  INFO(board_artiq::ad9154): AD9154-1 configuration...
[     2.828846s]  INFO(board_artiq::ad9154): AD9154-1 PRBS test
[     3.843844s]  INFO(board_artiq::ad9154): AD9154-1 found
[     3.847754s]  INFO(board_artiq::ad9154): AD9154-1 configuration...
[     3.946633s]  WARN(runtime): using default MAC address 02-00-00-00-00-01; consider changing it
[     3.954106s]  INFO(runtime): using IP address 192.168.1.60
[     3.959673s] ERROR(runtime::rtio_mgt): unrecognized startup_clock configuration entry, using internal RTIO clock
[     3.971213s]  INFO(runtime::mgmt): management interface active
[     3.984428s]  INFO(runtime::session): accepting network sessions
[     3.998651s]  INFO(runtime::session): running startup kernel
[     4.003168s]  INFO(runtime::session): no startup kernel found
[     4.008866s]  INFO(runtime::session): no connection, starting idle kernel
[     4.015745s]  INFO(runtime::session): no idle kernel found

@hartytp @gkasprow @jbqubit this issue is not the place to discuss unrelated hardware supplies.

[whitequark]

@sbourdeauducq No, I've built the gateware for the master branch at the time when I was testing after you told me that it was too old. Seems like random breakage.

[sbourdeauducq]

Problem is not consistent, https://github.com/sinara-hw/sinara/issues/536#issuecomment-377556508 @whitequark Have you reproduced it?

[whitequark]

@sbourdeauducq Maybe I'm doing something wrong but I can't see any output from DACs right now.

[whitequark]

I confirm that on M-Labs Sayma 3 all eight DAC channels have sawtooth output on both P and N SMPs.

[jbqubit]

@whitequark Great! Can you also set RF attenuators and switchs?

[whitequark]

@jbqubit It was too late to check that today, I plan to do it tomorrow.

[sbourdeauducq]

@whitequark Any update? When I reported this issue I wanted to have one Allaki on the first DAC chip and another on the second DAC chip, to test DAC synchronization. Are you able to get outputs from both DACs using two Allaki?

[whitequark]

Not yet.

[jbqubit]

@whitequark What exactly are the steps you've done to test on your board?

[sbourdeauducq]

@whitequark ping.

[jbqubit]

@whitequark Have you tried RF switches and attenuators?

[whitequark]

@jbqubit Right now there is the problem that the clock generator we have been using in the lab died, so I'm figuring out how to set up NVSynth.

[jbqubit]

There's a serial interface... https://windfreaktech.com/wp-content/uploads/docs/synthnv/serialcomm.pdf

watch out for large power in harmonics for these synths

[whitequark]

I've flashed an old firmware that has been proven to work some weeks ago built based on hashes I recorded then. I've tried using two other Sayma RTMs (and ensured the proper resistor is soldered to DAC_CLK_N). I've tried different cables. I've tried using NVSynth instead of our clock generator (which appears to not have died after all, it does output RF at frequencies I can check with a scope here). I've tried skipping DAC0 and starting with initializing DAC1.

None of this seems to make any difference, the AD9154 SERDES PLL just doesn't lock. What are the next steps? Do the AD9154 registers offer any insight into why it doesn't lock?

[sbourdeauducq]

Works for me with the hardware in whatever state you left it. I guess you are not setting the clock switches correctly. Use this patch:

diff --git a/artiq/firmware/libboard_artiq/hmc830_7043.rs b/artiq/firmware/libboard_artiq/hmc830_7043.rs
index 44bf0f967..a9fe589b7 100644
--- a/artiq/firmware/libboard_artiq/hmc830_7043.rs
+++ b/artiq/firmware/libboard_artiq/hmc830_7043.rs
@@ -22,7 +22,7 @@ mod clock_mux {
             csr::clock_mux::out_write(
                 1*CLK_SRC_EXT_SEL |  // use ext clk from sma
                 1*REF_CLK_SRC_SEL |
-                1*DAC_CLK_SRC_SEL);
+                0*DAC_CLK_SRC_SEL);
         }
     }
 }
@@ -254,6 +254,6 @@ mod hmc7043 {
 pub fn init() -> Result<(), &'static str> {
     clock_mux::init();
     /* must be the first SPI init because of HMC830 SPI mode selection */
-    hmc830::init()?;
+    //hmc830::init()?;
     hmc7043::init()
 }

This is ARTIQ f96f597e.

[jbqubit]

and ensured the proper resistor is soldered to DAC_CLK_N)

I'm not using a resistor on my boards. I use a 180-deg power splitter and simultaneously drive DAC_CLK_N and DAC_CLK_P with +10 dBm from my synth. But resistor should work fine too.

[jbqubit]

@whitequark What's the status of this?

[whitequark]

@jbqubit Today I've determined that on master, even if AD9154 is initialized (and it gets initialized with @sbourdeauducq's patch that correctly sets up the mux), there is no sawtooth output. I was going to rebuild the commit I initially tested this with and check if that works, but Sayma 3 was busy and I left the lab before it got too late. Will continue tomorrow.

[whitequark]

I've checked with the commit that used to work. RTMs "1" and "2" lock and there is just 3V at the DAC SMPs. RTM "F" does not lock. This happens with either clock generator.

[enjoy-digital]

@whitequark: if RTM "F" is the board i had before, the first AD9154 is not working. (hardware patch missing?). Only the second one works. (To test with ARTIQ i was changing this: https://github.com/m-labs/artiq/blob/master/artiq/firmware/libboard_artiq/ad9154.rs#L615 to start at 1).

[sbourdeauducq]

@whitequark: if RTM "F" is the board i had before,

Yes, it's that one.

[jbqubit]

@whitequark Do you get output from the other DAC? Sawtooth? SAWG?

if RTM "F" is the board i had before, the first AD9154 is not working. (hardware patch missing?)

Please attach a label to the board noting that the DAC is not functioning to prevent confusion. And so it makes its way eventually back to WUT for repair.

[whitequark]

@jbqubit I did not see any output from any of the DACs on any of the RTMs.

[jbqubit]

Did you check all three Sayma at M-Labs? What do you see on the terminal?

[whitequark]

As I've said, I checked all three RTMs. The terminal output indicates that AD9154 was initialized correctly on RTMs "1" and "2", and PRBS test passed.

[gkasprow]

Are you sure that the wires that supply termination voltages are OK? I tested all DACs prior shipment...

[whitequark]

@gkasprow I used to get the expected output from DACs before, so it's not your fault, it's something else.

[sbourdeauducq]

Tested it again today, got output from slots 1 and 3 using Allaki. Joe also got output on most channels: https://github.com/sinara-hw/sinara/issues/536#issuecomment-377556508

Closing this.