SYNC_IN jitter

[jordens]

The jitter on the SYNC_IN signal from Kasli to the AD9910 (throught the LVDS buffers and the fanout) is very high in some caes (the tester setup connected to the buildbot).

At validation delay 1 (hold and setup margin 1 tap) the window is just 2 taps wide (a tap is about 75 ps). http://buildbot.m-labs.hk/builders/artiq/builds/2669/steps/python_unittest_2/logs/stdio

This is the SMP_ERR matrix on tester, rows are increasing validation delay, columns are SYNC_IN delay on the AD9910.:

[1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

There also seems to be some bounce around the edges (top row).

On the systems I have here I get about 5-6 tap wide windows at validation delay 1. That's not stellar but OK. When using the SYNC signal on board from the first DDS, the window at validation delay 1 is 8 taps wide on tester, 8-9 taps here. Assuming equal tap delay for the validation delays and the SYNC_IN delays, the theoretically best case is validation delay 4 and a window width of ~4 or a validation delay of 1 and a window width of ~10 (i.e. SYNC_IN delay periodicity minus twice the validation delay).

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0]
[1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
[1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
[1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]
[1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

In both cases Kasli/v1.1 and Urukul-AD9910/v1.3, connected via MMCX to Kasli-J1.

The jitter seems to come in part from Urukul and in part (the larger part) from Kasli. And it varies between setups. I changed a couple things (see the artiq changelog) to optimize jitter on the RTIO clock but there was little effect. From Vivado the max peak-peak jitter on the clock driving the SYNC output buffer in the FPGA is ~90ps. I tried running the SYNC fanout from the supposedly quieter P1V8A rail but that doesn't seem to work at all.

@gkasprow @marmeladapk could you have a look at the jitter on SYNC_IN (EEM0:7, before and after the sync fanout, compared to the Kasli MMCX clock)?

@cjbe @klickverbot When you were playing with SYNC, did you look at SMP_ERR? How did you select the SYNC_IN delay and the validation delay? Did you scan them?

c.f. m-labs/artiq#1143

[marmeladapk]

@hartytp Compare different measurements I posted, that's what I see.

[jordens]

@marmeladapk You may want to crank up your horizontal scale. We're looking for something at a couple 100ps here. That'll be hard to see on 20ns/div. The clock to hit is 1 GHz and each delay/setup/hold tap "pixel" is 75-80 ps in the scan, horizontal periodicity 13 taps.

The REF_CLK to SYNC_IN phase should be determined only by the non-deterministic divider on Urukul. If you are using the MMCX on the backplane adapter it's also the other Si fanout divider on Kasli which may make deterministic absolute phase w.r.t. Kasli impossible. But that's irrelevant here and phase determinism doesn't matter at the moment for this issue.

I don't see why SYN_IN to SYNC_OUT should be deterministic at all if sync_sel is such that the SYNC from Kasli is used. Determinism also doesn't matter here.

The only thing I see is that SYN_OUT has half the swing as SYNC_IN. Maybe that's too much on SYNC_IN. But it seems unlikely (see below).

Notes to self so I don't check this yet again because for the tenth time...

• Unclear what the swing spec of the SYNC_IN is on the AD9910. If it's like REF_CLK then 100-2000 mV p-p swing are good. But that one doesn't have internal termination and is probably different from SYNC_IN which is declared "LVDS".
• Couldn't find a dev board that fully implements this. The single-chip eval has 100R diff DC term on SYNC_OUT
• 100R differential term DC coupled is fine for the Si outputs. Referenced in the SI datasheet. AD9910 claims to have that 100R internal diff on SYNC_IN.
• ADI ref suggests AD9510 as fan out: differential output voltage 250-450 (360 typ) mV, 100R term, datasheet shows 550 mV differential pk-pk at 800 MHz, 700 mV differential swing at 100 MHz.
• Also suggested ADCLK846. 250-450mV differential output, 600mV swing at 200 MHz.
• SI has single ended output swing of 250-490 mV (with 100 R term). That seems to match well.
• AD9510 and SI have the same VCM.

[hartytp]

The only thing I see is that SYN_OUT has half the swing as SYNC_IN. Maybe that's too much on SYNC_IN. But it seems unlikely (see below).

Isn't the observation that using DDS0 SYNC_OUT as the SYNC_IN source works well? If so, I don't see how the fanout buffer logic levels can be an issue, since we use the same buffer to distribute SYNC_IN regardless of the source.

[jordens]

Could you have a go at summarizing the observations? What I see in the screenshots above with sync_sel=1 are mostly really bad scans.

[hartytp]

@marmeladapk can you have a go at retaking that data with better horizontal resolution, so we can see jitter at the ~100ps level?

As @jordens says, it would be really useful to have a summary of the data you've taken, since it still looks like there is something funny going on in your scans with DDS0 SYNC_OUT as the SYNC_IN source -- none of them look like the one @jordens posted at the top of this issue (the eye vanishes for smaller window sizes much faster than one would expect based on the eye size with larger windows).

[marmeladapk]

@hartytp One moment, @wizath broke off some cables when he stole cables from my desk ;)

[marmeladapk]

Sync source = 1:

---

..[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0] [0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 11 2 [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

---

..[0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1] [0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1] [0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 14 0 [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1]

(changed scales here)

---

..[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0] [0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 13 0 [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]

---

..[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0] [0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 10 3 [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

---

..[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0] [0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 10 3 [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

---

..[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0] [0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 14 0 [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0]

---

Sync source = 0 (FPGA) ..[0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0] [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 12 0 [0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1]

[hartytp]

Thanks @marmeladapk to confirm: that was taken with the scope on persist and recording samples through the entire calibration process, right?

No evidence of glitches or excessive jitter (I assume it's limited by scope noise?). So, the signals look clean AFAICT ...

[marmeladapk]

@hartytp This was taken after the calibration. During the calibration sync_in signal shifts and you quickly have all screen pink. Tommorow I can record how it looks like with finite persitence.

[hartytp]

@marmeladapk what I'm worried about is cross-talk on the EEM cable that increases the jitter on the FPGA SYNC_IN during SPI transactions. So, it's important to look at the FPGA SYNC_IN during the entire calibration.

The FPGA SYNC_IN does not change during cal, so you should be able to do this.

[marmeladapk]

From bootup:

Closeup:

Closeup without power cycle:

These are all separate calibrations.

[hartytp]

what's generating SYNC_IN there?

[marmeladapk]

@hartytp FPGA (sync_sel = 0)

[hartytp]

Does that bottom plot imply that, without power cycling the FPGA, it puts out two different SYNC_IN phases during the calibration?

[hartytp]

Might be worth triggering off an external timebase to make it clearer what is a clock glitch versus a SYNC_IN glitch.

[marmeladapk]

Triggered from Kasli MMCX clock.

10s persistence, you can see there were 3 dds_clk phases, however afaict this doesn't change during scan.

[hartytp]

Okay, so the clock buffer is not giving a deterministic input -> output phase relationship for the DDS clock?

[hartytp]

Are all control pins on that buffer correctly driven (e.g. no floating divider reset etc).

[marmeladapk]

@hartytp During tests IN_SEL on IC19 switches for a moment to high.

[hartytp]

Odd...I'd have to recheck the ARTIQ code to make sure that's unexpected.

[hartytp]

@marmeladapk can you try shorting IN_SEL so that the MMCX clock is always used and then take another eye scan, please? If that looks good then this is just some SW issue with the CPLD config.

[marmeladapk]

MMCX OSC sel = 1, IN_SEL = 0, triggered from Kasli MMCX, sync_sel = 0 (FPGA)

..[0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0] [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 13 0 [0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1]

[hartytp]

Thanks! Okay, well still bad eye scans even though there are no visible glitches on the SYNC line (previous measurements suggest that there wasn't excessive jitter there either).

I'm out of ideas for things to test on the hw...

[hartytp]

@marmeladapk if you still have that setup in tact, can you repeat that last measurement on a finer timebase. I'd like to triple check there are no glitches on the SYNC_IN line.

[marmeladapk]

@hartytp I'll do it on Tuesday.

[hartytp]

thanks

[gkasprow]

we can try to see with same scope both FPGA clock and SYNC signal directly on the Kasli. SYNC is available on EEM, the clock is also easy to observe. In this way, we would see exactly what FPGA feels.

[marmeladapk]

[hartytp]

Thanks @marmeladapk! Nothing there looks suspicious. Since you're looking at SYNC_OUT as well as SYNC_IN, I believe this also rules out things like noise on the DDS PLL (as SYNC_OUT is derived from the internal clock independently of SYNC_IN).

I suggest we leave this for now and produce a release that fixes the other issues.

[hartytp]

@cjbe spent some more time looking at this issue a few weeks back. IIRC he was looking at the output of IC16 on a fast scope using a balun and coax lines soldered to the EEM connector pins. Scope on persist to also catch glitches etc.

Modified the ARTIQ driver so that we can switch between DDS0 and Kasli as the sync sources by only changing CLK_SEL and nothing else.

Running eye scans on DDS0 with the two sync sources. All done with 0 validation window as recommended by ADI (IIRC there is a post on the forum about how other validation delays aren't expected to work, but that's not explained in the data sheet).

Good eye scans with DDS0 as SYNC source, bad with Kasli as source. On the scope, there is no visible difference in the jitter with the two sync sources. Measurement was good enough to rule out there being enough jitter to account for the bad eye scans. The only visible change in waveforms (other than the DC phase being different due to cable lengths etc) was that the Kasli duty cycle is ~55%, while DDS0 is bang on 50%. Hard to see how small changes in duty cycle could cause issues (the DDS is DC coupled and edge-sensitive).

Next thing to do would be to repeat this measurement at the DDS pin, but so far this remains a bit of a mystery.

[gkasprow]

Can you include the EEM carrier or at least long ribbon cable to the loop? It might be an issue with crosstalk...

[hartytp]

Can you include the EEM carrier or at least long ribbon cable to the loop? It might be an issue with crosstalk...

That was included and doesn't appear to be the issue.

[WeiDaZhang]

The sync_smp_err shows a "double-window" in sync_receiver tap scan can be resolved by adding an ODDR in rtio.phy.ttl_simple.ClockGen in the gateware on Kasli.

It typically occurs as the following. The rows are sync receiver delay taps, and columns are 4 channels on a urukul. [0, 0, 0, 0] [0, 0, 0, 0] [1000, 1000, 917, 194] [1000, 1000, 1000, 1000] [7, 638, 1000, 1000] [805, 152, 0, 0] [1000, 1000, 1000, 5] [1000, 1000, 1000, 1000] [100, 334, 1000, 1000] [0, 0, 0, 1000] [0, 0, 0, 0] [0, 0, 0, 0] [0, 0, 0, 0] [0, 0, 0, 0] [847, 978, 2, 0] [1000, 1000, 1000, 1000] [1000, 1000, 1000, 1000] [0, 0, 675, 1000] [1000, 1000, 1000, 0] [1000, 1000, 1000, 503] [1000, 1000, 1000, 1000] [0, 0, 3, 1000] [0, 0, 0, 0] [0, 0, 0, 0] [0, 0, 0, 0] [0, 0, 0, 0] [0, 1, 0, 0] [1000, 1000, 1000, 0] [1000, 1000, 1000, 1000] [899, 113, 1000, 1000] [1000, 1000, 1000, 1] [1000, 1000, 1000, 0]

Or

We have done the following few tests in verifying the ODDR:

• Test 1, to reproduce the "double-window" on a satellite Kasli with two Urukuls

• Test 2, to swap the two Urukuls with their cables to each other's slots on Kasli

• Test 3, to swap back and flash the patched gateware with ODDR

• Test 4, to flash the original gateware back

The result is visualised in following figure, the bright yellow area is where SYNC_SMP_ERR asserts. All the scan are done in Sync validation delay[3:0] tap = 0.

After adding the ODDR, the window of the SYNC_SMP_ERR become clear and wide. It is not as wide as the DDS sourced SYNC_IN. The DDS results SYNC_SMP_ERR in 2 taps in general, whereas FPGA-ODDR sourced results SYNC_SMP_ERR in 3 taps occasionally 4. But the "double window" is avoid which allows "eye" detection algorithm to find the optimised tap. We will make a pull request to artiq soon.

[jordens]

Interesting. Let's understand this.

Oddr with both phases hooked up to the ttl frequency generator output? Which mode and which kind of pipelining? I don't understand yet how this helps. Assuming in both cases the register gets packed into the output buffer I don't get why it would make a difference. But if you accidentally changed it to negative clockedge then the ground bounce might be smaller... Does an oserdes give even better results?

[WeiDaZhang]

Oddr with both phases hooked up to the ttl frequency generator output? Which mode and which kind of pipelining?

Yes, and SAME_EDGE.

I don't understand yet how this helps. Assuming in both cases the register gets packed into the output buffer I don't get why it would make a difference.

AFAIK, rtio.phy.ttl_simple.ClockGen didn't pack the register into "I/O Tile" before adding the ODDR. If it does the two cases are then implemented identically, and would not make any difference.

Does an oserdes give even better results?

OSERDES and OLOGIC (where the register sits) are alternative to each other, aren't they? Didn't try it. It can't be better than DDS-sourced SYNC_IN really. From DDS we got a window of 10, and 9 from ODDR. I think there isn't much gap there.

[jordens]

Ok. SAME_EDGE would still leave it toggling on the rising edge. We generally expect the toolchain to pack these registers. I'd be surprised if it didn't here. If it didn't we should find out and check all similar cases (e.g. TTL I/O or SPI clocks) where we assume packing. I'm also confused why this would not be visible in my measurements of the jitter or Pawel's. Yes. OLOGIC and OSERDES are mutually exclusive. My assumption is that ORSERDES works with faster bitrates and thus has even better jitter properties.

[WeiDaZhang]

If it didn't we should find out and check all similar cases (e.g. TTL I/O or SPI clocks) where we assume packing.

I'll try to have a look.

My assumption is that ORSERDES works with faster bitrates and thus has even better jitter properties.

Yes, true, make sense.

[hartytp]

We generally expect the toolchain to pack these registers. I'd be surprised if it didn't here

@WeiDaZhang did you check the floorplan?

Can you also fix this by setting IOB=TRUE on the final FF in the clk gen?

[WeiDaZhang]

@WeiDaZhang did you check the floorplan?

That's what I was about to do.

Can you also fix this by setting IOB=TRUE on the final FF in the clk gen?

I frankly don't know how to do it in migen. I imagine it would be something like Instance("FD", i_C=ClockSignal("rio_phy"), i_D=blablalba, o_Q=blablabla, attr={("IOB", "TRUE")}), But my understanding is that the two ways are identical. The benefit might be IOB=TRUE should compatible with all Xilinx.

[hartytp]

This appears to be FPGA jitter, not a hardware issue with Urukul.