WR oscillator Si549 characterisation

[WeiDaZhang]

The thermal character of the DCXO Si549 we use in WR implementation has drawn our attention. It has been noticed the open-loop phase noise of the DCXO changes over different (thermal) environments. More precisely, the phase noise slopes and interception points between slopes changes in the very close-in region 1Hz and sub 1Hz, when the thermal environment changes. These frequency regions are not specified in its datasheet.

Oven tests suggest pure temperature does not make a big difference even in 20C changes. However, changes of airflow rate make obvious differences. Heat sinks attached to the DCXO make close-in noise less relative to the airflow (or thermal environment), presumably because of added thermal mass.

The test PCB is not designed to strengthen the thermal grounding of the DCXO particularly. The result suggests heavy coppering the area and vias are necessary, and potentially heat-sink mounting.

In general, PLL removes the close-in noise of a VCO. However, if the slope is steeper than f^-3, careful design of loop filters is required.

[hartytp]

Thanks @WeiDaZhang!

Adding a few details...

The original observation here was that the DCXO open-loop close-in phase noise on Weida's test FMC was a lot worse than previous measurements -- sufficiently bad that the 3rd-order loop filter couldn't bring the DCXO down to the reference (Wenzel oscillator) noise floor.

This appears to be due to thermal fluctuations -- static changes in the DCXO case temp don't change the noise floor (as one might suspect), but air currents do. Adding a heat sink also improves the noise performance (presumably since it increases the thermal mass).

On Weida's test FMC the thermal grounding of the DCXO wasn't good.

So, the main recommendations from this are:

• use a nice high-order loop filter, with the widest BW that one is able to without degrading the broad-band noise floor too far. This is why we use a 3rd order filter, compared with 2nd for standard wr (we could even go to higher-order filters eventually if necessary)
• be careful about thermal grounding of the DCXO (and also placement w.r.t. other thermal noise sources)
• ideally screen the DCXO from air currents as far as possible

[hartytp]

One other comment: on the KC705, Weida found that an external DFF improved the broad-band noise floor by 6dB (we haven't measured the close-in performance yet).

This suggests that with an external DFF we can use a wider loop bandwidth, since the DDMTD noise is lower. That would make us significantly more robust to close-in noise (the loop-filter is high order so even relatively small increases in BW have a large impact on low-frequency noise rejection).

We have to make a decision about how much we care about noise v complexity in designs like Kasli. But, if we want to get the most robust/lowest-noise performance, we should consider adding the external DFF.

[dnadlinger]

Would a shielding can covering the clock recovery section have the side effect of addressing this too? (Extra heat sinking through vias or via a glued-on gap pad if necessary.)

[hartytp]

Would a shielding can covering the clock recovery section have the side effect of addressing this too? (Extra heat sinking through vias or via a glued-on gap pad if necessary.)

Most likely, yes. Although it's hard to find screening cans that fit over these DCXOs without being excessively bulky. Also, soldered-on cans have the disadvantage of being difficult to remove when rework needs doing, while the clip-on ones can be more bulky.

The external DFF has the advantage that it somewhat decouples the CDR from the FPGA since the phase detector is now external -- although, it's not clear if that actually helps much, since the transceiver CDR could still limit (we're planning to get some data on this relatively soon).

But, yes, anything that increases the DCXO thermal time-constant by a factor of a few makes this a non-issue.

[hartytp]

@gkasprow I can only see the 549CBAC000112ABG available with a MOQ of 50 (https://octopart.com/search?q=%20549CBAC000112ABG). That's kind of annoying. Did it used to be available in smaller batches, or did Creotech buy a batch of 50 for Sayma? For now, should we consider switching to the LVDS part, which is available in singles (https://octopart.com/search?q=549BACB001937ABG%20)?

[gkasprow]

@hartytp They are in Mouser. Octopart sometimes do not show correct stock.

[hartytp]

@gkasprow

1. That mouser link was one of the results on Octopart
2. The mouser link you posts shows a MOQ of 50

[hartytp]

NB the spurs in that measurement are real (a product of the fractional-N PLLs inside the DCXO). However, when the feedback is enabled they get whitened out pretty effectively, giving a cleaner spectrum.

[mattiarizzi]

Hello, I did a lot of phase noise characterization on WR hardware, especially WR Switch. On the following links you can find lot of data about DDMTD & Gigabit transceiver performance. Document [1] is a more elegant document, discussing part of the data that you will find in [2] and [3]. Another suggestion on improving the local oscillator: use silicone resin. It fixed the issue on the WR switch (dow corning rtv 3140), or use enclosed vctcxo like dot050f.

cheers

[1] https://ieeexplore.ieee.org/document/8400550 [2] https://www.ohwr.org/project/wr-low-jitter/wikis/Documents/DDMTD-report [3] https://www.ohwr.org/project/wr-low-jitter/wikis/Documents/Xilinx-Virtex-6-transceivers:-phase-noise-and-stability [4] https://www.ohwr.org/project/white-rabbit/uploads/2fa1a438446fc6c85b4540faecf1017a/ISPCS2016-WRClockCharacteristics.pdf

[hartytp]

Thanks for the links to the lovely work. I’d already seen most of that :)

Do you have any data on performance of the vco before/after potting? That matches up with our experience as well that thermal fluctuations can really kill close in noise. Did you try using a more aggressive loop filter?

[hartytp]

@mattiarizzi Did you ever look at the phase noise of the CDR on an ultra-scale FPGA? I assume the flicker noise will be lower since one can get the recovered clock out of an ECL pin directly, but would be good to have some data.

Have you had a look at our proposed implementation for WR on Kasli/Sayma? Any advice/comments/feedback?

Like you, we've noticed that the temperature stability of the VCO is critical for close-in noise. We're hoping to fix that with a relatively high loop BW and an aggressive (>=3rd order) loop filter. Here, using an external DFF will help a little as the lower noise allows for a wide loop BW. But, if that doesn't work, then we may put some thermal shielding over the VCO for users who care about this. Any other feedback, or tips for the gateware/software implementation very welcome!

[mattiarizzi]

Hello @hartytp , thanks for the feedback! Yes, I tried with a bigger bandwidth, see document [4], Section VI.A VCO Noise Leaking. Witha bandwidth of 200Hz, you avoid the low-frequency phase noise induced by the VCTCXO. This is a value that is fine for the WR Switch but may vary with different designs. This bad behavior is know by VCTCXO manufacturer. When I asked to some tech guy working for one of these manufacturers, he told me that they do phase noise plots with a cup over the vctcxo (ha!). The reason is that the temperature sensor is attached to the metallic enclosure, and the feedback system is designed with the goal of keeping the frequency drift vs fast temperature change as low as possible. This impair Allan deviation as it amplify the thermal noise. Recently (2017), Epson started to seal vctcxo with the so-called "DoubleSeal", specifically to avoid this behavior. Below the picture with the VM53S3 (WR switch vctcxo) potted. http://oi68.tinypic.com/ri5b9w.jpg

cheers

[mattiarizzi]

Did you ever look at the phase noise of the CDR on an ultra-scale FPGA? I assume the flicker noise will be lower since one can get the recovered clock out of an ECL pin directly, but would be good to have some data.

I measured the flicker phase noise of Virtex-6 from the general (LVDS) I/O pins and from the GTX pins, it was the same. They claim ECL but seems ECL-compatible, made with CMOS process, not bipolar/hybrid. In document [1] I measured the flicker noise from the I/O pins of Kintex US, and it is equal/slightly better than in Virtex-6. No data on the CDR, yet.

Have you had a look at our proposed implementation for WR on Kasli/Sayma? Any advice/comments/feedback?

Not yet, I'll have a look after the break cheers

[hartytp]

This bad behavior is know by VCTCXO manufacturer. When I asked to some tech guy working for one of these manufacturers, he told me that they do phase noise plots with a cup over the vctcxo (ha!).

Reminds me of that Jim Williams app note about measuring the noise on the LTC6655.

The reason is that the temperature sensor is attached to the metallic enclosure, and the feedback system is designed with the goal of keeping the frequency drift vs fast temperature change as low as possible. This impair Allan deviation as it amplify the thermal noise. Recently (2017), Epson started to seal vctcxo with the so-called "DoubleSeal", specifically to avoid this behavior.

FWIW, this doesn't seem to be necessary with the Si549. AFAICT the temperature sensitivity is relatively low. In our initial tests with eval boards we didn't see any excess noise due to temperature even with all the boards on a desk in normal office space. The issues started for us on @WeiDaZhang's new test FMC, which has much worse thermal grounding than the eval boards. Some lessons we learn the hard way :)

[hartytp]

Did you think about using the Si549 for the new WR implementation. There are a few reasons I'm really keen on it...

1) For a given price point XOs tend to be much lower noise and higher stability than VCXOs. The architecture we use (similar to the SI5324 and kin) allows us to use an XO as our reference, with a microwave VCO locked to it with a high-bandwidth fractional-N loop. By tuning the loop parameters we can lock the output RF to our reference. This lets us inherit the phase noise/stability from the XO while still getting tenability. The result is a much better close-in phase noise performance for the price range. I suspect that this is also why we don't see such bad temperature effects -- XOs have much lower temp cos than VCOs typically, so the SI549 is better than even relatively expensive VCXOs.

2) As the Si549 can put out any frequency, it decouples the choice of (VC)XO from the output frequency we want. i.e. we don't need a 25MHz VCXO and then a separate PLL to generate our RF.

3) I always worry that the DAC + VCXO solution is potentially very noise sensitive as the tuning voltage is a DC signal. I worry that on something like Kasli even breathing on the grounding could cause nasty phase changes in the recovered clock. It seems much easier to use something like the Si549 where everything is digital and all the hard work is done by Si Labs (the package even has LDOs etc in so it's robust to supply noise, it's screened so quite EMI tolerant, etc). All-digital systems seem quite nice to me!

4) Overall component count is quite low for us. We replace DAC + VCO + synth with only the Si549. Just need 2xSi549 one fanout and a 3V3 LDO. AFAICT that's quite a bit more compact/cheaper than the WR implementations I've seen.

5) Not related to the SI549 but on the topic of our implementation: we use a 3rd order loop filter, which seems to make us more robust to low-frequency noise. Did you try doing that? Cheaper/easier than potting things in epoxy!

6) IIRC the tuning resolution of the Si549 is quite a bit better than the 16-bit DAC and VCO, so lower noise from that.

[hartytp]

I measured the flicker phase noise of Virtex-6 from the general (LVDS) I/O pins and from the GTX pins, it was the same. They claim ECL but seems ECL-compatible, made with CMOS process, not bipolar/hybrid. In document [1] I measured the flicker noise from the I/O pins of Kintex US, and it is equal/slightly better than in Virtex-6. No data on the CDR, yet.

Interesting, so maybe the US MGT clock output pins aren't lower noise than the old-fashioned methods. That would be a shame. Will be good to see some data on this

[WeiDaZhang]

A few tests have been performed on our WR implementation (KC705 + FMC with DCXOs, CLK fanout buffers, and D-FFs). The timing-temperature coefficient is measured. A Sanyo OMT Oven is used for controlling the environment temperature. Its specification suggests the temperature fluctuation is +/-0.3 degree C (with time). The timing error is measured with an RF mixer, low-pass filter, and a 6-1/2 digit Multimeter.

Test performed at 50-55-45 degree C each one for 24 hours continuously. All the components of the DDMTD are implemented on the FPGA, the DCXOs are on the FMC.

Another test performed at 50-45-55 degree C each one for 24 hours continuously. The difference from the above set-up is that the "Sampling" registers in DDMTD are implemented with a pair of standalone D-FF which are outside FPGA, on the FMC.

The results suggest:

• Temperature coefficient is about 1.5ps/K without ext DFF, and
• 1.2ps/K with ext DFF
• with external DFF the WR is less sensitive to temperature, but not really much change.
• The long term stability of the system is limited by the oven's stability, and is currently better than 1ps over hours without careful thermal control.
• These tests are excluding the CDR in the FPGA (REF signal is fed to non-MGT pins of the FPGA), temperature coefficients of the CDR will be measured in future tests.

[hartytp]

That’s pretty cool data! Nice to see you getting well sub-ps stability over long periods even without particularly good thermal control! At that point the stability is starting to get towards the ~100fs stability of or measurement apparatus (and I suspect we can probably hit that floor with decent thermal control).

In practice it looks like the wr pll stability will should not be the limitation in any non-trivial setup (at that level even cable drift can quickly become an issue).

[gkasprow]

That' really interesting data. Worth writing a dedicated article! @WeiDaZhang Did you look at the RF mixer thermal stability?

[hartytp]

That' really interesting data. Worth writing a dedicated article!

Yes, the plan is to write this all up (ideally with data from Kasli v2.0). More on that later.

@WeiDaZhang Did you look at the RF mixer thermal stability?

The configuration here is a Wenzel oscillator as the frequency reference, buffered/split using an ADCLK eval board. The mixer and reference/buffer are outside the oven, but not actively stabilised (@weidazhang correct me if I'm wrong about that).

Weida has some nice baseline data on the mixer stability, but IIRC it's around a couple of hundred fs pk-pk over 24 hours, probably limited as much by cabling as anything else. This baseline drift doesn't change much with temperature stabilisation.

[WeiDaZhang]

The stability analysis of the temperature-controlled data is done in the form of modified Allan Deviation and plotted on sigma-delta graph.

• (Circle Mark) The two None-DUT Instrument curves show the measurement limit (the difference in between is about a factor of 2, we believe it is contributed by cables, fanout, power splitters).

• Other traces are the stability of the WR output clock measured against the reference signal

• (Dagger Mark) The reference signal inputs via MGTREFCLK pins and went through CPLL results the lowest stability

• (Star Mark) The reference signal inputs via non-MGT pins provides better stability, the environment temperature makes small difference

• (Diamond Mark) When the reference signal and the main DCXO are both sampled with external D-FF outside FPGA gives the best stability, and the result is limited by measurement set-up.

• None of the measurement involves CDR

[WeiDaZhang]

@WeiDaZhang Did you look at the RF mixer thermal stability?

The RF mixer and the Wenzel are outside the oven, but the fanout is inside the oven (I was not able to squeeze too many cables through the hole on the top of the oven) We will get the MGT link on and perform the tests again referencing CDR output.

[mattiarizzi]

Hello,

@hartyp

FWIW, this doesn't seem to be necessary with the Si549. AFAICT the temperature sensitivity is relatively low.

Si549 is not temperature compensated, which is a good idea for a slave WR oscillator.

Did you think about using the Si549 for the new WR implementation. There are a few reasons I'm really keen on it...

I can a have a look. I'm just worried about the behaviur of the phase when you change the frequency, on a classic VC(TC)XO it's perfectly smooth, but Si549 has a PLL. I could do a test around October.

Not related to the SI549 but on the topic of our implementation: we use a 3rd order loop filter, which seems to make us more robust to low-frequency noise. Did you try doing that? Cheaper/easier than potting things in epoxy!

Interesting! Do you have a repository with the code? Thanks

[hartytp]

Si549 is not temperature compensated, which is a good idea for a slave WR oscillator.

It's extremely stable -- look at the data we've posted! What's the temp co of the VCXO you're using, and how does it compare to the Si549 anyway?

I can a have a look. I'm just worried about the behaviur of the phase when you change the frequency, on a classic VC(TC)XO it's perfectly smooth, but Si549 has a PLL. I could do a test around October.

It's glitch free and designed to be used in these kinds of PLLs. Again, see all the closed-loop phase noise plots we've posted, which would show any phase glitches up.

Interesting! Do you have a repository with the code? Thanks

Weida has some code he's written for our testing (I'm sure he can send it to you if you ask). M-Labs are writing cleaned up code to integrate into ARTIQ. See here https://github.com/m-labs/artiq/tree/wrpll/artiq/gateware/wrpll

[WeiDaZhang]

Phase noise measured on WR implementations with different reference path are plotted. The Carrier frequency is 125MHz. The reference paths are:

1. MGTREFCLK pins -> QPLL -> TXPLLREFCLK_DIV1 -> TXOUTCLK -> "USRCLK_SOURCE (MMCM)" -> DDMTD Register in FPGA (see ug476 Figure 3-28 on p150)

2. MGTREFCLK pins -> QPLL -> RXOUTCLKPMA -> RXOUTCLK -> "USRCLK_SOURCE (MMCM)" -> DDMTD Register in FPGA (see ug476 Figure 4-23 on p210)

3. CC pins -> DDMTD Register in FPGA

4. DDMTD Register (D-FF) outside FPGA -> SelectIO pins -> following DDMTD logics

The difference between 1), 2) may be contributed by "USRCLK_SOURCE (MMCM)". In 1) the MMCM converts 100M to 125M, but in 2) it is more or less a 125M fanout. The performance of the WR-Receiver (I think it is called Node officially) is generally limited by the C/QPLL and CDR facility in the FPGA. Curve 3) is similar to the performance of the WR-Distributor (Grandmaster) reported by other groups. For the WR-Distributor (Grandmaster), the limited seems to be the input stage. Therefore, with external D-FF curve 4) gives a better result than 3).

[hartytp]

@weidazhang thanks for posting that.

Let me check I understand you:

• the purple curve (4) shows the noise floor of the ECL DFF you're using for the DDMTD. The noise floor there is around -100dBc/Hz. This is the noise floor we could achieve in "grand-master" mode with an external DFF

• the yellow curve shows the phase noise of the internal DFFs used for the DDMTD. This is around 10dB worse than the external DFF. The difference is presumably the LVDS buffers in the FPGA or something like that. This data is roughly the same as the "Grandmaster" data published by CERN

• (1) and (2) (blue and red) are noise plots that should correspond to satellite mode, which is what we mainly care about here. These show that the limitation in the phase noise for the receivers/satellites will be limited by the CDR/transceiver PLLs. Is this consistent with the Cern data?

• I don't understand fully the difference between (1) and (2). Is the difference here that you only have a 100MHz reference for your tests, so in (1) you're using an extra PLL (MMCM?) inside the FPGA to convert that to 125MHz, and that PLL is degrading the phase noise?

• does the red curve correspond to the noise level we expect to measure on Kasli satellites in the final system?

• am I right in thinking that in satellites/receivers you don't see any improvement from the external DFF since the CDR/QPLL dominates the noise?

[hartytp]

So, the final measurements are the phase stability + temp co in satellite mode (i.e. when the QPLL/CDR are used)

[WeiDaZhang]

@hartytp

• the purple curve (4) shows the noise floor of the ECL DFF you're using for the DDMTD. The noise floor there is around -100dBc/Hz. This is the noise floor we could achieve in "grand-master" mode with an external DFF

Yes, ECL D-FF -100dBc/Hz at 125MHz

• the yellow curve shows the phase noise of the internal DFFs used for the DDMTD. This is around 10dB worse than the external DFF. The difference is presumably the LVDS buffers in the FPGA or something like that. This data is roughly the same as the "Grandmaster" data published by CERN

Yes, Fig. 10 in [1], Fig. 8 and Fig. 18 in [2] suggests the additive noise or the DDMTD floor is ~-105dBc/Hz @10MHz, which is similar to our ~-90dBc/Hz @125MHz

• (1) and (2) (blue and red) are noise plots that should correspond to satellite mode, which is what we mainly care about here. These show that the limitation in the phase noise for the receivers/satellites will be limited by the CDR/transceiver PLLs. Is this consistent with the Cern data?

(2) corresponding to satellite mode noise. The CERN results seem not very consistent (probably because I didn't understand fully), in Fig. 18 in [2] it is around -85dBc/Hz, but in Fig. 3.10 in [3], it seems about -105dBc/Hz. I need to carefully read again to tell.

• I don't understand fully the difference between (1) and (2). Is the difference here that you only have a 100MHz reference for your tests, so in (1) you're using an extra PLL (MMCM?) inside the FPGA to convert that to 125MHz, and that PLL is degrading the phase noise?

We did a few tests before, without involving CDR, but just let the reference go through the MGT for simplicity. That is (1). I noticed there is an MMCM (Xilinx wizard adds it automatically) when I was tracing the noise source.

• does the red curve correspond to the noise level we expect to measure on Kasli satellites in the final system?

Yes, more or less. I'll look into the MGT a bit to see which clock to lock gives better result.

• am I right in thinking that in satellites/receivers you don't see any improvement from the external DFF since the CDR/QPLL dominates the noise?

You are right. Not a single dB.

[1] https://www.ohwr.org/project/wr-low-jitter/wikis/Documents/White-Rabbit-Switch-performance-in-Grandmaster-mode [2] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8400550 [3] https://www.ohwr.org/project/wr-low-jitter/wikis/Documents/Xilinx-Virtex-6-transceivers:-phase-noise-and-stability

[hartytp]

Okay, thanks for explaining. If you find low noise ways of doing the clocking, make sure to discuss then with sebastien

[hartytp]

Once you’re done with that, will you measure the long term stability and temp co with the CDR? After that I think we have all the data we need

[WeiDaZhang]

Once you’re done with that, will you measure the long term stability and temp co with the CDR? After that I think we have all the data we need

Sure.

[hartytp]

:+1:

[WeiDaZhang]

Okay, thanks for explaining. If you find low noise ways of doing the clocking, make sure to discuss then with sebastien

Bypassing the "USRCLK_SOURCE (MMCM)" resulting 3dB improvement to the red curve (curve 2).

[hartytp]

okay, so the PLLs are all about the same noise contribution, rule is use as few PLLs as possible?

[WeiDaZhang]

okay, so the PLLs are all about the same noise contribution, rule is use as few PLLs as possible?

[hartytp]

?

[WeiDaZhang]

A "remote" link WR recovered clock has been tested. The phase noise measurement is plotted in the green curve. The "remote" link is set up, though in the same FPGA, with Tx path of the transmitter driven with Wenzel, and the Rx is driven with Main DCXO. The MGT set up the link first, and the WR locks the Main DCXO to the RXOUTCLK without losing the MGT link. The setup is sketched as follows. @hartytp Sorry, I clicked the wrong button earlier ....

[hartytp]

Thanks @weidazhang! Can you summarise how this relates to your previous findings? What are the recommendations about setting up the PLLs? How does this compare to the Cern results?

[WeiDaZhang]

The stability of the "remote" link has been analysed and shown follows. @hartytp I'll summarise them soon.

[hartytp]

@hartytp I'll summarise them soon.

:)

[hartytp]

NB one fun thing down the line might be to play with the new fan controller on Kasli to do temperature feedback based on the FPGA's internal thermometer. If we keep the whole WR loop inside the FPGA (so no external DFF) we can probably take out almost all of the temperature coefficient that way.

[WeiDaZhang]

The temperature stability of the remote link (CDR + DDMTD) has been tested. The result appears to be significantly different (much lower) from the test only involves DDMTD. The temperature coefficiant is around 100fs/^oC ~ 250fs/^oC. A few potential reasons which we are checking:

• CDR and DDMTD cancles each other.

• The temperature coefficients of the QPLL(Tx), CPLL(Rx), CDR and DDMTD, which are in the same FPGA, are correlated.

• Oven faulier / Incorrect measurement procedure etc.

[WeiDaZhang]

A summary of CERN papers/reports vs our measured results:

1. CERN paper [Rizzi 2018] observes noise of two kinds: white phase and flicker phase; (which we agree) it also identifies two sections: DDMTD noise and GTX noise

2. The DDMTD noise they targeted "IOBUFDS" as the main contributor (we also agree) the Helper noise has been given analytically in two references but appears not dominating (this may not be the case, and will be explained later)

3. the IOBUFDS results in -108dBc/Hz white and -100dBc/Hz flicker intercept @ 1Hz and 4e-13s @ tau = 1s (this is roughly the so called "Ultimate Limits")

   • -108 is similar to our internal-DFF result, about -110

   • -100 we can't verify since it is covered by our reference

   • 4e-13s @ 1s is equivalent to 4e-14s @ 10s (on their Virtex-6 and 2e-14s Kintex-7) and our internal-DFF result is 6e-15s @ 10s (our mod Allan measurement take 4s to get a sample). This means our internal-DFF result is about 3 times better than theirs (our measurement setup may contribute some portion of it).

   • Theoretically, flicker phase intercept point equivalent to tau^(-1) on mod Allan, I'll need some calculation to make a detailed comparison

   • Our ext-DFF is about 10 dB better than that IOBUFDS and 6e-16s is measured. It is at this point we are not sure if the helper noise is still neglectable.

4. the stability of the GTX is characterised by different data pattern in [Rizzi 2018], though not much difference is found.

5. the phase noise of the GTX is measured using DDMTD and differential phase noise meter, the result is much better than our result, for a few reasons and some unclear points:

   • Our WR-PLL loop bandwidth is much wider

   • A reason can be directly measuring phase noise of an internal signal in FPGA is not possible.

   • It doesn't seem very much consistent with other results they have got (fig.14 vs fig.15 vs fig.18), though it says p1733 "Since the performance of the DDMTD has been fully characterized..."

6. The stability of the GTX is measured 2 to 4e-14s @ 10s in their fig. 12. For our results involving FPGA Tx, SMA cable and CDR, the stability is between 8 and 9e-15s.

7. The paper then concluded that noise contributions to short-term stability "equally shared between DMDTD and GTX" p1735

   • Our results more or less agree with their statement which is equally shared.

   • this is the stability rather than the noise floor (white), which in our experiment GTX appears dominantly

8. They referenced a few modification "LJD - low jitter daughterboard" and "grandmaster mode switch performance".

   • Interestingly in the second article, a DFF is used at the output of FPGA generated 10MHz to improve the performance.

   • Though the DFF is used in a different way to ours, it improves their phase noise (white) by about 20dB and finally reaches the "Ultimate Limit"

9. Our results without the LJD or ext-DFF modification can get close to the "Ultimate Limit"

   • the phase noise floor (white) part is slightly worse, and

   • the stability (flicker) part is slightly better.

[hartytp]

the phase noise of the GTX is measured using DDMTD and differential phase noise meter, the result is much better than our result, for a few reasons and some unclear points:

You mean the white noise floor, right?

[hartytp]

A reason can be directly measuring phase noise of an internal signal in FPGA is not possible.

You mean they measure the phase noise inside the FPGA, while we measure outside and the buffers etc could add some noise?

[hartytp]

With reference to the stability plot here https://github.com/sinara-hw/meta/issues/39#issuecomment-527547429 can you remind me which curve is which, and what the differences are. e.g. IIRC the worse curves are with the CDR and are presumably limited by the QPLL noise. etc

[hartytp]

Also, IIRC you tried both the CPLL and QPLL. Conclusion was that both add about the same level of noise, so it's good to minimize the number of PLLs, but no particular recommendation for optimizing the noise was found.

[hartytp]

Interestingly in the second article, a DFF is used at the output of FPGA generated 10MHz to improve the performance.

Can you clarify what this means? IIRC you had a nice BD of this.

[WeiDaZhang]

A reason can be directly measuring phase noise of an internal signal in FPGA is not possible.

You mean they measure the phase noise inside the FPGA, while we measure outside and the buffers etc could add some noise?

[Rizzi 2018] measures phase noise in two ways - meter and DDMTD - on three different setups. There is one measured inside FPGA with DDMTD. This one suggests flicker noise of the Rx CLK by WR should intercept 1Hz at -95dBc. The other two (fig 14 and 18) are on WRPLL output with the meter. The former shows the flicker intercept 1Hz at about -100dBc without obvious floor observed. The latter shows a FLOOR at ~ -80dBc without an obvious flicker. The latter also shows by adding the LJD the floor reduced to ~-95dBc without an obvious flicker.

If comparing the fig 14 and 18 without LJD, assume there is a flicker intercept 1Hz at -75dBc but the slope is distorted by the loop filter, there is a 20dB difference. I didn't find a clear explanation for this apart from the fiber distance is increased from 3m to 10km. Note the grandmaster output also downgraded by more than 20dB which can not be contributed by the fiber.

These differences are recovered by using LJD. So basically at the end, the result with LJD gets close to fig 14, and similar to our result.

[WeiDaZhang]

With reference to the stability plot here #39 (comment) can you remind me which curve is which, and what the differences are. e.g. IIRC the worse curves are with the CDR and are presumably limited by the QPLL noise. etc

The worst one is not actually CDR, but Wenzel -> MGTREF -> QPLL -> TxCLKOUT -> DDMTD -> MainDCXO. The second worst is with CDR. As shown in BD, it involves QPLL -> Tx -> 2.5Gbps on 3-inch SMA cables pair -> Rx -> CDR ->RxCLKOUT -> DDMTD -> MainDCXO. Apart from the Rx and Tx are physically located in the same MGT channel in one FPGA, this is very close to the real use case.