Micro Adjust to compensate for XTAL inaccuracies

[mungewell]

I have been looking at how we can bias the XTAL, to compensate for any inaccuracies between it and the 'other' devices in use.

We have a scheme where the PIO clock divider is toggled up/down by 1/256 fraction at a defined duty factor. This means that the clocks are slight faster/slower for a fraction of the time. With two Picos connected together we can see values of +0.1, -0.1 and -0.5 affect the synchronization (error bar) 10 minutes after Jamming.

The question is how do we measure the inaccuracy to figure out the correction....

[mungewell]

For now you can set a hardcoded value here: https://github.com/mungewell/pico-timecode/blob/main/pico_timecode.py#L636

This is the duty factor for the compensation, which is plus/minus one fractional clock. It's likely different duty factor will be needed for different framerates.

[mungewell]

In a long term test, confirmed that the XTAL on device can be biased one way or the other. This is using a +/-1 fraction on the PIO clock divider for a duty cycle of ~60s period.

Between two devices (A -> B) the error bar value was recorded every minute, with no adjustments the devices were out-of-sync by 1 frame after 5hr 48mins, with A being ahead. In the following chart 'Blue' is uncorrected, 'Red' is the clock on B slowed further (increasing the rate of divergence) and 'Yellow' is the clock on B made faster (keeping them in sync for longer).

In this case I don't know what the correct rate should be, only the comparison between the two. I'd suspect that both devices are off, just don't know by how much

[mungewell]

With further adjustment I was able to limit the XTAL drift to less than 3500us in 12hrs!

$ head A_to_B_30fps_super_fast.txt 
00:02:00:05 77.77777
00:03:00:02 73.77777
00:04:00:05 37.77777
00:05:00:02 94.77777
00:06:00:05 81.77777
00:07:00:06 83.77777
00:08:00:02 58.77777
00:09:00:05 103.7778
00:10:00:04 103.7778
00:11:00:01 90.77777

$ tail A_to_B_30fps_super_fast.txt 
11:52:00:06 3322.778
11:53:00:04 3309.778
11:54:00:05 3289.778
11:55:00:03 3320.778
11:56:00:05 3322.778
11:57:00:03 3310.778
11:58:00:05 3325.778
11:59:00:02 3291.109
12:00:00:04 3315.778
12:01:00:01 3355.778

It also occurred to me is that we might be able to actively monitor the delta/error bar when a source is connected, and change the adjustment to minimize the drift between two device. The second would then 'learn' the first's XTAL rate.

[mungewell]

I was able to use 'Simple PID' to track the time delta between RX and TX (blue trace) and adjust the PIO dividers (red trace) so that the local clock rate exactly matched that of the external reference. This is set to happen automatically after a Jam, until the RX monitor screen is closed - at which point the adjustment is locked/held.

On my test bench the Pico tracked the signal from my SG-4 Blackburst generator (with LTC produced by a Sync-IO).

The RX signal was tracked up to 7620s, after this the delta remains stable for a while. I think that the 'droop' is actually temperature related. The test equipment was near a west facing window, and may have been heated slightly by the sun... after a while the drift changes direction.

02:04:00:02 18.50002 2.368298 (0.01850002, 2.349541, 0.0002566667)
02:05:00:03 7.400015 2.361011 (0.007400016, 2.353981, -0.00037)
02:06:00:04 -4.699985 2.346058 (-0.004699985, 2.351161, -0.0004033334)
02:07:00:03 -1.099985 2.346058
02:08:00:04 1.700015 2.346058

The 'delta' then continues drifts, and is a whole frame late after 15 1/2hrs. Not bad indeed!

17:39:00:04 -253.9978 2.346058
17:40:00:02 -148.2979 2.346058
17:41:00:05 -36.69786 2.346058
17:42:00:04 106.3021 2.346058
17:43:00:02 170.6021 2.346058
17:44:00:02 167.3021 2.346058

I think the next improvement will be to use the hardware timer IRQ to modulate the PIO dividers, rather than doing this in pico_timecode_thread().

[mungewell]

After a long time away from the project, I'm back thanks to a fix in mircoPython everything seems to be working now.

I made a quick video on the clock accuracy, with compensation code working a group of 4 stock Pico's where within 4 frames 18hrs after Jamming. This is good, but probably not good enough... on with the XTAL replacement idea.

https://www.youtube.com/watch?v=ilfQTCmmGfg

[mungewell]

In my latest changes to the code, it attempts to use a PID loop (just PI at the moment) to change the 'adjustment' value.

This 'adjustment' is fractional, the whole units are the steps of the fractional dividers and the fractional part is a time based modulation between adjacent values... meaning that X.5 is 50% duty factor between 1+0/128 and 1+1/128.

System is both noisy and slow (as it's display on OLED - which we could turn off). By default I'm modulating on a period of 10s, this might have to be extended to get more resolution in the modulation.

I had used a rolling average of the last N values of the micro-adjust value, but it seems that this is NOT usable - even the last 240 values still has not settled in to a value. I need to rethink that, anyone have suggestions?

Notes on above plot:

After ~4000 samples, I stopped calibration and the micro adjust locks into last (averaged) value. But just prior to this there are 'wild swings' so really this locked value could be anything - slightly high or low means time will drift, but in this dataset we were drifted by 4bits after ~5hr (so actually pretty good).

[mungewell]

I've improved the calibrations process, and added some scripts that collect/plot data so we can see what's happening.

It is not the first time that I see the unit selecting a wildly different value for it's 'adjust' value...

This may be related to the sync not being spot on when the unit starts with jam (as apposed to the user starting it).

[mungewell]

I think I found the bug, that allowed the adjust value to be carried over from previous run - which if stopped during the high phase would mean a increase of one.

Latest calibration runs are pretty closely grouped, note that a difference of '1' is actually 1/256th of a frequency divider so these are actually a pretty small variance. If all these are averaged I guess that would be the true value.

[mungewell]

From the last run of that plot I let the units run for 15hours, and they were still well within 1 frame accuracy...

Noted that this is NOT compensated for temperature.

Mechanism for 'micro-adjust' seems to be working well. Closing.