Improve EFM PLL and filtering, especially for CLV and CD audio disks

[happycube]

The current filters work well outside the low-MTF zone, but issues like #220 can occur with CLV disks. I was unable to replicate, much less improve on, the base EFM filter in rev5 development, but hopefully with some effort can do so in rev6.

In addition, ld-ldstoefm functionality should be brought into ld-decode and/or the integration refined.

[Gamnn]

Here are some CLV snips. Clip 6 breaks twice 1 frame apart.

tomoe-broke1.lds.zip tomoe-broke2.lds.zip tomoe-broke3.lds.zip tomoe-broke4.lds.zip tomoe-broke5.lds.zip tomoe-broke6.lds.zip tomoe-broke7.lds.zip

[atsampson]

Re integrating ld-ldstoefm into ld-decode, I had a go at porting pll.cpp to Python. It works, but even with help from numba it's not very fast in its current form.

[atsampson]

To experiment with filter parameters, I've written evaluate-efm-filter, which runs ld-ldstoefm and ld-process-efm automatically against a bunch of short samples, and collects the statistics from ld-process-efm. You can tweak it to just show you the stats for a range of filter values, or to optimise them iteratively using a GA.

Caveat: currently it's using ld-decode's existing filter structure and orders - Butterworth HPF, elliptic LPF applied twice, raised-cosine shaping - but applied directly with lfilter rather than converting to FFT form, so it won't exactly match ld-decode's output.

Here's some sample output: evolving from completely random filter parameters, and improving an existing set of filters.

The executive summary so far is that raising the LPF cutoff to about 2.15 MHz, with the HPF cutoff at about 0.5 MHz, gives a measurable improvement across the less-successful samples - but the filter shape probably needs changing. The two toughest samples I've got are MLV49153 Squibnocket and CC1390L Spinal Tap, and I can't find a single set of parameters with the current filter structure that works well for both. Maybe there's a different filter shape that would work better, or maybe it'd be worth thinking about adaptive filtering based on the expected spectrum of the EFM signal - which wouldn't be too hard given the FFT approach.

(I did also try evolving IIR coefficients or lists of filter zeros/poles directly with a GA, but this doesn't work very well - there are a couple of interesting papers on this but they're usually solving much simpler problems with cheaper evaluation!)

[simoninns]

I'm not convinced that this is only a filter issue; it looks like the PLL implementation requires some improvement too. I will update the issue title to reflect this

[atsampson]

I wrote a tool for exploring EFM filter shapes by hand - edit-efm-filter:

The FFT filter implementation is in efmfilter.py, largely inspired by the equaliser design from WSJT-X. efm-filter will apply the same filter to RF samples for decoding. (It's not very fast; it would be better to decimate before the FFT.)

The executive summary is that it's very sensitive to the filter shape, both in amplitude and phase, particularly below 1MHz. The default parameters in there give a substantial improvement for Spinal Tap and Squibnocket, although the PLL still struggles a bit in the first few minutes of the discs; you can see on the histogram that T3/T4 aren't very well separated on some samples.

[happycube]

Thanks to @atsampson's great work, this is coming along nicely but there are still spurious E32 errors that should be fixed before closing.

[happycube]

Closing this since things work most of the time - please open issues with specific data when things don't work on a particular disk.