MASS can sometimes give terrible arrival-time bias when we use "new filters"

[joefowler]

Original report by Joseph Fowler (Bitbucket: joe_fowler, ).

---

@jojo978 pointed out that certain data sets give a terrible result (energy resolution) when we use new-style filters but not with old-style filters. I have identified ch 25 from 20170407_233407 and ch 31 from 20170408_172919 at the APS last weekend as examples of this problem.

Specifically, a scatter plot of p_filt_value_dc vs p_filt_phase shows emission lines or other narrow features having a strong non-zero slope. Although data.phase_correct() can help improve this slope, it should not have to! I need to get to the bottom of this problem and make it go away.

@oneilg might be interested.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

Here is the plot I mentioned, with all 4 channels shown. Only channel 31 (cyan) has the steep slope that I'm talking about, though ch 5 (purple) also shows a hint of it.

The spectrum shown here consists mainly of 920 eV elastic scattering of monochromatic beam off a gold sample, though some copper fluorescence (~933 eV) is also visible and supplies the thinner, higher-energy line. Although that specific run (20170408_172919) also contains some 950 eV elastic scattering, I have cut that data and not plotted it.

Each color shows 920 eV at a different y-value on the plot because these are raw, uncalibrated pulse heights.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

The source of this problem is the way that the (good) data in channel 31 happen to have a median energy that is nearly on the 525 eV oxygen K line and nearly on the 920 eV beam. We study its first 1000 good pulses, and these happen to be roughly 45% less than 550 eV and 45% greater than 800 eV. Thus the median sample value is not a robust estimator, but is a terrible estimator of what the "representative value" is! See below:

The y-axis is the value of sample 688 relative to the peak value, and the lower group is the 525 eV data; the upper group is the 900+ eV. Notice how the median of each "arrival time bin" (= color group) jumps between the 525 cluster and the 900 cluster.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

Note that simply cutting the pulses < 700 eV works around the above problem, and turns the offending channel into a 1.5 eV one at 920 eV and 1.7 at 950 eV, which is pretty much in line with all the others.

But the problem now is to fix, rather than just work around the problem shown in the rainbow plot.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

Slight reordering of steps in compute_newfilter.

Also eliminate some unused steps. But see issue #93: there are still problems with it.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

What I've learned is that we can use entropy minimization to find the slope that "best explains" the correlation such as we see in the plot above. However, it is slower and at least 10x as noisy as when we use the finite-difference estimate: finite_diff_dpdt = avgpulse[1:]-avgpulse[:-1]. Here is a plot that compares the two:

These match very well on all scales, though the entropy one is noisier. Idea: use entropy minimization on the rising edge, and the finite difference on the remainder.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

If entropy minimization is slower and noisier, then why would we use it ever?

The answer is that the rising edge is special. In that part of the pulse, we do not expect the pulses of various arrival times to look like mere shifts of the average pulse. Instead, effects like SQUID nonlinearity and the feedback loop's finite bandwidth modify the samples that we observe. That means that the finite difference of the average pulse is not a great representation of the first-order term in the pulse shape's Taylor series as it's expanded in arrival time.

So the idea is to use entropy minimization on the rising edge where the assumptions behind the finite-difference method break down, but use the finite-difference for the falling part of the pulse.

[joefowler]

Original comment by Joseph Fowler (Bitbucket: joe_fowler, ).

---

Fix regression test targets.

Fixes #93. Or at least, I certainly hope so.