Smoother of aocalibrate solutions is broken

[tjgalvin]

Something is really broken with how the solutions are smoothed. it is looking very off. Amplitudes are changing of the gains are changing by 10x. Very very odd stuff. Will make a note and track this down.

[tjgalvin]

Looking into this the past couple of days there are two compounding problems. These were tests performed against 51998 (bandpass SBID), and 51997 and 52087 (3C286 at different rotator angles).

Throughout data were calibrated using calibrate and were made against data of 1934-638. No field-model was used. Basic ionosphere corrections were made by assuming that there is no internal RM from 3C286 and running 1D RM-synthesis. Any RM is attributed to the ionosphere and removed - credit to @AlecThomson developed this cross-check

The first figure here is the IQU spectrum of 3C286 using solutions with only a light set of flagging. This agrees nicely with the Perley and Butler model in all respects.

The following is when the bandpass solutions are modified by (1) dividing the solutions by a phasor of a nominated reference antenna, and (2) the solutions are smoothed using a median filter + Savgol filter

The Stokes I spectrum is noisier and there are some significant ripples in the polarized intensity. These do shift around when the rotator is at a different angle. I convinced myself from earlier toying around with the primative phase outlier flagger that a lot of the structure across phase is common between all antenna, so dividing out by a reference antenna would be useful. It did indeed seem to help produce a smoothed spectrum that better tracked the noisier raw solutions.

When dividing by just the phasor of the reference antenna

It is clear to see where the smoother is getting the structure from.

Applying just the smoother also shows that there are artefacts introduced:

My initial feeling is that the Savgol filter it perhaps over-fitting to the data. In a nutshell it is applying a boxcar type operation, and to the datapoints within the boxcar it is fitting a polynominal of a particular order. At the moment this is set to window_size=16 and polynomial_order=4. I got those by eye-balling how well the smoother was dealing with solutions in a notebook, and then I set and forgot. Now that there is a spectrum we can properly test against this can be experimented with to see if it is really the source of the issue.

It might be worth just see how well the median filter by itself does. In the current implementation a median filter is first applied before attempting the Savgol filter.

For reference, the function that is constructing and dividing by the reference antenna phasor is: https://github.com/tjgalvin/flint/blob/a1b68edde497e881a530ebfe67a7ae3a87597370/flint/bptools/smoother.py#L8

[tjgalvin]

I have been playing around with different combinations of smoothing kernels, their window sizes and which elements of the Jones to operate over (Xx and Yy only, or Xx, Xy, Yx, YY). The net results are not that great. Improvements in the Stokes I spectrum are minimal, if any, and there seems to be considerable negative features introduced into the PI, Q and U spectra.

Its not clear to me whether this is a consequence of the procedure itself, smoothing over genuine structure, rotation of visibilities etc.

Since there does not seem to be any real improvement I think dfor the immediate future smoothing should just be disabled. There will have to be a conversation or two to try to understand this better.

[tjgalvin]

Ok. Ultimately. Math is the problem, and is the solution. Smoothing of Jones is not straight forward: arxiv.org/abs/1209.5492. The initial attempt to phase reference the solutions was close, but the off-diagonal references were around the wrong way. This appears to be the proper way:

    g_x_prime = g_x / ref_g_x
    g_xy_prime = g_xy / ref_g_y # Leakage of y into x, so reference the y
    g_yx_prime = g_yx / ref_g_x # Leakage of x into y, so reference the x
    g_y_prime = g_y / ref_g_y

The method that produced the plots above had the reference antenna for g_xy_prime and g_yx_prime around the wrong way. This approach is only valid for cases where the xy-phase is 0, which should be try via the ODC.

With the corrected division the spectrum in its glory.

and some early smoothing results that have not blown up with error and clear wobbles

[tjgalvin]

Fixed in #66 . A huge thanks to @danmitch for the assist and truth bombs