PREG0 solutions made with gain_cal_flux solint

[o-smirnov]

As far as I can see, this is the case here: https://github.com/ska-sa/meerkathi/blob/master/meerkathi/workers/cross_cal_worker.py#L309. The default is 'inf'.

Surely this is wrong, or do I misunderstand the point of the PREG0 solutions? I thought it's meant to align the time-variable phases inside the bandpass scans, prior to solving for bandpass. If so, it shouldn't be an 'inf' interval, it should be something short. And in any case it should not be the same parameter as ['gain_cal_flux']['solint'], but rather an independent setting? (You probably DO want to solve for a single gain per scan... you certainly do want to solve on shorter intervals inside the bandpass scans!)

As things stand, PREG0 has no effect, since a single gain solution per scan may as well be absorbed by the bandpass.

[IanHeywood]

I think solint for PREG0 can be as short as int, and should definitely be distinct from the solint on the bandpass task. PREG0 should also be applied whenever the bandpass is applied, otherwise the bandpass solutions won't be correct.

[paoloserra]

I think this can be used in a few different ways.

I usually solve for a fully time-independent bandpass (bp_cal: solint: inf, bp_cal: combine: scan), in which case the PREG0 solutions (infinite solint, no combine) align the time-variable phases among all bandpass scans, rather than within individual scans. To me this makes sense because when calculating a time-independent bandpass the scan-to-scan phase variations are the main source of troubles. Phase variations within a scan are less severe. This is however not the default in the schema file because by default "combine" is empty. I agree with you that the current default is not very useful.

[IanHeywood]

That sounds like the same philosophy with different time scales:

Phases per int --> Bandpass per scan

vs

Phases per scan --> Bandpass for entire observation

[o-smirnov]

Yeah I can see the sense in both, but the current pipeline default seems to straddle no-man's land. And in any case the PREG0 solint setting needs to be decoupled into its own setting.

Here's another question then. We solve for PREB0 with K0, for PREG0 using PREB0 and K0... then for B0 using PREG0 and K0... but then when we apply the bandpass solutions (to e.g. the bandpass scan itself), we apply K0, B0 and F0. Surely PREG0 needs to be in there too, as per @IanHeywood's remark above? (Or is it absorbed into F0 in a way I don't understand?)

[paoloserra]

That sounds like the same philosophy with different time scales:

Agreed -- that's what I meant to say with "it can be used in a few different ways". Both should be possible in MeerKATHI. Maybe we should just make it clearer to users that the solint for PREG0 is the same as that for the flux calibration. We should probably also extend this behaviour to "combine" in PREG0. Alternatively, we could duplicate both parameters within the "remove_ph_time_var" section of the config file. Any opinion on that?

[paoloserra]

Here's another question then. We solve for PREB0 with K0, for PREG0 using PREB0 and K0... then for B0 using PREG0 and K0... but then when we apply the bandpass solutions (to e.g. the bandpass scan itself), we apply K0, B0 and F0. Surely PREG0 needs to be in there too, as per @IanHeywood's remark above? (Or is it absorbed into F0 in a way I don't understand?)

Both PREB0 and B0 are normalised, so the only effect of applying PREG0 when calculating B0 is to remove that phase variations. PREG0 is ignored when calculating G0 (from which we bootstrap the flux scale and create F0) so both G0 and F0 include the flux and phase calibration. So I think it's correct that PREG0 is ignored once B0 is in place.

[paoloserra]

Yeah I can see the sense in both, but the current pipeline default seems to straddle no-man's land. And in any case the PREG0 solint setting needs to be decoupled into its own setting.

Sounds like your answer to my question on duplicating parameters is yes. I kind of agree.

[o-smirnov]

That sounds like the same philosophy with different time scales:

I can also see the merit in both... I can see how you could, in principle, get a better bandpass solution if you take out the short term phase variations per interval using PREG0 (think outer ring antennas + bad ionosphere) -- but still assume a global bandpass solution for B0...

both G0 and F0 include the flux and phase calibration. So I think it's correct that PREG0 is ignored once B0 is in place.

Right, since we apply B0 (but not PREG0) when solving for G0 and F0, F0 will absorb the time-variable phase component (which is what PREG0 was, when solving for B0), so it's not necessary. OK, I can see that.

[IanHeywood]

Both PREB0 and B0 are normalised

It's worth having a look at whether this works for you or not. I think the normalisation (like many CASA tasks) assume the data arrive in low fractional bandwidth SPWs, and ends up doing funny things in the 60% fractional bandwidth case. I ditched normalisation a long time ago.

It might be OK for "I know where the hydrogen is" type projects, but I know Krishna for example has some strong things to say about the treachery of solnorm for full-band RM synthesis work.

[o-smirnov]

Possibly apropos to @IanHeywood's distrust of solnorm, this is what the spectra of my gain calibrator look like, after the default calibration strategy has been applied.

All is not well! Yes I have a wedding to attend over the weekend, but I'm not ready for that many bowties yet...

[SpheMakh]

I'm with ya'll on not trusting solnorm

[IanHeywood]

Nothing good ever happens when there's a bowtie involved.

[svw26]

cough Except when you get excellent spectral coverage at low-frequencies... #bowtiesarecool

On Mon, Oct 14, 2019 at 9:25 PM IanHeywood notifications@github.com wrote:

Nothing good ever happens when there's a bowtie involved.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/622?email_source=notifications&email_token=AD7NU7DL2X4WPKYAG5CAN3LQOS2SVA5CNFSM4I7MFLVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBF5NVQ#issuecomment-541841110, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7NU7CGEZ53N5KXNA3TH6DQOS2SVANCNFSM4I7MFLVA .

[ratt-priv-ci]

As I mentioned before you should try rate calibrate (over the entire bandwidth) your calibrators. I've seen this before with fleetingpol when I only computed a single delay solution over the span of a long observation period.

On Mon, 14 Oct 2019, 19:07 Oleg Smirnov, notifications@github.com wrote:

Possibly apropos to @IanHeywood https://github.com/IanHeywood's distrust of solnorm, this is what the spectra of my gain calibrator look like, after the default calibration strategy has been applied.

[image: Screenshot from 2019-10-14 18-59-09] https://user-images.githubusercontent.com/6470079/66769485-8c30a400-eeb5-11e9-9768-524cbb859e02.png

All is not well! Yes I have a wedding to attend over the weekend, but I'm not ready for that many bowties yet...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/622?email_source=notifications&email_token=AEIVPJTOXCLTDR4P5RBN363QOSRNHA5CNFSM4I7MFLVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBFTK6A#issuecomment-541799800, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIVPJXJQ4TWBA63SN64HCDQOSRNHANCNFSM4I7MFLVA .

On Mon, 14 Oct 2019, 19:07 Oleg Smirnov, notifications@github.com wrote:

Possibly apropos to @IanHeywood https://github.com/IanHeywood's distrust of solnorm, this is what the spectra of my gain calibrator look like, after the default calibration strategy has been applied.

[image: Screenshot from 2019-10-14 18-59-09] https://user-images.githubusercontent.com/6470079/66769485-8c30a400-eeb5-11e9-9768-524cbb859e02.png

All is not well! Yes I have a wedding to attend over the weekend, but I'm not ready for that many bowties yet...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/622?email_source=notifications&email_token=AEIVPJTOXCLTDR4P5RBN363QOSRNHA5CNFSM4I7MFLVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBFTK6A#issuecomment-541799800, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIVPJXJQ4TWBA63SN64HCDQOSRNHANCNFSM4I7MFLVA .

[o-smirnov]

@paoloserra, thanks. Check out this discussion as well: https://github.com/o-smirnov/old-devils-meerkat/issues/1#issuecomment-542744165

As we just figured out, the bowties above are due to lack of delay calibration on the secondary...

[paoloserra]

Thanks for sharing.

Maybe it's me being lazy, but "delays on the secondary" sounds like something the observatory should fix.

[o-smirnov]

Yes and no... They can and should fix some/most it, but we'll still have some time-variable tropospheric delay on the long baselines, right?

---

Sent from my phone. Quality of spelling inversely proportional to finger size.

On Wed, 16 Oct 2019, 23:22 paoloserra, notifications@github.com wrote:

Thanks for sharing.

Maybe it's me being lazy, but "delays on the secondary" sounds like something the observatory should fix.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/622?email_source=notifications&email_token=ABRLTPYYBD2GLZ3WP2AIMZ3QO6AZZA5CNFSM4I7MFLVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBN7IZA#issuecomment-542897252, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTP4WRBKAIAQDCUV4D33QO6AZZANCNFSM4I7MFLVA .

[SpheMakh]

Now that we have a CWL implementation of stimela, we have to re-write the workers anyway so it is a good time to re-think a few things.

Here is a sample 1GC recipe with the new interface

https://gist.github.com/SpheMakh/ef08ad1e89497fafd1b15a2413d7c5ba

[ratt-priv-ci]

No its atmospheric in nature. You can clearly see this on the long spacings if you track a calibrator over a long timespan. This is why I do rate selfcalibration when I image

On Wed, 16 Oct 2019, 23:22 paoloserra, notifications@github.com wrote:

Thanks for sharing.

Maybe it's me being lazy, but "delays on the secondary" sounds like something the observatory should fix.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/622?email_source=notifications&email_token=AEIVPJUCPBJBYD4YU7KAPG3QO6AZZA5CNFSM4I7MFLVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBN7IZA#issuecomment-542897252, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIVPJVUOIQSANLTY6M5BD3QO6AZZANCNFSM4I7MFLVA .

[o-smirnov]

No its atmospheric in nature.

Yep, and here's the plot to prove it. These are delay solutions from my secondary, as a function of antenna number:

[paoloserra]

Cool, so I think the badly needed MeerKAT template config file to be posted at https://github.com/ska-sa/meerkathi/tree/master/meerkathi/sample_configurations should have the crosscal strategy required to deal with this.

[paoloserra]

https://github.com/ska-sa/meerkathi/pull/629