Estimate Spectral weights

[SpheMakh]

Could we add an option to calculate per-channel weights from the noise estimates. Then write them out to a file, or (preferably) a column in the MS. This would be useful for MSs which don't have a WEIGHT_SPECTRUM column. Even CubiCal could have an option to use these weights (something like 'weight-estimate-spectral-weights = True')

[o-smirnov]

Yeah, given that some MSs still shockingly lack a WEIGHT_SPECTRUM column, this would be quite a handy thing to have.

I would provide two options: --weight-apply-esw True|False and --out-weight-column COLNAME. The first one would cause the estimated spectral weights (ESW) to be multiplied into whatever weights are already supplied (if any), e.g. the WEIGHT column. The second option would cause the resulting weights to be written out to an MS column.

The latter mechanism (saving weights) is handy to have anyway, since we want to absorb @ebonnassieux's weighting scheme at some point. Come to think of it, we should probably spit out weights multiplied by the diagonal of J^H.J anyway: this corresponds to the variance of the gain estimate, and you do want to downweigh time/freq cells with bad solutions.

[ebonnassieux]

Yeah, that diagonal should correspond exactly to the sensitivity-optimal weights. I talked to Jon about this while I was in CT; it's worth double-checking or digging further into, but regardless, I think it's relatively sound to say that CohJones assumes it's a good estimate for the Fisher matrix anyway... it would make sense to make some kind of CALIB_WEIGHTS output column a default for any given calibration run.

ETN

On Thursday, October 26, 2017 15:16 CEST, Oleg Smirnov notifications@github.com wrote:

Yeah, given that some MSs still shockingly lack a WEIGHT_SPECTRUM column, this would be quite a handy thing to have.

I would provide two options: --weight-apply-esw True|False and --out-weight-column COLNAME. The first one would cause the estimated spectral weights (ESW) to be multiplied into whatever weights are already supplied (if any), e.g. the WEIGHT column. The second option would cause the resulting weights to be written out to an MS column.

The latter mechanism (saving weights) is handy to have anyway, since we want to absorb @ebonnassieux's weighting scheme at some point. Come to think of it, we should probably spit out weights multiplied by the diagonal of J^H.J anyway: this corresponds to the variance of the gain estimate, and you do want to downweigh time/freq cells with bad solutions.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/ratt-ru/CubiCal/issues/85#issuecomment-339662246

[o-smirnov]

it would make sense to make some kind of CALIB_WEIGHTS output column a default for any given calibration run.

Yes, I like it. So have --out-weight-column=CALIB_WEIGHTS by default, and use that in DDFacet. And ask @aroffringa to implement that in WSCLEAN as well. (Come to think of it, a sensible default behaviour for DDFacet would be to look for CALIB_WEIGHTS first, then WEIGHT_SPECTRUM if not found, then WEIGHT...)

A new MS column is definitely needed. I didn't realize CASA modifies weights in-place: https://casaguides.nrao.edu/index.php/DataWeightsAndCombination#Weights_in_CASA. I mean, what could go wrong?

[ebonnassieux]

That's basically what the IMAGING_WEIGHT output column is in kMS, btw - at least, as far as I know (the per-baseline weight reconstruction formula used to be slightly dodgy, but I think Cyril's fixed it). I'm not sure you'd want DDF to read the weights by default, though...seems like something users ought to be aware of before using, given that the weight estimation isn't necessarily well-conditioned?

As for WEIGHT_SPECTRUM......apart from the usual CASA nonsense, it's got a different shape than what you might want for calibration weights: 1 weight per polarisation, whereas it's more conservative to explicitly have 1 weight for all 4 polarisations when it comes to calibration quality weights. This is the default behaviour for IMAGING_WEIGHT. Might be something to keep in mind in terms of MS footprint and to mitigate the ever-delightful numpy operation dtype "gotcha" ?

ETN

On Thursday, October 26, 2017 15:39 CEST, Oleg Smirnov notifications@github.com wrote:

it would make sense to make some kind of CALIB_WEIGHTS output column a default for any given calibration run.

Yes, I like it. So have --out-weight-column=CALIB_WEIGHTS by default, and use that in DDFacet. And ask Andre to implement that in WSCLEAN as well. (Come to think of it, a sensible default behaviour for DDFacet would be to look for CALIB_WEIGHTS first, then WEIGHT_SPECTRUM if not found, then WEIGHT...)

A new MS column is definitely needed. I didn't realize CASA modifies weights in-place: https://casaguides.nrao.edu/index.php/DataWeightsAndCombination#Weights_in_CASA. I mean, what could go wrong?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/ratt-ru/CubiCal/issues/85#issuecomment-339668932

[o-smirnov]

Actually, there's no defined standard for IMAGING_WEIGHT shape. It's not documented in the MS v2 spec, and never was -- it was just something the old old CASA imaging stack used to write out, but no longer does (and hasn't, for a few years). So at this point it just persists as a kind of meme.

From that point of view, we're entitled to define/redefine it as we see fit.

[ebonnassieux]

Fair enough! I thought Cyril just made his own weight-column, rather than picking up an old standard. I would thus think that taking up the "denser" shape (nbl*nt,nchan) is preferable, unless we expect that per-polarisation calibration will become fashionable again for some reason.

[IanHeywood]

Yeah, given that some MSs still shockingly lack a WEIGHT_SPECTRUM column, this would be quite a handy thing to have.

If you want one, you can have one! The first thing I usually do when using CASA to apply the referenced calibration is remake the MS with a WEIGHT_SPECTRUM column using the mstransform task.

[o-smirnov]

So where does it get the spectral weights in this case? (Can have recipe please?)

[IanHeywood]

I think the WEIGHT_SPECTRUM column initialised to unity (which is obviously reasonable) when added by this task. The VLA can write sensible weights directly to the MS as part of the online calibration, however MeerKAT Measurement Sets (or at least the ones I've made with h5toms.py) don't have WEIGHT_SPECTRUM or SIGMA_SPECTRUM columns. See also ASKAP. Most of CASA's calibration tasks will make use of them if they're there, so I add them at the start. I also add a non-volatile MODEL_DATA column (which in later versions of CASA seems to have gone the way of the old IMAGING_WEIGHTS column). This might be a matter of personal preference though. Here's an example using mstransform:

mstransform(vis="input.ms",
    outputvis="output.ms",
    datacolumn='data',
    realmodelcol=True,
    usewtspectrum=True,
    hanning=False)

You can also add/remove SPWs boundaries with this task, which has come in useful in the past. I think you can also use the initweights task to add the weights columns to a MS in more recent CASA versions. Tasks like statwt can also be useful for tweaking the weights based on their statistics towards the end of a reduction. I've found this useful for e.g. suppressing RFI that is too low to be excised by an autoflagger when making full-band continuum images. As always, your mileage may vary.