TODO: 7-17 Nov 2017

[SpheMakh]

A list of things to do this week

Add to MSUtils

• To estimate thermal noise for an MS(s). Use what @paoloserra has done
• A tool to estimate spectral weights
  • Mode 1: Use spec. SEFD values
  • Mode 2: Use estimates of visibility noise. In this case, the difference of adjacent channels gives a good estimate of the noise, when using a calibrator field after 1GC. May also get good results by the cross correlations (XY, YX) , or maybe even STOKES V
• Create an initialise columns related to spectral weights; SIGMA_SPECTRUM and WEIGHT_SPECTRUM. Use existing addcol function
• Tool to plot antenna gains as well the plots in the inspect_data worker

Add to Politsiyakat

• Revamped flagging techniques for pre and post 1GC flagging. @bennahugo can explain the details Flag based on [uncalibrated/calibrated] autocorrelations and flag based on [calibrated] calibrator phase. This should catch closure errors and cases where antenna temperature chances drastically compared to itself or the rest of the array

Add to PyPi

• Sunblocker (also add to stimela)
• RFI Masker
• MSUtils

Add to MeerKATHI

• New flagging strategies (aoflagger)
• CubiCal
• Initialise and populate spectral weight columns
• Politsiyakat flagging and diagnostic plots
• Thermal noise estimator
• Unit tests (every commit must pass test before going to master)
• Block direct commits to master branch (only commit master after successful unit test)
  • @bennahugo will setup a jenkins cron job to do the test. (PR BUILDER JOB)
  • The initial IC5264 observations should be a good test dataset (DONE SMALL 4k 2hr dataset... not great continuum or HI to be seen but there is HI)
• GUI (HTML) for meerkathi configuration editing
• QA report (jupyter notebook)

[bennahugo]

okay politsiyakat and rfimasker is in. Still need to integrate them into meerkathi. Need to get sunblocker and the updated msutils onto pypi next. @SpheMakh I'm striking through as I go along

[bennahugo]

@gigjozsa @paoloserra After this we will lock master and only allow changes via pull requests. This is needed to run changes through automated assurance testing (using IC5264)

[paoloserra]

What tests are you thinking of? Remember that with IC5264 we are still above thermal noise (also in spectral line). We don't know how much of the excess noise is due to our pipeline and how much to limitations in the data.

[bennahugo]

@paoloserra noise doesn't matter in this case. I just need the pipeline to run through end to end to ensure changes don't break functionality.

[gigjozsa]

We (at least Paolo and I) can approve pull requests, right?

[SpheMakh]

Everyone in the meerkathi-admins team can approve pull requests. But I couldn't add @paoloserra to the team since he is not part of the ska-sa github repository. Maybe @gigjozsa (or @o-smirnov ) can add him?

[o-smirnov]

I upgraded @paoloserra to admin status for this repo.

[SpheMakh]

On adding spectral weights to the ms. We got a tool to do this using the spec. sensitivity values, but we'll have to rely on H5TOMS to create the WEIGHT_SPECTRUM and SIGMA_SPECTRUM columns. In the interim, I'm using CASA's initweights task.

[o-smirnov]

Can't you create your own columns from scratch?

[SpheMakh]

I tried a bit, but creating MS dminfo is a bit involved. It would be better if a column with the same type, and shape already existed in the MS.

[o-smirnov]

If only you spent your nights studying the DDFacet code, you'd know there is a simple solution to this:

            desc = self.ms.getcoldesc("DATA")
            desc["name"] = col_name
            desc['comment'] = desc['comment'].replace(" ", "_")  # got this from Cyril, not sure why
            # if a different type is specified, insert that
            if dtype:
                desc['valueType'] = dtype
            self.ms.addcols(desc)

[SpheMakh]

I create the coldesc using "pyrap.table.makearrcoldesc" its the dmiinfo that I couldn't get. But I see in the last line that its not being parsed, probably will be auto-generated if not given.

[o-smirnov]

I think the idea above is to just get that info from the DATA column, and only change the data type.

[SpheMakh]

The data types for the coldmi (data management info) are all ints, and there are a bunch of Keywords are don't really understand.

{'NAME': 'TiledDATA',
 'SEQNR': 17,
 'SPEC': {'ActualMaxCacheSize': 0,
  'DEFAULTTILESHAPE': array([  4,  40, 819], dtype=int32),
  'HYPERCUBES': {'*1': {'BucketSize': 1036640,
    'CellShape': array([   4, 1705], dtype=int32),
    'CubeShape': array([     4,   1705, 376320], dtype=int32),
    'ID': {},
    'TileShape': array([   4, 1705,   19], dtype=int32)}},
  'IndexSize': 1,
  'MAXIMUMCACHESIZE': 0,
  'SEQNR': 17},
 'TYPE': 'TiledShapeStMan'}

[o-smirnov]

That's the tiling information, but why try to understand it? The whole idea of the recipe above is that you grab it from the DATA column and not worry about it. The only thing you have to change is desc['valueType'] = float.

[SpheMakh]

Yup, that's what I'm trying.

[SpheMakh]

Cool, it works when I don't give dminfo.

[SpheMakh]

So we can now create an populate WEIGHT/SIGMA_SPECTRUM columns. We''ll continue discussion about estimating the weights on the MSUtils repo

[SpheMakh]

As discussed with @o-smirnov and @bennahugo. The best way to estimate expected thermal noise, is to simulate noise using the spec. SEFD values into a NOISE column in the MS, then image that column. This way you account for:

• Flags
• Imaging weights
• Weights in WEIGHT or WEIGHT_SPECTRUM column

I'll extend the addnoise function in MSUtils to use a frequency dependent SEFD . Which will be straight forward with the noise estimation tools in MSUtils.ClassESW

[paoloserra]

Can we keep the simpler approach (i.e., no imaging) for natural noise estimates?

[o-smirnov]

We should have both approaches available.

[gigjozsa]

I vouch for that. We need to know how far we are from thermal as calculated in a simple formula.

[SpheMakh]

Yup, we will have both approaches. The expected, natural weighting, estimate should also be a product of this function.

[SpheMakh]

Changes to MSUtils are in. Now I need to:

• Push to PyPi
• Add sunblocker to stimela

[bennahugo]

Politsiyakat along with the changes to aoflagging and delaycal are almost in as well

[gigjozsa]

Concerning report: In the future the obsreport should also contain: For continuum date

• uv coverage
• PSF
• 1-2 1d cuts through PSF
• inverted PSF
• inverted dirty image

For line data: 2 big PV-cuts through centre of the cube

[gigjozsa]

@SpheMakh Can this be closed?

[bennahugo]

No changes not implemented yet

[gigjozsa]

Can this be renamed?

[KshitijT]

I am closing this issue, since the name is utterly opaque, issue refers to a TODO list and not a particular feature. If anyone wants to keep it alive (if stuff here needs to be implemented (@gigjozsa ?) ) or for historical reasons and/or nostalgia, please reopen it. :)