32k data

[bennahugo]

So @gigjozsa and I are trying to bandpass analysis and image some full bandwidth 32k data (20mins on 1934)

So far I've run into the following hitches:

• The MeqServer crashes out midway.
• The current AOFlagger strategy doesn't work on 32k data. The RFI gui segfaults when loading the data.

I suppose now is the time to start thinking about rolling our own flagger @sjperkins. Can we lift the data loading and reordering infrastructure out of tricolor and implement a MAD flagger similar to that in cubical @o-smirnov

[o-smirnov]

Khutso just told me they have received approval to open-source all of SDP software under the MIT license. So that kind of breathes new life into tricolor.

Is MadMax any better than what's in tricolor?

[bennahugo]

From what I recall there was substantial RFI left over from tricolor runs, but it may just mean we need to sit down and think about how to improve it

[gigjozsa]

Just to say: these are 20 mins. I am not sure that we need AOFlagger for this analysis but it's a nice headsup.

[bennahugo]

Yea well I'm thinking long term polarimetry analysis more than HI subbands...

[sjperkins]

Perhaps a comparison with AOFlagger is in order. Does @mauch have some insight into how SDP Flagger compares?

[bennahugo]

(In which case you want to make a 32k cube

[bennahugo]

You can maybe fit a polynomial and subtract out the continuum iteratively... sort of an iterative uvcontsub + flag approach

[gigjozsa]

@bennahugo not sure about that, that's 24 kHz resolution. @sjperkins comparison with AOFlagger should be done on a subset (as it fails to work on the whole thing). @all Anyway, heads up, because we might encounter a near-future problem here. Although we might run the flagger on subsets anyway.

[bennahugo]

Well the finer the resolution in lambda^2 the better the support in rad/m^2

[gigjozsa]

@bennahugo Corresponding to a Faraday depth of a few times 10^5 rad/square meter. How many objects require this? :)

[sjperkins]

I suppose now is the time to start thinking about rolling our own flagger @sjperkins. Can we lift the data loading and reordering infrastructure out of tricolor and implement a MAD flagger similar to that in cubical

Yeah I've got some of that infrastructure in place from the busyweek. Part of it involves writing compressed per-baseline time-frequency windows to disk (zarr) to handle truly large scans. Do you want to throw something together to test madmax or are you thinking of something more polished?

[ratt-priv-ci]

I'm working on implementing a form of robust continuum fitting and subtraction for flagging in tricolour. Should have something togerher next week

On Fri, 08 Mar 2019, 11:56 Simon Perkins, notifications@github.com wrote:

I suppose now is the time to start thinking about rolling our own flagger @sjperkins https://github.com/sjperkins. Can we lift the data loading and reordering infrastructure out of tricolor and implement a MAD flagger similar to that in cubical

Yeah I've got some of that infrastructure in place from the busyweek. Part of it involves writing compressed per-baseline time-frequency windows to disk (zarr) to handle truly large scans. Do you want to throw something together to test madmax or are you thinking of something more polished?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/392#issuecomment-470871684, or mute the thread https://github.com/notifications/unsubscribe-auth/ARFXpjDWJFDFVcl0gYp0UqCy6X1AlwLNks5vUjO0gaJpZM4bjEal .

[bennahugo]

So far so good! Made significant improvements to tricolour. Just need to make it chunk up finer than scan boundaries and we should be home free. This is 32k flagged uncorrected data! :)

[bennahugo]

(and yes it flags on quadrature polarized power)

[sjperkins]

Nicely done @bennahugo!

[o-smirnov]

Awesome stuff!

Is it just me, or does a lot more of the band appear usable (compared to 4k)?

[rdeane]

it's not just you - looking very good Ben! Similar effect seen in LOFAR data at higher frequency resolution. I guess this is a short scan, but still, looking forward to seeing the flagged fractions for 32k vs 4k.

On Wed, 13 Mar 2019 at 21:20, Oleg Smirnov notifications@github.com wrote:

Awesome stuff!

Is it just me, or does a lot more of the band appear usable (compared to 4k)?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/392#issuecomment-472568272, or mute the thread https://github.com/notifications/unsubscribe-auth/AD29R5VMEVkiD6EQMAAecJE3oAkXOUKvks5vWU-GgaJpZM4bjEal .

[ratt-priv-ci]

Yup this is just a short 2 minute (41gb) scan. It is only about 30% flagged compared to an aoflagger normal of 50%The flagger burns a hole in the dell box with 100% cpu time in user space for most of the runtime and running through this in only 8 minutes flat. By my estimates we shall be able to flag 5 TB (4hrs ish) of 32k data in less than a day :)

On Wed, 13 Mar 2019, 21:53 Roger Deane, notifications@github.com wrote:

it's not just you - looking very good Ben! Similar effect seen in LOFAR data at higher frequency resolution. I guess this is a short scan, but still, looking forward to seeing the flagged fractions for 32k vs 4k.

On Wed, 13 Mar 2019 at 21:20, Oleg Smirnov notifications@github.com wrote:

Awesome stuff!

Is it just me, or does a lot more of the band appear usable (compared to 4k)?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/392#issuecomment-472568272, or mute the thread < https://github.com/notifications/unsubscribe-auth/AD29R5VMEVkiD6EQMAAecJE3oAkXOUKvks5vWU-GgaJpZM4bjEal

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/392#issuecomment-472579768, or mute the thread https://github.com/notifications/unsubscribe-auth/ARFXpurFiL39K77HzBgRAC6C9Rdzf9Tgks5vWVckgaJpZM4bjEal .

[o-smirnov]

BTW why such a wide range of power across antennas? These are autocorrs being plotted I presume?

[bennahugo]

Nope these are only 10m and up.

[bennahugo]

Note that Sean probably still needs to autoadjust the digitizer and fengine gains for 32k, since the mode is undergoing commissioning.

[gigjozsa]

@bennahugo @sjperkins Can this be closed?

[bennahugo]

I've tested the flagger on 32k data where aoflagger just falls on its hindside :)

Feel free to replicate implementation of the flagger in your pipeline with my strategies here: https://github.com/ska-sa/vermeerkat/blob/master/vermeerkat/plugins/BAT/crosscal.py#L257

[gigjozsa]

In other words, this has been solved as the flagger is implemented, right?

[gigjozsa]

Not an issue any more as tricolour is implemented.

[ratt-priv-ci]

I don't think it works with the correct flagging strategies in the pipeline though, although there is nothing wrong with tricolour. The way I defined my pipeline is the following: bp: flag data (with high sigma strategy) - DFT - KGB - flag residual (with lower sigma strategy) - KGB gc: flag data (with high sigma strategy) - B transfer - image - KG phaseup

• flag residual - flag residual (with lower sigma strategy) - B transfer - KG phaseup target: KGB transfer - flag data (target high sigma strategy), typically resulting in 50-60% of the data flagged, but I've never run into problems imaging. I find that any higher tolerances produces maps with noticeable RFI fringes.

Of course you can tweak it to your requirements, but don't just run it with default settings and expect it to work.

On Thu, Jan 16, 2020 at 8:07 AM Gyula Jozsa (Josh) notifications@github.com wrote:

Closed #392 https://github.com/ska-sa/meerkathi/issues/392.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/392?email_source=notifications&email_token=AEIVPJTQ6OCXP3EKHXPQ5ZLQ572SJA5CNFSM4G4MI2S2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWAD3FTY#event-2953294543, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIVPJRJLK7HYVDSCICFSFLQ572SJANCNFSM4G4MI2SQ .

--

---

Benjamin Hugo

Junior Software Developer SARAO Black River Park, 2 Fir Street, Observatory, Cape Town, Western Cape, 7925 Contact: [+27] 0716293858 <+27%2071%20629%203858>

PhD. student, Radio Astronomy Techniques and Technologies, Department of Physics and Electronics, Rhodes University

Skype: benna.cn