Final decision of sources suppression as a function of radius from the phase center

[atemkeng]

So, what is our final decision? Averaging 100s or 50s over time? Averaging 6.2MhZ or above over frequency at 1.4GHz. I have to produce these graphs. Also, what is the final field of view? 1deg and 2deg? 2deg and 3deg? or ...

@o-smirnov

[o-smirnov]

I can't say it's final yet, so let's make all the following combinations and then refine based on what the results look like:

Filters: avg, sinc, sinc 100x25, sinc 150x50

FoVs: 1, 2, 4 deg

Bandwidth: 1 MHz, 2 MHz, 4 MHz

Time: 25s, 50s, 100s

Calculate suppression as a function of radius (r=0...6 deg)

I count 108 combinations in total, think it's reasonable?

Also, as discussed, I'd like to see: flux of source in image plane (natural & uniform weighting), amplitude of longest baseline, amplitude of medium baseline, amplitude of shortest baseline.

Thus for each of the 108 combinations you need to generate four vectors of numbers. Store them in a cPickle or something, because I'd like to load them into a notebook and play with the best way of presenting the plots.

On Fri, Jun 12, 2015 at 4:38 PM atemkeng notifications@github.com wrote:

So, what is our final decision? Averaging 100s or 50s over time? Averaging 6.2MhZ or above over frequency at 1.4GHz. I have to produce these graphs. Also, what is the final field of view? 1deg and 2deg? 2deg and 3deg? or ...

@o-smirnov https://github.com/o-smirnov

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10.

[o-smirnov]

And besides those four vectors of numbers, we also need the noise ratios of course.

[atemkeng]

Yes ~ 108 combinations, in exact 90 combinations as Avg doesn't varies as a function of a parameter that defined the FoV. Thus, 9 combinations for Avg and 27 for each of the sinc. I will produce the plots.

[o-smirnov]

Yes, and get me the raw data that goes into the plots as well please.

On Fri, 12 Jun 2015 21:42 atemkeng notifications@github.com wrote:

Yes ~ 108 combinations, in exact 90 combinations as Avg doesn't varies as a function of a parameter that defined the FoV. Thus, 9 combinations for Avg and 27 for each of the sinc. I will produce the plots.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-111597145 .

[atemkeng]

To produce all these plots under 4hrs observation is a had journey! takes a lot of time to process. I wrote the pipe line and run some few (18plots) yesterday till now in process

[o-smirnov]

How long? If too long (i.e. more than a day), then just give me snapshots. Remember, this is exploratory still, I'm still going to ask you for a "final" set of plots based on what we find.

On Sun, Jun 14, 2015 at 12:23 PM atemkeng notifications@github.com wrote:

To produce all these plots under 4hrs observation is a had journey! takes a lot of time to process

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-111810396 .

[atemkeng]

Ok cooll, I think I should do this for snapshots and when you find a suitable plots let me know I should produce directly. The pipe line is 18 curves for 1Mhz, sinc-0x0 and sinc-100x25 for all FoV, all integrations as you mentioned. yesterday at 7pm till know on going.

[atemkeng]

I just noticed something on the thermal noise. It is very cool to re-sample over short time like (25s,50s) and use wide FoV (3deg, 4deg) and overlap the filters (sinc 3x2, sinc 4x3). The noise goes down well. I will send you some of the results.

[o-smirnov]

Good, I was just going to ask for a shorter timescale result. Do a 1deg fov as well please.

On Mon, 15 Jun 2015 08:31 atemkeng notifications@github.com wrote:

I just noticed something on the thermal noise. It is very cool to re-sample over short time like (25s,50s) and use wide FoV (3deg, 4deg) and overlap the filters (sinc 3x2, sinc 4x3). The noise goes down well. I will send you some of the results.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-111949365 .

[atemkeng]

Check out 30 plots data for 1Mhz and all integrations (25s,50s,100s) and all possible overlap. See the ReadMe.

https://github.com/atemkeng/Windowing-Functions/tree/master/ResultsData

[atemkeng]

And in this regime (25s, 50s) sinc(r) will do well.

[atemkeng]

Longest, Shortest and medium baseline length source suppression as a function of radius.

[o-smirnov]

Aha, looking good. But, I don't see the red curve. Also, the legend has four "avg" and only two "sinc" entries?

On Mon, Jun 15, 2015 at 6:40 PM atemkeng notifications@github.com wrote:

Longest, Shortest and medium baseline length source suppression as a function of radius.

[image: baseline_longmidshort] https://cloud.githubusercontent.com/assets/6806881/8165081/bcc88a40-138d-11e5-9e32-38b0862980bd.png

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-112130947 .

[o-smirnov]

I can't load your pickle. Getting "ValueError: insecure string pickle". The interwebs suggests this may be due to you forgetting to close the file, so it wasn't completely written...

While we're at it, could you please include the per-baseline (shortest, medium, longest) amplitudes in the data structure? Your notes suggest this data is not in there yet.

Also, make sure you take shortest, medium, longest in terms of E-W distance on Earth, rather than absolute length.

[atemkeng]

Sorry, these are the update.

Zoom on the Mid baselines Sinc Vs Avg

Zoom on the shortest baseline Sinc Vs Avg

[o-smirnov]

Ah I see. So even on the medium-length baselines the sinc is pretty much a boxcar. Is this for a Fov of 1deg? Non-overlap?

[atemkeng]

Yes, a FoV of 1deg without overlap.

[o-smirnov]

OK, makes sense. Going to need a massive overlap for BDWFs to make a difference in this regime, otherwise we're just fiddling with longer baselines. Any reason why we can't do an overlap factor of 10? Of 100?

[atemkeng]

Yeah we can do this but not for a 1 deg FoV. For a FoV >1 it is interesting and more interesting when we re-sample over short time. As you mentioned in the paper, the computational time is the only drawback.

[o-smirnov]

Why can't we do this for a 1deg FoV?

On Mon, Jun 15, 2015 at 7:39 PM atemkeng notifications@github.com wrote:

Yeah we can do this but not for a 1 deg FoV. For a FoV >1 it is interesting and more interesting when we re-sample over short time. As you mentioned in the paper, the computational time is the only drawback.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-112149608 .

[atemkeng]

When the overlap time and/or frequency get too large, we started to suppress across the Field of view, what is not good although be reduce the noise significantly by doing that.

[o-smirnov]

That's strange, why would that be? Need to understand this better.

On Mon, 15 Jun 2015 20:02 atemkeng notifications@github.com wrote:

When the overlap time and/or frequency get too large, we started to suppress across the Field of view, what is not good although be reduce the noise significantly by doing that.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-112155710 .

[o-smirnov]

Interesting. At FoV=1deg, 1 MHz, there is absolutely no gain in any of the BDWFs as far as smearing is concerned, they all do worse than averaging. Even at FoV=2deg, the gains are miniscule. Your old plots seem to show BDWFs doing a lot better -- but they were at 6.25 MHz -- so let's wait for the higher-bandwidth plots.

[o-smirnov]

By the way, is this sinc(r), or sinc(x)sinc(y)?

[atemkeng]

All are Sinc(x)sinc(y). But my mind tell me that sinc(r) will work better for this regime. I have to run  and see.

-------- Original message -------- From: Oleg Smirnov notifications@github.com Date:15/06/2015 10:30 PM (GMT+02:00) To: atemkeng/Windowing-Functions Windowing-Functions@noreply.github.com Cc: atemkeng m.atemkeng@gmail.com Subject: Re: [Windowing-Functions] Final decision of sources suppression as a function of radius from the phase center (#10)

By the way, is this sinc(r), or sinc(x)sinc(y)?

— Reply to this email directly or view it on GitHub.

[atemkeng]

I have added 2MHz data in the same Rep just the naming that change.

[o-smirnov]

OK check out the notebook at https://github.com/o-smirnov/Windowing-Functions/tree/master/paper1/data.

[o-smirnov]

2MHz looks "more useful". But see the calculation at the end of the notebook. In terms of "uv-distance" 100s is 10MHz at 1.4Ghz. We should've worked this out earlier!

Anyway what this tells me is that we also need data at 8 MHz BW to complete the picture. 100s/8MHz and 50s/4MHz and 25s/2MHz are "balanced" in that the sampling bins become square-ish. 100s/1MHz is not balanced at all (long "thin" bins), hence the poor results. Will be a useful point to make in the paper.

[atemkeng]

I prefer to make Figure 12. and table3 when we have chosen a proper window. I have made correction in the paper. Thanks

[atemkeng]

I have added 10Mhz and 100s, 2.5Mhz and 25s.

https://github.com/atemkeng/Windowing-Functions/blob/master/ResultsData/picklefile_chandwidth-10Mhz-int-100s_2.5Mhz-int-25s_FoV-1-2-4deg_overlaps.data

[atemkeng]

This is just a proof of concept. I have simulated a 4hrs hires of 1s integration then I re-sampled to 100s and 1Mhz channels width to see how the noise increased. The noise doesn't rely increase you can compare the plot with the snapshot of 100s and 1Mhz. But I noticed that the noise is amplified when I increase the channels widths to 2Mhz, .....

The plot below is from a 4hrs observation, integration 1s and 180 channels of width 0.0669MHz. I then re-sampled the time to 100s (average 100 samples) and the channels to 1Mhz (average 15 samples) then applied sinc-4x3. Indeed the different between this plot and the above plot is on the number of channels of the lores MS. Here the lores MS has 10 channels and the above the lores MS has 1 Channel. see the data and the configuration file:

https://github.com/atemkeng/Windowing-Functions/blob/master/ResultsData/picklefile_chandwidth-1Mhz_int-100s_FoV-2deg_overlaps.data

https://github.com/atemkeng/Windowing-Functions/blob/master/ResultsData/wfconfigfileVLAC-Bandwidth-1MhZ_Sinc-150x50_FoV-2deg_int-100s.cfg

This illustrate what you mentioned the last time. The noise increases.

@o-smirnov

[atemkeng]

Run configuration https://github.com/atemkeng/Windowing-Functions/blob/master/Code/wfconfigfileVLAC-Bandwidth-1MhZ_Sinc-150x50_FoV-2deg_int-100s.cfg

python runwf-fluxradius.py "ls wfconfigfileVLAC-Bandwidth-1MhZ_Sinc-150x50_FoV-2deg_int-100s.cfg"

[o-smirnov]

Your latest data update is missing the "avg" entry for int25s.

[atemkeng]

The avg entry is in, see below or maybe I don't understand well the question.

In [14]: file = pickle.load( open( "picklefile_chandwidth-10Mhz-int-100s__5MHz-int-50s_2.5Mhz-int-25s_FoV-1-2-4deg_overlaps.data", "rb"))

In [15]: file.keys() Out[15]: ['bandwidth2.5Mhz', 'bandwidth10Mhz', 'bandwidth5Mhz']

In [16]: file['bandwidth2.5Mhz']['int25s']['avg'] Out[16]: {'flux': array([ 1.00000072, 0.99432814, 0.98798674, 0.98429561, 0.97074866, 0.94986469, 0.94238389, 0.92736363, 0.90549356, 0.88878238, 0.87042594, 0.84695655, 0.81951201, 0.79293972, 0.77000087, 0.74441427, 0.71704745, 0.68868494, 0.66035086, 0.63737839, 0.61452162, 0.59138399, 0.56836116, 0.54577726, 0.52386862, 0.50282097, 0.48292813, 0.46506119, 0.44789526, 0.43149257, 0.41588369, 0.40106604, 0.38703731, 0.3737621 , 0.3612107 , 0.34933251, 0.3380923 ]), 'noise': 0.00033000507}

On 26 June 2015 at 16:28, Oleg Smirnov notifications@github.com wrote:

Your latest data update is missing the "avg" entry for int25s.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-115708700 .

[o-smirnov]

It's not there for 10MHz/25s though.

Please also provide 10MHz/50s for completeness.

[atemkeng]

Ok I get it now. The data is only for 2.5MHz/25s, 5MHz/50s and 10MHz/100s.

[o-smirnov]

I'm still bothered by the sinc2 results going above unity in recovered flux. As far as I can see, this is mathematically impossible. For a 1 Jy source off-centre, all the input (hi-res) visibilities have unity amplitude and some phase. The sum of N complex numbers of unity amplitude cannot have an amplitude larger than N. It has to be an error in your normalization somehow! Please triple-check.

[atemkeng]

10MHz and 50s, 10MHz and 25s added. See link below

https://github.com/atemkeng/Windowing-Functions/blob/master/ResultsData/picklefile_chandwidth-10Mhz-int-50s_ints-25s_FoV-1-2-4deg_overlaps.data

[atemkeng]

Looks very strange to me. I have to check this with the hi-res data and see if I have the same fluctuation within a radius of 2deg.

Observed frequency 14GHz channels width 10Mhz integration time 10s

Need to increase the resolution upto 0.25arcsec then you will the following

@o-smirnov

[atemkeng]

Visibility amplitude Vs radius @o-smirnov

[atemkeng]

A 5s re-sampling time and 16Mhz re-sampling frequency plot. Here the noise is kite amplify with 2deg sinc, the noise ratio is 6.412. It is just very difficult to find where averaging drop below 0.9 in this regime (5s, 16Mhz). I am simulating again with a filter of 0.5deg, 1deg to see how the noise behave and also with Bessel. It is clear that with BDWF's the is no smearing over the VLBI FoV but now as you said the noise matter, I have to fine the best filter for this. @o-smirnov

[o-smirnov]

Let's split VLBI out into a separate issue.

In the meantime, I spent the flight trying to get the last section of the paper into shape. Here'sthe "final" plots I've arrived at:

I think these are the filters we'll want to do a full run for. But first you must figure out the noise situation. I have looked at the noise obtained by different averaging settings:

(10.0, 100, 'avg', None) 0.000311286
(2.0, 25, 'avg', None) 0.000330005
(4.0, 100, 'avg', None) 0.000478045
(1.0, 100, 'avg', None) 0.000487739
(1.0, 50, 'avg', None) 0.000451033
(4.0, 25, 'avg', None) 0.000505531
(1.0, 25, 'avg', None) 0.000471339
(4.0, 50, 'avg', None) 0.000461966
(2.0, 100, 'avg', None) 0.000497547
(2.0, 50, 'avg', None) 0.000491046

This makes no sense! For boxcar, noise should be the same no matter how you average, since in the end the 0,0 pixel of the image is just a sum of all the high-res visibilities. The discrepancy worries me.

[atemkeng]

Yes but this is not how I am computing the noise. I have to make a noisy image and take the std. I wonder if the noise will stay constant for the same resolution and image size. Anyway I will check this again.

[o-smirnov]

Yeah there will be subtle differences. Your pixels have to sample the psf by a factor of several. Check the two "low noise" cases first.

On Tue, 4 Aug 2015 12:50 atemkeng notifications@github.com wrote:

Yes but this is not how I am computing the noise. I have to make a noisy image and take the std. I wonder if the noise will stay constant for the same resolution and image size. Anyway I will check this again.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-127563435 .

[atemkeng]

I verified , for (10.0, 100, 'avg', None) I simulated to time and I got the following noise 0.00030597381 and 0.00030580908.

[o-smirnov]

Compare (1.0, 100, 'avg', None) and (10.0, 100, 'avg', None). You should have the same (or at least very close) image noise, as long as the number of output channels in the second case is 1/10th of those in the first case (i.e. the effective output frequency range is the same).

[atemkeng]

I really do not know what happen. I run all the configurations files again.

For (1., 100, 'avg', None) I simulated this twice to see, and this is what I have 0.00031786656 and 0.00031524149. The hires, lores MS and the configuration file are here: elwood: /home/atemkeng/NVSS/TEST-VERIFICATIONXXXXXX/data-1MHz-100s

For (10., 100, 'avg', None) I also simulated this twice, and this is what I have 0.00032379036 and 0.00031327232. The hires, lores MS and the configuration file are here: elwood: /home/atemkeng/NVSS/TEST-VERIFICATIONXXXXXX/data-10MHz-100s

@o-smirnov

[o-smirnov]

OK, I can only assume you changed or fixed something in your scripts midway through. Please understand the problem. We're going to need to re-run everything from scratch now. May as well use the new intervals we settled on: 25s, 50s, 100s, 2.5 MHz, 5 MHz, 10 MHz.

Start with the combinations 25x2.5, 50x5, 100x10, and the filters I used in my plots above. I can then at least re-do the plots and carry on with the interpretation.

[atemkeng]

Refer to https://github.com/atemkeng/Windowing-Functions/tree/master/FinalData for the data, 25x2.5, 50x5, 100x10 for sinc-1x1,sinc-4x3,bessel-1x1,bessel-4x3 and averaging and the configurations files that you can also run.

Notice that I have change the hires channels width to be able to do this as with the old hires channels width I couldn't find "number-of-resample-channels" such that {channel-width}x{number-of-resample-channels}=2.5MHz.

I will suggest that in your above plots we should use for averaging: (25s, 2.5MHz), (50s, 5MHz) instead of (25s, 2MHz) and (50s, 4MHz) to keep the configurations parameters the same and avoid the noise problem.

@o-smirnov

[o-smirnov]

Yep, I'll switch to 25/2.5 and 50/5 for consistency throughout. Thanks, I'll redo the plots tonight and see what things look like.

On Wed, Aug 5, 2015 at 6:21 PM atemkeng notifications@github.com wrote:

Refer to https://github.com/atemkeng/Windowing-Functions/tree/master/FinalData for the data for 25x2.5, 50x5, 100x10 for sinc-1x1,sinc-4x3,bessel-1x1,bessel-4x3 and averaging and the configurations files that you can also run.

Notice that I have change the hires channels width to be able to do this as with the old hires channels width I couldn't find "number-of-resample-channels" such that {channel-width}x{number-of-resample-channels}=2.5MHz.

I will suggest that in your above plots we should use for averaging: (25s, 2.5MHz), (50s, 5MHz) instead of (25s, 2MHz) and (50s, 4MHz) to keep the configurations parameters the same and avoid the noise problem.

@o-smirnov https://github.com/o-smirnov

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/10#issuecomment-128056939 .