VLBI-Investigating-the-effect-of-averaging-vs-correlator-windowing-functions

[atemkeng]

This shows us the effect of averaging across 1.4Khz and 3s. I was wondering if it is not a good idea to meet some one who specializes in VLBI so that we could really know the idea of the re-sampling channel width and time. I wonder how on Iniyan's dataset they average across 16MHz and 4s.

[atemkeng]

I am just playing with the hires integration time and channel width, also the lores re-sampling time and channels width.

[atemkeng]

The following shows a 1s and 10KHz compression for sinc and Avg

This plot shows a 5s compression for sinc and 1s compresion for averaging both with the same channels compresion rate, 10KHz.

From what now I understand, we need to compress more with BDWF in the VLBI regime and therefore we reduce noise and recover more FoV. So, I have some scripts still running

@o-smirnov

[o-smirnov]

What's the frequency, 1.4 GHz?

On Fri, Aug 7, 2015 at 7:02 PM atemkeng notifications@github.com wrote:

The following shows a 1s and 10KHz compression for sinc and Avg [image: vlbi_1s_10khz] https://cloud.githubusercontent.com/assets/6806881/9141071/247adc66-3d36-11e5-8090-5c4c447c74a1.png

This plot shows a 5s compression for sinc and 1s compresion for averaging both with the same channels compresion rate, 10KHz. [image: vlbi_1s_10khz] https://cloud.githubusercontent.com/assets/6806881/9141116/83d72d72-3d36-11e5-832f-a47d76782689.png

From what now I understand, we need to compress more with BDWF in the VLBI regime and therefore we reduce noise and recover more FoV. So, I have some scripts still running

@o-smirnov https://github.com/o-smirnov

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/15#issuecomment-128766394 .

[atemkeng]

The frequency is 4.9 GHz as in Inyan dataset. You will surely have something better than this at 1.4GHz

[o-smirnov]

Hmmm, I'm not sure about 10kHz/1s then. Remember that we get best results when the degree of bandwidth and time averaging is comparable. At 1.4 GHz, 1 MHz ~ 10s. At 5 GHz, it would be (5/1.4)*1 MHz ~ 10s i.e. roughly 3 MHz ~ 10s. So maybe 30 kHz ~ .1s would be a better combination? But do double-check my numbers, as it's late on a Friday and I may be barking up the wrong tree.

On Fri, Aug 7, 2015 at 8:39 PM atemkeng notifications@github.com wrote:

The frequency is 4.9 GHz as in Inyan dataset. You will surely have something better than this at 1.4GHz

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/15#issuecomment-128791412 .

[atemkeng]

ok will do that, I am still runing some few combinations. But I evaluted everything. averaging can only recover a FoV of radius approximately 28Arcsec with 1s and 10KHz integrations at this frequency. So, now I have to find a window and compresions factor than can do better than that.

[atemkeng]

From my investigations, these are the best regimes, (3s, 1MHz), (5s, 1.8MHz) and so on. The VLBI calculator confirm averaging smearing rate on the following plots at 4.9GHz with the longest baseline been 9000Km. The noise getting up is trivial in this VLBI. In fact, here all our windows have high time and frequency resolution, so here there is nothing like boxcar on the shorter baselines as it was with an array like JVLA or an array with long and short baselines. Notice that this VLBI shortest baseline is ~78 time longer than the JVLA C longest baseline. BDWF will be better for VLBI that include also short baselines, in term of noise ratio. @o-smirnov

[o-smirnov]

The sheer flatness of the sinc curve is mighty impressive! Still, 2.88 is a hefty noise penalty, so if we can trade off some FoV loss against lower noise I'd do it. What's the FoV parameter of the sinc set to in this case? What happens if you make the FoV smaller?

Also, what about lower averaging values? 1s 300kHz and .1s 30 kHz would be interesting to see.

[atemkeng]

[atemkeng]

Things are showing up now clearly. Need to simulate now many and observe the behaviour. Now we can control the FoV on VLBI, need to confirm this with many simulation. The plot shows a FoV of 1deg across.

@o-smirnov

[atemkeng]

Hires 0.01s integration and 2KHz bandwidth

@o-smirnov

[o-smirnov]

Could you please point me to the VLBI MS you're using? I'm confused, this

https://github.com/atemkeng/Windowing-Functions/blob/master/FinalData/Data-VLBI/ANTENNAS-conf.txt

...says the shorttest baseline is 267m, but how can this be? None of these stations are that close to each other. Don't you mean 267km?

It may be useful to make a uv-coverage plot for the VLBI experiment.

[atemkeng]

Hi Prof, Sorry for the late reply, this is because Yesterday after the workshop there were no internet, I am just seeing my emails now. Let me point you on the VLBI MS, but let me check again. Thank you

On 11 September 2015 at 15:45, Oleg Smirnov notifications@github.com wrote:

Could you please point me to the VLBI MS you're using? I'm confused, this

https://github.com/atemkeng/Windowing-Functions/blob/master/FinalData/Data-VLBI/ANTENNAS-conf.txt

...says the shorttest baseline is 267m, but how can this be? None of these stations are that close to each other. Don't you mean 267km?

It may be useful to make a uv-coverage plot for the VLBI experiment.

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/15#issuecomment-139550757 .

[atemkeng]

Thank you very much, it is very nice the paper, I had read and made all the corrections in red and the references. There are two references that I could not find, I mentioned in red in the paper. The VLBI shortest baseline is 266.519Km I made an error when I was writing the readMe. I am doing some more simulations for the VLBI, not just a snapshot. So, I will prefer to use this MS for the coverage elwood:/home/atemkeng/VLBI_5s_16MHZBIS/DATA_RESULT/vlbi-hires-0.01s-500Hz.MS/

[atemkeng]

A re-sampling over 1s and 12.5KHz with sinc and bessel. The Hires is sampled at 0.01s and 500Hz channels widths.

[atemkeng]

This combinations improve up to 115KHz@o-smirnov

[o-smirnov]

Now that's a killer result! Be sure to show this to the JIVE guys. This is a snapshot I presume though?

Why this time/freq combination? Wouldn't things work better around 0.5s/50kHz, or 1s/100kHz, ensuring "square" bins (as per the arguments in the paper). This is 1.6GHz observing freq, right?

Also, need to get a few more points close to 0, because otherwise you cannot tell at all what the averaging FoV is -- first data point is already at 0.6.

[atemkeng]

Yeah the results are very promising! I tried 1s and 115KHz but the noise penalty was slightly above 2 what I do not want to see. Yes I am very sure, you can try it also. I will try 1s and 50Khz but as my discussion with Zolt, to achieve a wide FoV (~30arcmin) with VLBI at 1.6GHz you need 0.01s resampled in time and 11KHz in frequency. So for 0.1s resampled in time it is already a compression rate of factor 10 in time and 115KHz it is approximately a factor of 10 in frequency. My aims were to achieve at least a compression rate factor of 10 in frequency. I think this is not a snapshot, 5.5s synthesis with this VLBI (with very long baselines) all the baselines will sweep a complete ellipse during this time.

[o-smirnov]

BTW, is that a frequency overlap of "1.4"? Really? Why such a strange value?

[atemkeng]

No reason, I just have that because I am playing around to find better options.

On 18 September 2015 at 10:20, Oleg Smirnov notifications@github.com wrote:

BTW, is that a frequency overlap of "1.4"? Really? Why such a strange value?

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/15#issuecomment-141379166 .

[o-smirnov]

4x2 will cause less questions from the referee. And BTW, I now realize that with overlap taken into account, you do have more square-ish bins. 0.5sx4 and 115x1.4 is closer to square... but 4x2 will be even closer.

[o-smirnov]

In the data files currently checked in, all the noise penalties appear to be 1 or 1.01. This is impossible.

[atemkeng]

These are the best cases for VLBI. The data for all these plots are saved on github, VLBI repertory.

Hires of 0.001s integration time and 4.75s synthesis at 16GHz with a bandwidth of 150000Hz channelized into 300 channels each of width 500Hz.

1-resampled 0.01s in time and 12.5KHz in frequency

2-resampled 0.1s in time and 25KHz in frequency

3-resampled 0.25s in time and 25KHz in frequency

[o-smirnov]

So is this plot: https://github.com/atemkeng/Windowing-Functions/issues/15#issuecomment-141169800 still correct? Why is it so "flat" along the top, and the most recent ones aren't?

[atemkeng]

Yes if you run with the same conditions you will have the plots #15, but that noise ratio is not correct. You can have flat as you want, the problem is the noise penalty.

[atemkeng]

"Assume 1.6 GHz, FoV tuned to 5' radius. The question is, what is the largest integration time and channel width we can use and have a noise penalty of no more than 1.5 (for the best-performing filter, i.e. bessel-4x3). "

All will depend if you want high time resolution or frequency. The two plots answer your questions (see title of the plots).

Resampling time = 0.5s, Resampling frequency 13KHz

Resampling time =0.25s and Resampling frequency 23KHz

@o-smirnov

[o-smirnov]

OK, and if we allow a noise penalty of up to 2?

On Mon, Oct 26, 2015 at 3:09 PM, atemkeng notifications@github.com wrote:

"Assume 1.6 GHz, FoV tuned to 5' radius. The question is, what is the largest integration time and channel width we can use and have a noise penalty of no more than 1.5 (for the best-performing filter, i.e. bessel-4x3). "

All will depend if you want high time resolution or frequency. The two plots answer your questions (see title of the plots).

Resampling time = 0.5s, Resampling frequency 13KHz

[image: figure_05s] https://cloud.githubusercontent.com/assets/6806881/10729558/f95a17ee-7bf2-11e5-80de-e98d7eb7d587.png

Resampling time =0.25s and Resampling frequency 23KHz

[image: figure_025s] https://cloud.githubusercontent.com/assets/6806881/10729561/fba14d42-7bf2-11e5-81bc-e53ef4860282.png

@o-smirnov https://github.com/o-smirnov

— Reply to this email directly or view it on GitHub https://github.com/atemkeng/Windowing-Functions/issues/15#issuecomment-151127952 .

[atemkeng]

Resampling time = 0.5s, Resampling frequency 26KHz

Resampling time = 0.25s, Resampling frequency 52KHz

@o-smirnov

[atemkeng]

integration time = 10ms, Resampling frequency 0.5MHz

@o-smirnov

[atemkeng]

resample time 0.01s, resample channels widths = 0.125MHZ

resample time 0.01s , resample channels widths = 0.25MHZ

resample time 0.25s , resample channels widths = 0.05MHZ=50kHz

resample time 0.5s , resample channels widths = 0.025MHZ=25kHz

Find the data here and the readME. https://github.com/atemkeng/Windowing-Functions/tree/master/FinalData/Data-VLBI/Data-1.6GHz-VLBI-0.01s-bandwidth-0.12MHz-0.25MHz

https://github.com/atemkeng/Windowing-Functions/tree/master/FinalData/Data-VLBI/Data-1.6GHz-VLBI-0.25s-0.5s-bandwidth-0.05MHz-0.025MHZ

@o-smirnov

[o-smirnov]

Those are some exciting R values! Full synthesis I assume? But we don't have airy2 described in Section 3, so we can't just drop it into the paper at that point unannounced -- either add a subsection on it to Sect 3, or redo with bessel.

Question, is airy2 then significantly better than bessel in this case? Can you put avg, airy2 and bessel on the same plot? (If I'm the referee, I'll ask why the rest of the paper is not using airy2...)

Finally, please do a 5x1 and 10x1 case for 10ms, 0.125 and 0.25MHz. If they need spectral resolution for dedispersion, then overlap in frequency becomes undesirable. So we need to see how well it works without frequency overlap. On the other hand, I want to see what happens when you increase overlap in time.

[atemkeng]

Sorry for that confusion, airy2="bessel2"

I rely do not know if these parameters are snapshot for VLBI or full synthesis.

From your new sets of questions, I changed again the hires number of timeslots and channels.

These are the settings:

Hires: integration time = 0.001s, number of timeslots = 100 channels widths = 500KHz, number of channels = 2500

From these: I fixed 10 timeslots to average and overlap_left=45, overlap_rigth=45 I fixed 500 channels to average and overlap_left=1000, overlap_rigth=1000

LOres: (10ms, 0.12MHZ):

resample time 0.01s (10 timeslots averaged), resample channels widths = 0.125MHZ (250 channels averaged)

bessel-4x3:   overlap_time = 2*15, overlap_freq = 2*250

bessel-4x5:   overlap_time = 2*15, overlap_freq = 2*500

bessel-5x1:   overlap_time = 2*20, overlap_freq = 0

bessel-10x1:  overlap_time = 2*45, overlap_freq = 0 

LOres: (10ms, 0.25MHZ):

resample time 0.01s (10 timeslots averaged), resample channels widths = 0.25MHZ (500 channels averaged)

bessel-4x3:   overlap_time = 2*15, overlap_freq = 2*500

bessel-4x5:   overlap_time = 2*15, overlap_freq = 2*1000

bessel-5x1:   overlap_time = 2*20, overlap_freq = 0

bessel-10x1:  overlap_time = 2*45, overlap_freq = 0 

The noise penalty is correct, this is the output for my noise penalty:

{'avg-0.01s-0.12MHz': 0.0015800094, 'avg-0.01s-0.25MHz': 0.001579888, 'bessel-0.01s-0.12MHz-10x1': 0.0010915709, 'bessel-0.01s-0.12MHz-4x3': 0.00211978182, 'bessel-0.01s-0.12MHz-4x5': 0.0015492204, 'bessel-0.01s-0.12MHz-5x1': 0.0014753153, 'bessel-0.01s-0.25MHz-10x1': 0.0015106348, 'bessel-0.01s-0.25MHz-4x3': 0.0028219804, 'bessel-0.01s-0.25MHz-4x5': 0.0022566614, 'bessel-0.01s-0.25MHz-5x1': 0.0019909651}

If you want to run it see : elwood/home/atemkeng/VLBI_5s_16MHZBIS/VLBI_overlap/TEST-0.5s-256channels/Verify

pyxis noise_estimate[noise=1.,hiresms=vlac-hires-0.001s-500kHz.MS]

this will save you a pickle file name: pickle-noise.data

@o-smirnov

[atemkeng]

To achieve the same FoV with averaging at 10% smearing, we have to re-sample 10ms in time and 15.5KHz in frequency. the compression factor can therefore be evaluate for BDWFs