UV extent significantly changes PS

[nicholebarry]

Changing from a uv grid size of 2048 to 4096 and 2048 to 6144, while keeping the same grid resolution (.5 wavelength) and the same subgrid resolution (1/7000th of a wavelength), has significantly changed the resulting PS. Diffuse was taken out for these runs, since Ian pointed out that the diffuse is image-based, thus is grid-dependent. I am currently making PS through gridded uvf to see if Healpix is a problem. Documenting here as well as MWA_data_analysis. First plot: regular minus double extent. Second plot: regular minus triple extent. Note: perfect calibration, imperfect sky model in both simulations

[nicholebarry]

Ian had me look at the residual images of the triple extent and double extent tests, and there is something weird. Below are the Residual XX images for double and triple extents, respectively.

Just to reiterate what we saw, the triple extent run has smeared sources (?) all throughout the beam, but regular, expected faint sources at the edge of the beam (!).

Grid_out files attached firstpass.o64521.1_farextent3.txt firstpass.o65515.1_farextent2.txt

Settings files attached (4096=double extent, 6144=triple extent) 1061316176_settings_4096.txt 1061316176_settings_6144.txt

And the beam images for XX, just to prove they look fine.

[nicholebarry]

Turning on the dft approximation had a significant effect! Even though the power spectrum still looks really bad, the "streaked sources" in the images has decreased. Points to an overflow problem.

[dannyjacobs]

Should I ask what the EW lines are?

[nicholebarry]

You could. I don't have an answer for you. Fingers crossed that it is still related to the overflow problem, but we'll see.

[isullivan]

It looks like there are multiple effects going on here. The obvious overflow bug for 6144x6144 images is now fixed, but 4096x4096 images would have been unaffected. Still, I recommend re-running the full simulations in double precision. Because the value of Pi is slightly different in double precision, model visibilities will never be absolutely identical between runs with floating point and double precision.

[nicholebarry]

I tried a more apples-to-apples comparison tests involving the changing of the image resolution.

Bryna had been worrying about the differences we view being a real consequence of including longer baselines due to the image resolution. I constrained the maximum baseline AND the maximum calibration baseline while varying the image resolution. This way, we really should be seeing very little difference between the two tests. This is not the case.

The stats: Test one -- dimension 3072, deconvolution on zenith obsid, max baseline 700, max cal baseline 700 Test two -- dimension 2048, deconvolution on zenith obsid, max baseline 700, max cal baseline 700

The results:

Difference PS (dim 3072 minus dim 2048) -- This is surprising, since I expected no significant difference!

The difference ratio PS (dim 3072 minus dim 2048)

Difference in the apparent uv frame (dim 3072 minus dim 2048) -- Did this out of curiosity.

This is all very surprising to me. I don't know what is going on here. Suggestions for other tests to try are appreciated.

firstpass.o.fhd_nb_decon_March2016_small_maxall.txt 1061316296_settings_fhd_nb_decon_March2016_small_maxall.txt firstpass.o.fhd_nb_decon_March2016_maxall.txt 1061316296_settings_fhd_nb_decon_March2016_maxall.txt

[nicholebarry]

I did this test using simulations with perfect calibrations, and the difference PS look more along the lines of what I expected, though the fact that there is still some difference bothers me. Nevertheless, this might help narrow down what is causing the problem in the previous post.

The important thing to note here is that all of the sources used in the input data were used in the model, therefore the differences in the residual PS don't really tell us much. I can redo this test with subtracting only a subset of sources if there is interest.

The stats: Test one -- dimension 3072, point source sim on zenith obsid, max baseline 700, perfect cal, all sources in model Test two -- dimension 2048, point source sim on zenith obsid, max baseline 700, perfect cal, all sources in model

I made difference dirty cube images from eppsilon as well...maybe this explains it

[nicholebarry]

Miguel suggested looking at the FHD images. Immediately, I can see a difference between the two runs in terms of the PSF in ways that I didn't think changing the resolution would have influenced.

2048x2048 image

4096x4096

I made sure that I wasn't setting any image tapering filters (since this looks eerily close to that effect!). More specifically, I can change the PSF seen in the images by tapering the uv plane before making images, but this does not get handed down the pipe to the PS (which I have confirmed in the past). What is going on here?

[nicholebarry]

Reviving this issue.

I am comparing two runs, one with a uv extent of 2048x2048, and one with a uv extent of 4096x4096. Both are constrained to only include baselines below 512 wavelengths, and thus really should be the same. In these runs are 5 simulated sources of 5 Jy each, clustered near the center of the image.

Below is the dirty cube image from each, 2048x2048 then 4096x4096.

And the difference between the two (2048x2048 minus 4096x4096).

For the residuals, I calibrated and subtracted three of the source. 2048x2048 then 4096x4096

And the difference between the residuals (2048x2048 minus 4096x4096).

What could be causing these differences? Is this a projection effect like Miguel feared?

[nicholebarry]

We're trying to wrap up the differences of changing the UV extent. I've done two simulations with just five point sources with the same flagging, with perfect calibration, and with including the same baselines. The only difference should be the amount of zero padding in UV, which in turn will change the resolution in image space.

Here is the two tests in image space with uniform weighting. (The model only has four sources in it). Squint and you'll see the image space resolution change between the two. Cool, all looks good.

Here is the same two tests in image space with natural weighting. Cool, all looks good.

If I take the difference between the two natural weighting tests (dimension 2048 minus dimension 4096) in xx.... And in yy... Bryna's theory is that the interpolation from a gridded image to healpix image might change enough between the 2048 and 4096 tests to account for this.

And for those who are curious, here is the 2D PS. Five sources go into the dirty, four sources are modelled, and one zenith source is left over in the residual. Differences are small, yet still about the value of the EoR.

[bhazelton]

Nichole, did you get a chance to check that the pre-Healpix uvf cubes are the same in the region of overlap?

[nicholebarry]

Can't run IDL on MIT at the moment...but it's on my todo list as soon as it comes back online.