Floor in calibration simulations with perfect calibration

[nicholebarry]

This is a clarification of a problem hinted at in other issues.

There is a power floor in the EoR window when perfect calibration is used. It seems related to how many sources are subtracted. While it is expected that the wedge will change and be affected by source subtraction, the bleed into the EoR window is not expected in simulations.

Here is a gif of the residual images (xx) given various source subtraction numbers, starting with full subtraction, to 6000, to 5000, and so on until 1000 sources subtracted. There seems to be a weird transition between which sources are subtracted about halfway through the gif. The "source gaps" do not appear physical...

This is the resulting PS, in the same order as above. The floor shows up as yellow in the EoR window in the residual PS.

The floor is evident in the 1D PS, presented in the same order as above.

Bryna suggested modeling and subtracting the dimest sources and compare. Here is a difference PS between the model of the the 2950 (out of 6950) of the dimest sources and the residual from subtracting 4000 of the brightest sources (leaving 2950 of the dimest unsubtracted). These two tests have the same sources in them, but are probing different areas on the code. Same flags were used for both runs.

There appears to be a depression of power in the EoR window for the model of the dim sources when compared to the residual from subtracting the bright sources. This might provide a clue as to where the floor is coming from.

1061316176_settings_bright.txt 1061316176_settings_dim.txt

firstpass.o66457.bright.txt firstpass.o66420.dim.txt

[nicholebarry]

Cube images should also be included.

Here is the difference between the dirty cubes for the model dim sources and model bright sources test. Since the dirty relies on the input data and calibration, these should have the same dirty images. They don't.

Here is the difference between the model of the dim sources and the residual of the bright sources. These tests should have the same sources in each, with the only difference being that one is probing the model and one is probing the residual. They should be the same. They are not.

I did check the scale, and this is a couple of orders of magnitude smaller than the images themselves. This leads me to believe that subtraction was performed correctly, and that there are inherent differences.

[nicholebarry]

Comments to questions that Ian had.

Could the sources be actually subtracting perfectly, but the residual image (and not the model) still have EoR in it? I made a difference between a dirty and a model of a run that calibrated and subtracted all sources perfectly, therefore removing all foregrounds and chromatic effects. (note: flagging different for the resulting cube image). The image should just have EoR in it.

So yes, you're right. Most of the signal in the residual image is EoR. I'm not terribly confident that they should have been different EoR realizations though, as what appears to be happening. This test was months ago, so I'm not really worried. I do see "point source" holes in the previous comment's cube image, though. I think I'll do some more investigating on that.

In the output logs, it looks like there are flagging differences between the two runs (one has 1 tile flagged, the other 7), but in the settings files they both have a single flagged tile and the same number of visibilities. Do you know what happened there, and do you think there is a possibility flagging differences are creeping in? (I suspect the settings file is correct and there are no flagging differences, but since it's a usual suspect I want to be sure) Never caught that before. I'm fairly certain tile flagging is working (I'll come up with some for-sure checks, though)

[nicholebarry]

I reran the tests that produced the images above: removing the brightest 4000 sources, and removing all but the brightest 4000 sources. This time, I left out EoR so the final difference between residual and model, respectively, should be 0. Flagging fine, by the way.

Holes! Argh! Am I just hitting the same bug from multiple angles?

[isullivan]

My best guess is that there was a subtle difference between the way the dirty visibilities were made, and the model. For example, the settings files show that the models used a single beam model for all frequencies. Could the original simulation that's being used for the dirty visibilities have used a frequency-dependent beam model? Also, is there still a difference between the dirty images for the dim and bright runs?

Finally, if there are spare CPU cycles, I would recommend re-running everything after a fresh pull and with the double_precision=1 keyword set.

[nicholebarry]

Ian,

The original simulation that's being used for the dirty visibilities also has a single beam model for all frequencies. Cube images for the dirty cubes are now perfectly identical.

Spare CPU cycles currently do not exist, but I'll put it on the list.

On Fri, Jan 8, 2016 at 2:35 PM Ian Sullivan notifications@github.com wrote:

My best guess is that there was a subtle difference between the way the dirty visibilities were made, and the model. For example, the settings files show that the models used a single beam model for all frequencies. Could the original simulation that's being used for the dirty visibilities have used a frequency-dependent beam model? Also, is there still a difference between the dirty images for the dim and bright runs?

Finally, if there are spare CPU cycles, I would recommend re-running everything after a fresh pull and with the double_precision=1 keyword set.

— Reply to this email directly or view it on GitHub https://github.com/miguelfmorales/FHD/issues/39#issuecomment-170146463.

[nicholebarry]

I don't think the proceeding plots/run will help anyone at all...read at your own risk.

I generated visibilities of a sky with one, relatively bright source in it using the actual simulation code in FHD, and robustly tested by Zach. No EoR and no diffuse is included (if the code listened to me). This was input as "dirty" visibilities into the calibration simulation. Again, no EoR or diffuse added. The calibration was "perfect" in the fact that I set it to be so.

The resulting dirty cube image includes the source in the far upper right corner, and a bunch of rumbly business throughout.

The resulting model cube image looks like what I would expect. Nothing but the source and it's PSF.

And if you're curious, the residual cube image.

This seems to make me think that the code ignored my requests for no EoR/diffuse in some capacity. I'm going to input the resulting model visibilities from this run as the new "dirty." This will at least let me know if it's the input or how the input is managed that is wrong.

Ideas welcome.

[dannyjacobs]

Try setting the flux of the single source to something really tiny. 0.0001 Jy. The visibilities should be basically zero.

On Wed, Jan 13, 2016 at 12:59 PM, Nichole Barry notifications@github.com wrote:

I don't think the proceeding plots/run will help anyone at all...read at your own risk.

I generated visibilities of a sky with one, relatively bright source in it using the actual simulation code in FHD, and robustly tested by Zach. No EoR and no diffuse is included (if the code listened to me). This was input as "dirty" visibilities into the calibration simulation. Again, no EoR or diffuse added. The calibration was "perfect" in the fact that I set it to be so.

The resulting dirty cube image includes the source in the far upper right corner, and a bunch of rumbly business throughout.

[image: fhd_nb_sim_unflagged_nodiffuse_onebeam_zenithpointing_calvisflag_overfit_onesource_obs_id_6176_even_dirty_xx_image] https://cloud.githubusercontent.com/assets/5588290/12305968/38935420-b9ec-11e5-9b9c-77c8bd5377dd.png

The resulting model cube image looks like what I would expect. Nothing but the source and it's PSF.

[image: fhd_nb_sim_unflagged_nodiffuse_onebeam_zenithpointing_calvisflag_overfit_onesource_obs_id_6176_even_model_xx_image] https://cloud.githubusercontent.com/assets/5588290/12305999/62d34fe2-b9ec-11e5-8133-41cc03998b05.png

And if you're curious, the residual cube image.

[image: fhd_nb_sim_unflagged_nodiffuse_onebeam_zenithpointing_calvisflag_overfit_onesource_obs_id_6176_even_res_xx_image] https://cloud.githubusercontent.com/assets/5588290/12306047/9a10681e-b9ec-11e5-97db-a4ffacf25ec6.png

This seems to make me think that the code ignored my requests for no EoR/diffuse in some capacity. I'm going to input the resulting model visibilities from this run as the new "dirty." This will at least let me know if it's the input or how the input is managed that is wrong.

Ideas welcome.

— Reply to this email directly or view it on GitHub https://github.com/miguelfmorales/FHD/issues/39#issuecomment-171415430.

Daniel C. Jacobs KE7DHQ National Science Foundation Fellow Arizona State University School of Earth and Space Exploration Low Frequency Cosmology Phone: (505) 500 4521 Homepage: http://loco.lab.asu.edu/danny_jacobs/ MWA: mwatelescope.org HERA: reionization.org PAPER: eor.berkeley.edu

[nicholebarry]

I've simplified the testing a bit in order to be less confusing.

I inputted 10 bright sources (no EoR, no diffuse) as dirty visibilities. For reference, the dirty image cube looks like this (9 sources can be seen within the image, either I indexed wrong or the other one is outside the image)...

I then modeled and subtracted the brightest five of those sources with perfect calibration. The model looks like this...

I then modeled and subtracted the dimmest five of those sources with perfect calibration. The model looks like this...

I can then compare the model of the brightest 5 sources with the residual after subtracting the dimmest 5 sources. Each should have the same sources in them, and theoretically should be identical in the perfect calibration world. They are most definitely not. The difference is small (note the color bar) but it appears as though the bright sources have residual features about them.

I can make the comparison the other way. I can compare the model of the dimmest 5 sources with the residual after subtracting the brightest 5 sources. Again, the sources should be the same, and theoretically should be identical. They also are not. I think it is important to note that now it appears as though the dim sources have residual features about them.

I checked to make sure the dirty for both is in fact the same in the cube images, and they are. Flagging is exactly the same.

As a side note, I ran a single source simulation as well. I modeled and subtracted the single source with perfect calibration, and the residual was exactly zero.

Comments welcome.

[nicholebarry]

Since the level of "weird stuff" is small by about seven order of magnitude, one might think that this is float/double related.

I generated visibilities given the 10 sources in the sky in double precision, and input these visibilities as "dirty" visibilities. I modeled and subtracted the brightest five sources to double precision. I also, separately, modeled and subtracted the dimmest five sources to double precision.

I compared the model of the 5 bright sources and the residual after removing the dimmest 5 sources, which should be the same.

I compared the model of the 5 dim sources and the residual after removing the brightest 5 sources, which should be the same.

Well, the image differences have gotten better. Here is a side-by-side of the brightest sources, with float precision on the left and double precision on the right.

Directories for all the things: Double precision for model/subtract bright 5: /nfs/mwa-09/r1/djc/EoR2013/Aug23/fhd_nb_sim_perfect_cal_noeor_ones_maxcalsources_nod_zenithpointing_notileflag_5outof10_double/ Double precision for model/subtract dim 5: /nfs/mwa-09/r1/djc/EoR2013/Aug23/fhd_nb_sim_perfect_cal_noeor_ones_maxcalsources_nod_zenithpointing_notileflag_minus5outof10_double Float precision for model/subtract bright 5: /nfs/mwa-09/r1/djc/EoR2013/Aug23/fhd_nb_sim_perfect_cal_noeor_ones_maxcalsources_nod_zenithpointing_notileflag_5outof10/ Float precision for model/subtract dim 5: /nfs/mwa-09/r1/djc/EoR2013/Aug23/fhd_nb_sim_perfect_cal_noeor_ones_maxcalsources_nod_zenithpointing_notileflag_minus5outof10

[adampbeardsley]

Just for fun I tried looking at the visibilities from Nichole's tests. Specifically, I formed residual visibilities from the float precision runs, dirty - dim model - bright_model. I then made a "waterfall" of the amplitude of this difference, with channel on the horizontal axis and baseline number on the vertical axis (just showing the first 128*127/2 baselines). There are definitely streaks of actual zero values. If I zoom the color scale it becomes more obvious. Interestingly some baselines have zero difference at higher frequency, but non-zero at lower. The reverse does not appear to happen.

Does anyone want to speculate what's causing this?

[bhazelton]

Hi Adam,

One thing that could be causing baselines to be zeroed at high frequency is if they fall outside the uv grid area at high frequencies but are inside the area at low frequencies. Since this is a simulation, once the baselines go outside the uv grid area they are not calculated, so they'll be zero.

Cheers, Bryna

On Fri, Jan 15, 2016 at 3:29 PM, Adam Beardsley notifications@github.com wrote:

Just for fun I tried looking at the visibilities from Nichole's tests. Specifically, I formed residual visibilities from the float precision runs, dirty - dim model - bright_model. I then made a "waterfall" of the amplitude of this difference, with channel on the horizontal axis and baseline number on the vertical axis (just showing the first 128*127/2 baselines). [image: image] https://cloud.githubusercontent.com/assets/4752917/12367970/1157dd72-bba4-11e5-8fd1-a0f2f12b43c9.png There are definitely streaks of actual zero values. If I zoom the color scale it becomes more obvious. [image: image] https://cloud.githubusercontent.com/assets/4752917/12368013/6d86f7b8-bba4-11e5-9eff-5fdc5ff88fc3.png Interestingly some baselines have zero difference at higher frequency, but non-zero at lower. The reverse does not appear to happen.

Does anyone want to speculate what's causing this?

— Reply to this email directly or view it on GitHub https://github.com/miguelfmorales/FHD/issues/39#issuecomment-172125510.

[adampbeardsley]

Ah, that makes sense. Especially because Nichole just noticed that the "differences" I was seeing to be zero were actually the visibilities themselves being zero. 0-0=0.

On Fri, Jan 15, 2016 at 4:37 PM bhazelton notifications@github.com wrote:

Hi Adam,

One thing that could be causing baselines to be zeroed at high frequency is if they fall outside the uv grid area at high frequencies but are inside the area at low frequencies. Since this is a simulation, once the baselines go outside the uv grid area they are not calculated, so they'll be zero.

Cheers, Bryna

On Fri, Jan 15, 2016 at 3:29 PM, Adam Beardsley notifications@github.com wrote:

Just for fun I tried looking at the visibilities from Nichole's tests. Specifically, I formed residual visibilities from the float precision runs, dirty - dim model - bright_model. I then made a "waterfall" of the amplitude of this difference, with channel on the horizontal axis and baseline number on the vertical axis (just showing the first 128*127/2 baselines). [image: image] < https://cloud.githubusercontent.com/assets/4752917/12367970/1157dd72-bba4-11e5-8fd1-a0f2f12b43c9.png

There are definitely streaks of actual zero values. If I zoom the color scale it becomes more obvious. [image: image] < https://cloud.githubusercontent.com/assets/4752917/12368013/6d86f7b8-bba4-11e5-9eff-5fdc5ff88fc3.png

Interestingly some baselines have zero difference at higher frequency, but non-zero at lower. The reverse does not appear to happen.

Does anyone want to speculate what's causing this?

— Reply to this email directly or view it on GitHub <https://github.com/miguelfmorales/FHD/issues/39#issuecomment-172125510 .

— Reply to this email directly or view it on GitHub https://github.com/miguelfmorales/FHD/issues/39#issuecomment-172126595.

[nicholebarry]

I'm going to start off by saying this is a little...convoluted.

Gridded UV data is saved. Let's use it. I can take the FFT of a gridded UV plane to go to image space (plus some appropriate shifting). I can subtract "dim sources made in model" from "dim sources left over in residual" (see previous posts) in the image plane and in the UV plane and see if there is a difference.

Here is a dirty image (so all 10 sources) for reference. Somehow the images are flipped, but that's okay. Color scale is also really small here.

The model of the dimmest 5 sources looks like this, again for position reference.

Here is the comparison between "dim sources from a residual" and "dim sources made in a model", where each component was FFT'ed, then subtracted to make the comparison. The color scale is really small, and the distribution is not what I expected.

When I make the comparison between "dim sources from a residual" and "dim sources made in a model", where the subtraction for the comparison happened before the FFT, I get something same by eye.

I can take the difference between the comparison done in UV space then FFT'ed to image space and the comparison done in image space. I then get something interesting.

Hey, that looks familiar! Here's a side-by-side comparison of the "super comparison" above and the dirty image.

It's not clustered about dim or bright sources only, but rather all 10 sources. It's also excruciatingly small, and apparently fully negative. This means that the comparison done in image space is slightly larger than the comparison done in UV space then FFT'ed to image space

[isullivan]

I repeated @nicholebarry 's simulations after a couple of bug fixes, and wrote a custom fhd_main to be extra careful. This simulation involved generating model visibilities from the 10 brightest sources in obsid 1061316296, then using those as the dirty visibilities for firstpass runs where I simulated only the brightest 5 or faintest 5 of those 10 sources. I repeated that entire process using the dft approximation and the true dft, floating point precision throughout and double precision throughout (for a total of 4 sets of 3 firstpass runs [make dirty, model bright, model faint]). I verified that in each of the 4 cases the dirty images for the bright and faint models were identical, then made residual images where residual = dirty - bright - faint, which should be zero. In the images below, I have gridded using natural weighting, and scaled the resulting images up by a factor of 1e10. All residual images are on the same color scale.

Residual using the DFT approximation and floating point precision throughout:

Residual using the DFT approximation and double precision throughout:

Residual using the true DFT and floating point precision throughout:

Residual using the true DFT and double precision throughout:

[nicholebarry]

In remaking some old plots, Adam found the potential cause to the floor problem.

Changing the psf_resolution, or the number of calculated points per wavelength of the look-up table that describes the instrumental psf kernal, changes the magnitude of the floor. To summarize, Adam found that changing the default from 100th of a wavelength to 16th of a wavelength not only introduced the aliasing "ghost line" but also raised the floor.

I then ran some tests with the calibration simulation to see if this cleared up our problem. I ran two tests, both with 10 input sources that model/subtracted the brightest 5, with two different psf_resolutions: 200th of a wavelength (double the default) and 1000th of a wavelength (10x the default).

Here is the difference PS of the default minus the doubled psf resolution.

And here is the difference PS of the default minus the 10x psf resolution.

Apparently, doubling the psf resolution did not seemingly change much. However, there is a significant rise is the window and wedge for 10x the psf resolution. We were hoping to see a decrease.

There does seem to be some benefits from the increased resolutions. Below is the 1D PS from a default psf resolution run, the doubled psf resolution run, and the 10x psf resolution run. Note the changes at high k: there is a decrease in systematic rises in the PS. Looking back at the 2D PS, you can barely see this change, which appears as decreases in the lines that span between where the course band lines should be.

I also made difference cube images of these runs out of habit. Below is the difference cube image of default - 2x psf resolution and the difference cube image of default - 10x psf resolution. They are the difference between the dirty cubes with all 10 sources, just to keep things simple.

I don't understand why there should be any significant change in the cube images. What this seems to point to is some sort of normalization issue, since there really shouldn't be a significant increase in image space when the psf resolution is increased.

firstpass.o66802.default.txt firstpass.o66968.2x.txt firstpass.o66981.10x.txt 1061316176_settings_10x.txt 1061316176_settings_2x.txt 1061316176_settings_default.txt

[nicholebarry]

Miguel suggested PS ratio differences to see if, regardless of a normalization issue, the psf resolution is causing the floor problem.

Below is the PS ratio difference of default - 2x psf resolution and default - 10x psf resolution.

Noise-like ratio differences seem to indicate that this is not the cause of the EoR window floor. Better luck next time, I guess.

[nicholebarry]

More testing was done on psf_resolution, now with interpolation!

New tests include changing the psf_resolution, but employing interpolation as implemented by Ian for each test. Each test was done with a 10 bright source input and modeling/subtracting the brightest 5. No simulated EoR signal was added. Calibration was set to be perfect. Each test is compared with the default (psf_resolution of 100 and no interpolation).

2D difference plots are as follows: (default psf res minus interpolation with psf res of...) 8,16,32,100,500.

There are drastic EoR window effects and wedge effects in the residual. I'll be the first to raise my hand and say I don't know why.

Another interesting effect can be seen in the 1D PS. For reference, there are "bumps" at high k that appeared to smooth out with increasing psf resolution during tests without interpolation, which indicated that we needed higher resolution. This lead to the interpolation testing. Below are 1D PS of interpolation tests of various psf resolutions (again in the order of 8,16,32,100,500).

It appears as though the "bumps" are wholly dependent on the psf resolution, and not the interpolation. This is not expected, in part because the "bumps" have been confirmed to be specifically in the EoR window (tested via limiting the the kpar range of allowed values), and therefore should benefit from the interpolation.

1061316176_settings_psf8.txt 1061316176_settings_psf32.txt firstpass.o67196.1.psf32.txt firstpass.o67198.1.psf8.txt

[nicholebarry]

Closed. See comment on Part 2 (#41).