Imaging Tests

[steven-murray]

In 0c1e69c2adca54776b9740e9094977e6430ff0e6 I've added a new "test" script. In the future, we can modify this to actually do automatic testing. At the moment, it just creates plots for manual inspection.

So far, I've made three tests. These are not supposed to be full MCMC tests, just tests that the mechanics of everything works OK. The ones there so far make some kind of sky (single source at zenith, line of sources through zenith, or a ring of sources at some radius), then they sample the sky with a set of baselines (all gridded centres at the moment), then grid the baselines, and FT back to image space. Output plots show each step -- the original sky, the stitched/coarsened sky, the beam-attenuated one, then the re-constructed sky (follow the plots in an anti-clockwise pattern).

At this point, the original sky is huge, and therefore doesn't need stitching. I've also set pretty much all defaults for every parameter (some of them I had to fix what the default was, on the commit mentioned above). So I use 500 cells, and the same number of UV cells as image-plane cells. There is no umax, so it uses all the baselines and UV cells.

Here's the outputs:

All of them seem to do reasonably well, which is nice. There is a "cross" feature at the centre, which may be a bit worrying. Also, I'm unsure of the normalisation of each.

However, producing these plots brought up a couple of issues... at first the reconstruction was very poor, and I realised it was because the umax was set to 290 by default (it's a very arbitrary default!). When I set it to None (to use all baselines), the reconstruction got a lot better. This means a few things, I think:

1. We should have checks within the code that can ensure that the user hasn't selected a stupid value for a parameter.
2. We should be running these kinds of diagnostic plots for every MCMC we do, to make sure things aren't going really wrong (they're pretty quick). Especially for the instrumental part (i.e. we can take out the EOR and actual foregrounds, sub in a "special" foreground like the ones I've used, but use the same instrumental parameters as in our real run, then check that those parameters actually work).
3. We can use a series of tests like this to explore the limits of what parameters actually work, which can inform how we can do (1).

I've also noticed a few things that need to be fixed from doing this:

1. the stitching doesn't preserve the centre. This is OK if the foregrounds are centreless and periodic (which ours are), but really messes up the examples here (unless no stitching is done, of course!).
2. The foregrounds really should be produced directly in angular space, and should not have the limitation of being confined to the same box size as the EoR simulation. In principle, each foreground could be defined over a different size sky, and this should be able to be dealt with.

The point is, doing these kinds of tests are super-useful, and we should run them after every update.

I'd value your ideas on whether stuff is wrong in these plots, and how we might fix it!

[steven-murray]

The latest commit (b5add1c313a9b2331186d2ae6f80ccb2381d253f) attempts to fix point (2) from above (making foregrounds have independent size) amongst a few other little fixes. New imaging test plots:

The units still have to be checked...

The most striking thing that's wrong is the Gaussian sky. There seems to be a blank cross through the centre (this persists in all cases if you look hard enough). We will need to fix this!

[BellaNasirudin]

I finally managed to fix this issue in commit 80f1aca6939c3408a9aebbb7106d0f9d67b5619f

Long story short, it is caused by the multiple discrepancies when we're setting the values of the grid centre "baselines" (see commit for details). I have set everything to be based on the minimum frequency.

Another reason for this issue is in gridding the visibilities in likelihood, it is not right to set umin=-umax. With fft, the umin always have a different value when the number of cells is even. In my original code, I have something that "preserves" this quality, even when we have a cutoff uv.

[steven-murray]

Ah! Excellent! This is exactly the next thing I was going to check this morning, but you already solved it :-)

I will check whether the MCMC turns out better now.

[steven-murray]

With your fixes, I was also able to get good-looking reconstructions. The next step, it seemed, was to try imaging with a tiled/coarsened sky. In 1c4aed4f98abce7f5e09ea6dc86bd74f5da3e6a4 I added a test to this effect. It uses a periodic sin-wave sky, which is tiled to almost exactly double its breadth, with very little coarsening (i.e. the original cell size is about 0.0402 or something, while the tiled cell size is exactly 0.04). I get the following:

The direct and instrument-based reconstructions look the same, which we would expect since we're still using the same "grid" of baselines. The reconstructions look very good, but the tiled+beam sky has a little bit of weirdness in it (though it's mostly good). I think it's probably a resolution effect, but it will be good to keep in mind. One shortcoming of the tiling/coarsening at the moment is that any cell whose centre belongs in a given coarse cell is fully added to the cell. It would be better if it were added in proportion to how much of the cell resided in the coarse cell, but that's a lot trickier. I'll create an issue for that to keep it in mind. Meanwhile, I'll see if I can get it to use exactly no coarsening, to see if the problem persists.

[steven-murray]

On another note, that same commit also includes a "diagnostics" module, which has a function for making the above plot. This means we can regularly call it before doing an MCMC, for example, to assess how well we are imaging before we begin.

[steven-murray]

I checked when I don't do any coarsening and get:

This looks perfect. Just have to remember that the tiling starts at the left edge, not the centre, so in general the centres of the top two left-most plots don't align. This is not a problem as if we're tiling the simulation, it must be periodic (i.e. centreless) anyway.