Tweaking calibration parameters -- robust, uvmin

[twshimwell]

Need to conclude what the best killms weighting and uvlimit.

On multiple fields do: no uvmin natural weighting uvmin XX natural weighting uvmin XX robust weighting

where XX could be 1.5km, auto or something else in that region.

[mhardcastle]

just robust weighting is also an option -- seems to do at least as well as uvmin (I suspect effectively it imposes a uvmin as some points get zero weighting).

Tests are in progress at this end.

[cyriltasse]

@mhardcastle that's with the new kMS weighting fix?

[mhardcastle]

Some points still get zero weights with the new killms. I guess we won't know which of these options is better till the tests have all run with the new version.

[cyriltasse]

I did not check the zero weights aspect (are you talking about the IMAGEING_WEIGHT?) - I just fixed so that the uvcut vs zero weights outside uvrange was behaving the same...

[mhardcastle]

No no, this is the weights in Briggs weighting of the visibilities in killms. Some of them are zero. I don't think it's a problem.

[twshimwell]

OK, for these tests I'm intending to start with exactly the same image_dirin_SSD* files and use:

[machine] NCPU_DDF=32 NCPU_killms = 28

[data] mslist=mslist.txt full_mslist=big-mslist.txt

[image] imsize=20000 robust=-0.15 psf_arcsec=12.0 final_robust=-0.5 final_psf_arcsec=6.0 do_decorr=True HMPsize=10

low_imsize = 6000 low_psf_arcsec = 20 low_robust = -0.25

[control] restart=True dryrun=False bootstrap=False second_selfcal=False

[masking] tgss=/home/shimwell/Dropbox/TGSSADR1_7sigma_catalog_3.fits extended_size=2500

[solutions] ndir = 60

[bootstrap] catalogues = ['/home/shimwell/Dropbox/Tier1-survey/bootstrap-cats/VLSS.fits.txt','/home/shimwell/Dropbox/Tier1-survey/bootstrap-cats/wenss.fits.txt'] names = ['VLSS','WENSS'] radii=[40,10] frequencies=[74e6,327e6]

Then another few runs with

[solutions] uvmin = 1.5

[solutions] uvmin = 1.5 wtuv = 0.1

[solutions] robust = 0

[mhardcastle]

ndir=45 perhaps?

[twshimwell]

OK, sounds good.

[mhardcastle]

Also maybe ncpu_killms=28 is a bit eccentric?

[twshimwell]

ah yes thats an artefact. It was meant to be 32.

[twshimwell]

Is there a way to see the blbased normalisation values?

[twshimwell]

As with previous tests the suppression of very extended flux is too high with no uvcut so I think we can rule out using nouvcut in the calibration at the moment (attached shows comparison of an ~8arcmin faint extended structure with left nouvcut and right uvcut=1.5km)

[twshimwell]

In the auto_uvmin for my field at the first killms I get a uvmin of just 0.22km which is a bit too short to keep the diffuse emission. Could this be because I had copied the data from a complete run? Whereas maybe this relies on having just run the image_dirin_SSD step?

[mhardcastle]

Re auto_uvmin: a rootname_uvmin.txt file will be read instead of re-doing the calculation if it exists. So it'd be possible for a copy where something has changed to end up giving the wrong answer. Might be worth checking whether that's the problem.

Failing that, it sounds like the auto_uvmin is just not working for some reason. I've just pushed a slight change to the test code in histmsamp.py so calling it with python histmsamp.py mslist.txt image_rootname will make a plot which should show how max flux varies with baseline. Might be interesting to see what this shows for your problem dataset.

We discussed a solution where we impose a minimum below which auto_uvmin won't go and that'd be my preferred solution. To do this we could just change things so that solutions_uvmin is used as a fixed minimum if auto_uvmin == False and as a lower bound if auto_uvmin == True.

[twshimwell]

That sounds like a good idea, does 1.5km seem reasonable for that minimum?

Here is the histogram

[mhardcastle]

I might go for 1km.

What's the total apparent flux that the code finds in the image (just off the top of the scrollback in your terminal there)?

Do you have something weird in the image like a bright source down the beam?

[twshimwell]

Yeh the brightest source is 1.5deg from the pointing centre (two lobes which are quite compact and have peak fluxes of 5Jy and 4Jy)

The total apparent flux is 110Jy.

[twshimwell]

1km sounds fine to me too.

[mhardcastle]

Hmmm, not sure that small errors in the normalization on a 10Jy source could be responsible for the total flux missing the maximum flux like that. It's possible that something in the sampling/histogramming code is screwing up for this particular dataset, but I can't see what.

Anyway, pushed a change that should make solutions_uvmin act as a lower bound if auto_uvmin is True. That gets us the best of both worlds I think.

[twshimwell]

OK thanks, I've started my run using this latest version.

[mhardcastle]

I think we've converged on this, closing for now.

[twshimwell]

The autouvmin is coming out at 5.3 for one of my fields (one with a very large galaxy) and remaining around this value for all 3 times it is calculated. This results in very blotchy looking images so far (I just have the image_ampphase image so far). I wonder if we should also put a maximum in the uvmin?

[mhardcastle]

I think we'd need to test whether that actually solves the problem. Presumably this is coming out as it is because there's lots of unmodelled large-scale structure.

If we are basically going to end up fixing the uvmin somewhere between 1 and say 2 km then we might as well bite the bullet and just put a standard value in the config file, but it may also be that there are just some fields where it's hard to get this right.

Want to test whether a fixed uvmin would do better in this case? Reopening...

[twshimwell]

Yeh I can run this field with the standard auto_uvmin and again with a uvmin of 1.5km. Will take a little while to go through these runs. Perhaps in the meantime if fields are coming out with very large uvmins then just don't run those ones for now.

[twshimwell]

The imaging is taking quite a while for this field, perhaps because this bright galaxy is so large. I have a comparison between image_ampphase1 though (attached left is uvmin=1.5km right is auto_uvmin which comes out about 5.3km). You can see that using 5.5km as a uvmin produces a very blotchy image (1.5km is still pretty blotchy for this field). The full high resolution image is at higher resolution which improves it a bit but not great still. The histmsamp for this field is also attached and looks a bit odd maybe?

Are any of your fields coming out with very large uvmins @mhardcastle ? Perhaps this is the worst one in HETDEX for this (maybe the brightest very large object).

[mhardcastle]

OK, yeah, I have one that sticks at 3.2 km and it looks (still in progress) as though it will be rather blobby, though most start high and get lower through the selfcals as intended.

I guess we conclude that this is just too affected by image structure, noise, RFI and so on to be useful, and go for a fixed 1.5 km.

[twshimwell]

To me that seems like a reasonable thing to do or in a effort to make things consistent over all the fields already done we could alternatively have a small range... although perhaps 1.5km is just simpler (especially when we come to doing simulations). I'd be happy with 1.5km.

[mhardcastle]

Let's not worry too much about consistency and just go for 1.5km. I have changed tier1.cfg accordingly.

Closing this again (-: