severe aliasing artifacts when using wsclean as facet imager

[rvweeren]

I noticed clear aliasing in the facet images (and final mosaic), see examples. There are at least a few dozen(!) of these in the mosaic. Note that I used the latest WSClean v1.11. I think we need to invoke the padding (as in the manual facet calibration, when we do the FFT (padfits) to see if that helps. This clearly needs to be resolved before I would trust any science results.

This looks like a different sort of aliasing problem. Similar to what Wendy noticed in the 10SB medium resolution direction independent image (not related to the FFT subtract, but rather the facet imaging itself)

[rvweeren]

In addition to the padding in the FFT step, we need to increase the image size for the facets by 10-20% or so (to have more "boundary" and avoid mirroring/aliasing of sources around image edges which would otherwise up mosaic). I see evidence for this issue in my mosaic. (it is a different problem then the "ghosts" problem above). So we likely need (1) significant padding at the FFT subtract and (2) increased image sizes for the facet imaging ("images are too tight around the facet mask now").

[AHorneffer]

I've seen similarly bad things in my data: Aliasing with WSClean in the NGC891 field. These kind of "close" aliases don't show up in the same way when using casa as facet imager.

I think we should talk with Andre if he has better ideas to solve this, otherwise I guess we indeed have to do the padding.

[darafferty]

Is the padding (for the FFT and for the facet image) only needed for WSClean? If so, I suspect that once you use the full bandwidth and have to pad the images, WSClean will often be slower than casapy. Should we set casapy to be the default imager until these issues are resolved?

[rvweeren]

Note that the more "major" padding only needs to be done at the FFT predict step for the subtract (not for making the facet image). For making the facet image we only need to pad 10% or I think. Note that Andre already recommends to pad by default and that is why wsclean has the "trim" option (also casa by default pads by a factor of 1.2, although it is hidden for the user as it trims automatically)

Martin in your processing with the "manual" scripts do you see problems with wsclean aliasing ? (I thought that I remembered that the padding at the FFT facet image predict step almost completely fixed it)

[rvweeren]

From the WSClean webpage

Summary: This version adds primary beam correction for LOFAR and uses a stronger default anti-aliasing kernel. The new kernel decreases the chance of sources far out appearing as ghost sources, but increases the noise added near the borders of the image. It is recommended to trim the borders with the -trim option.

[mhardcastle]

I think 'almost completely' is right, yes. It was certainly greatly reduced after we put the padding in.

[rvweeren]

Martin: By what factor did you increase the image size with the padding in the FFT model predict ? (the default 1.2 used in your script?)

[mhardcastle]

Looks like the default in the old code was 1.2.

[mhardcastle]

(That's the default in the padfits code, obviously, but also the default in the main doDDE... code when we allowed the user to control it. I think that's what I was using in production.)

[AHorneffer]

About making casa the default imager: cleaning with casa, in particular when using multiscale clean, can take a huge amount of time (hours between major cycles). So even if the predict step with WSClean would take twice as long as the casa predict step it would be faster with WSClean in that case.

[darafferty]

I've added the padding (and reduced -nwlayers-for-size to 12288; see issue #41), but because our cluster is down for maintenance today and tomorrow I could not really test it. So I haven't committed the changes to the master branch, but instead only to the "facetpeel" branch. (The facetpeel branch also has the calibrator-peeling functionality requested in issue #8, which should work but also needs to be tested.) So, if you try this branch, don't be surprised if there are some bugs!

The padding is controlled by two options under [global]: wsclean_image_padding and wsclean_model_padding. The defaults are 1.2 for images and 1.4 for models. (We can remove these options at some point if we find settings that always work...)

[rvweeren]

There seems to be a problem with the padding.

2016-04-04 15:38:00 DEBUG facetimage_facet_patch_539.executable_args: Results for job 0 submitted by ('127.0.0.1', 56336) 2016-04-04 15:38:00 INFO node.lofar.executable_args: Total time 1473.5510s; user time: 9244.3584s; system time: 4532.7627s 2016-04-04 15:38:00 DEBUG node.lofar.executable_args: Start time was 1459797206.7195s; end time was 1459798680.2708s 2016-04-04 15:38:01 WARNING facetimage_facet_patch_539.executable_args: Remote command stderr: /home/rvweeren/software/lofarapr2016/opt/LofIm/lib/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!

2016-04-04 15:38:01 DEBUG facetimage_facet_patch_539.executable_args: compute.dispatch results job 0: job_duration: 1474.57964396, returncode: 0 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Adding node_logging_information 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Writing data map file: /p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/mapfiles/wsclean_image_full2.mapfile 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Writing data map file: /p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/mapfiles/wsclean_image_full2-image.fits.mapfile 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Writing data map file: /p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/mapfiles/wsclean_image_full2-model.fits.mapfile 2016-04-04 15:38:02 INFO facetimage_facet_patch_539.executable_args: recipe executable_args completed 2016-04-04 15:38:02 INFO facetimage_facet_patch_539: Running task: pad_image 2016-04-04 15:38:02 INFO facetimage_facet_patch_539.executable_args: recipe executable_args started 2016-04-04 15:38:02 INFO facetimage_facet_patch_539.executable_args: Starting /usr/local/lib/python2.7/dist-packages/Factor-1.0-py2.7.egg/factor/scripts/pad_image.py run 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Pipeline start time: 2016-04-04T18:52:13 2016-04-04 15:38:02 INFO facetimage_facet_patch_539.executable_args: Limiting to 24 simultaneous jobs/node 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Job dispatcher at 127.0.1.1:39412 2016-04-04 15:38:02 INFO facetimage_facet_patch_539.executable_args: ****** Remote method is local 2016-04-04 15:38:02 INFO facetimage_facet_patch_539.executable_args: Waiting for compute threads... 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Spawning subprocess: cmd=['/bin/sh', '-c', 'python /home/rvweeren/software/lofarapr2016/opt/LofIm/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 39412'], cwd=None, env=None 2016-04-04 15:38:02 DEBUG facetimage_facet_patch_539.executable_args: Request for job 0 from ('127.0.0.1', 45227) 2016-04-04 15:38:02 DEBUG node.lofar.python_plugin: infile = /p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/L343226_SBgr025-10_uv.dppp.pre-cal_chunk9_12656E813t_0g.wsclean_image_full2 2016-04-04 15:38:02 DEBUG node.lofar.python_plugin: executable = /usr/local/lib/python2.7/dist-packages/Factor-1.0-py2.7.egg/factor/scripts/pad_image.py 2016-04-04 15:38:02 DEBUG node.lofar.python_plugin: working directory = /p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539 2016-04-04 15:38:02 DEBUG node.lofar.python_plugin: arguments = ['/p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/L343226_SBgr025-10_uv.dppp.pre-cal_chunk9_12656E813t_0g.wsclean_image_full2', '1.4'] 2016-04-04 15:38:02 DEBUG node.lofar.python_plugin: arg dictionary = {} 2016-04-04 15:38:02 DEBUG node.lofar.python_plugin: environment = {'SHLIB_PATH': '/home/rvweeren/software/root/lib:', 'COMPIZ_BIN_PATH': '/usr/bin/', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', 'LOFARDATAROOT': '/opt/lofar/data', 'PYTHONPATH': '/home/rvweeren/software/lofarapr2016/opt/LofIm/lib/python2.7/site-packages:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib/python:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/::/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp', 'MANDATORY_PATH': '/usr/share/gconf/ubuntu.mandatory.path', 'LIBPATH': '/home/rvweeren/software/root/lib:', 'DYLD_LIBRARY_PATH': '/home/rvweeren/software/root/lib:', 'LOFARROOT': '/home/rvweeren/software/lofarapr2016/opt/LofIm', 'MANPATH': '/home/rvweeren/software/root/man:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/man:/usr/local/man:/usr/local/share/man:/usr/share/man', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'PATH': '/home/rvweeren/software/lofarapr2016/opt/LofIm/bin:/home/rvweeren/software/lofarapr2016/opt/LofIm/sbin:/home/rvweeren/software/root/bin:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rvweeren/software/casapy/casa-release-4.5.2-el6/bin:/home/rvweeren/software/WSClean/wsclean1.11/wsclean/build2/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6/bin:/home/rvweeren/software/WSClean/wsclean1.11/wsclean/build2/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6/bin:/home/rvweeren/software/WSClean/wsclean1.11/wsclean/build2/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools', 'LD_LIBRARY_PATH': '/home/rvweeren/software/lofarapr2016/opt/LofIm/lib:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib'} 2016-04-04 15:38:02 INFO node.lofar.python_plugin: Processing /p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/L343226_SBgr025-10_uv.dppp.pre-cal_chunk9_12656E813t_0g.wsclean_image_full2 2016-04-04 15:38:03 DEBUG facetimage_facet_patch_539.executable_args: Results for job 0 submitted by ('127.0.0.1', 45229) 2016-04-04 15:38:03 ERROR node.lofar.python_plugin: invalid literal for int() with base 10: '1.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.41.' 2016-04-04 15:38:03 INFO node.lofar.python_plugin: Total time 0.7424s; user time: 0.6028s; system time: 0.0784s 2016-04-04 15:38:03 DEBUG node.lofar.python_plugin: Start time was 1459798682.4986s; end time was 1459798683.2411s 2016-04-04 15:38:04 DEBUG facetimage_facet_patch_539.executable_args: Remote command stdout: size is 3150

2016-04-04 15:38:04 ERROR facetimage_facet_patch_539.executable_args: Remote process python /home/rvweeren/software/lofarapr2016/opt/LofIm/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py ['/p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/L343226_SBgr025-10_uv.dppp.pre-cal_chunk9_12656E813t_0g.wsclean_image_full2', '/usr/local/lib/python2.7/dist-packages/Factor-1.0-py2.7.egg/factor/scripts/pad_image.py', ['/p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539/L343226_SBgr025-10_uv.dppp.pre-cal_chunk9_12656E813t_0g.wsclean_image_full2', '1.4'], {}, '/p3600/rvweeren/P214_52_flagged/results/facetimage/facet_patch_539', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'SHLIB_PATH': '/home/rvweeren/software/root/lib:', 'MANDATORY_PATH': '/usr/share/gconf/ubuntu.mandatory.path', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', 'LOFARDATAROOT': '/opt/lofar/data', 'PYTHONPATH': '/home/rvweeren/software/lofarapr2016/opt/LofIm/lib/python2.7/site-packages:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib/python:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/::/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp', 'COMPIZ_BIN_PATH': '/usr/bin/', 'LIBPATH': '/home/rvweeren/software/root/lib:', 'DYLD_LIBRARY_PATH': '/home/rvweeren/software/root/lib:', 'LOFARROOT': '/home/rvweeren/software/lofarapr2016/opt/LofIm', 'MANPATH': '/home/rvweeren/software/root/man:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/man:/usr/local/man:/usr/local/share/man:/usr/share/man', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'PATH': '/home/rvweeren/software/lofarapr2016/opt/LofIm/bin:/home/rvweeren/software/lofarapr2016/opt/LofIm/sbin:/home/rvweeren/software/root/bin:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rvweeren/software/casapy/casa-release-4.5.2-el6/bin:/home/rvweeren/software/WSClean/wsclean1.11/wsclean/build2/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6/bin:/home/rvweeren/software/WSClean/wsclean1.11/wsclean/build2/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6/bin:/home/rvweeren/software/WSClean/wsclean1.11/wsclean/build2/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools', 'LD_LIBRARY_PATH': '/home/rvweeren/software/lofarapr2016/opt/LofIm/lib:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib'}] failed on localhost (status: 1) 2016-04-04 15:38:04 DEBUG facetimage_facet_patch_539.executable_args: compute.dispatch results job 0: job_duration: 2.03805088997, returncode: 1 2016-04-04 15:38:06 DEBUG facetimage_facet_patch_539.executable_args: Adding node_logging_information 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539.executable_args: All jobs failed. Bailing out! 2016-04-04 15:38:06 WARNING facetimage_facet_patch_539.executable_args: Note: recipe outputs are not complete 2016-04-04 15:38:06 WARNING facetimage_facet_patch_539.executable_args: recipe executable_args completed with errors 2016-04-04 15:38:06 WARNING facetimage_facet_patch_539: pad_image reports failure (using executable_args recipe) 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: Failed pipeline run: facet_patch_539 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: Detailed exception information: 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'> 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: pad_image failed 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: 2016-04-04 15:38:06 ERROR facetimage_facet_patch_539: LOFAR Pipeline finished unsuccesfully. 2016-04-04 15:38:06 WARNING facetimage_facet_patch_539: recipe facetimage_facet_patch_539 completed with errors

[rvweeren]

BTW This seems to be the padding of the model images for the FFT step (the padding in the wsclean imaging seems to work)

[darafferty]

Ah - this is due to the darn pipeline sending a string to the padding script instead of a float. I forget about this all the time. It should be fixed now.

[rvweeren]

It now works without any crashes. Still have to confirm the aliasing problems are reduced/gone (will have to calibrate a few more facets for that).

[rvweeren]

The extra padding in the facet imaging makes a big difference. It general looks it looks like you cannot trust anything within ~30% of an image edge, all kind of new (fake) sources/artifacts appear there.

Attached image shows the difference for a facet between the extra padding (left) and the old situation (right). I marked differences with green circles. These differences all appear around the image boundaries.

So my general conclusion is that one should not trust -anything- near image edges. Given how bad the situation is, the 1.2 padding we do for the facet imaging does not seem to be enough (btw because of rounding the padding was only ~1.15). I think we need to increase it to 1.6.

(we should also contact Andre to check if there is anything else we can do )

(the FFT subtract padding of 1.4 is probably ok, since it already works on top of the padding in the facet imaging)

[rvweeren]

Ok with a few more fields done I start to notice improvement with the aliasing (aliasing always occurs but since we have larger boundaries we are not affected by it so much anymore).

(left: new code with extra padding, right: old code without padding)

So I think the padding can be merged into the main branch, but with the facet imaging padding increased even more from 1.2 now to 1.6. Based on inspections of the image boundaries I see at least mirroring of sources up the a factor of 1.4 (if they are very bright). So 1.6 should be relatively safe.

[rvweeren]

Digging a bit deeper into the mirroring of sources. I think what we are seeing is not a an issue with wsclean, in the sense that this behavior is expected. According to the wsclean documentation:

WSClean can perform either nearest neighbour interpolation ("-gridmode nn") or 63x supersampling with by default a 7 pixel low-pass filter kernel with a Kaiser-Bessel function ("-gridmode kb"). The latter is the default, and more accurate. With NN, a sample is placed at the nearest uv-pixel. Especially for small images, that might create aliasing and inaccurate fluxes. The supersampling places samples more accurately on the uv-grid, giving slightly more accurate fluxes. A windowed low-pass filter attenuates aliased sources. Sources that are just outside the field of view might otherwise be aliased back into the image. A Kaiser-Bessel function is very similar, but is considered slightly less optimal compared to optimized prolate spheroidal functions that are e.g. used by Casa to window the low-pass filter. However, I found that a 7 pixel low-pass filter is not very strong anyway, and strong aliased off-axis sources will show up as ghosts in both Casa and WSClean images. The best method to deal with those is to increase the field of view; that will also allow proper cleaning of those. The effect of the windowed low-pass filter image can be outputted by specifying '-savegridding', which will output a file named like "wsclean-gridding.fits".

Using -savegridding you can see what happens with the windowed low-pass filer. Example show the windowed low-pass filter with the default gkernelsize 7 (right) and 17 (left). The red colored region is more or less the "safe" area, free from aliasing.

You can also see that aliased sources around the image edges are better suppressed when using -gkernelsize=17 (instead of the default 7). Question is what is more efficient in terms processing, increasing gkernelsize or padding (from the documentation it seems that Andre recommends the padding).

[mhardcastle]

Am I right in thinking that if you increase gkernelsize you would need a larger border round the image anyway? So we would need padding even if we increased gkernelsize?

I think this is a case for systematic experiment -- starting with your factor 1.6 for padding, what happens to (a) execution time and (b) image quality as you decrease padding and increase gkernelsize, where image quality includes both the presence of aliased sources and the reproduction of real sources near the image borders? What's the minimum execution time we can get away with?

I'd be happy to write some code to test this starting from your MS, but I'm away for a week starting tomorrow...

[Joshuaalbert]

I think the best way to solve this is with a simulation pipeline. Francesco de Gasperin and I discussed this and how a simple implementation would go miles in terms of usefulness of determining what is real and what is not. We plan to do it.

On 2016-04-08 10:46, Martin Hardcastle wrote:

Am I right in thinking that if you increase gkernelsize you would need a larger border round the image anyway? So we would need padding even if we increased gkernelsize?

I think this is a case for systematic experiment -- starting with your factor 1.6 for padding, what happens to (a) execution time and (b) image quality as you decrease padding and increase gkernelsize, where image quality includes both the presence of aliased sources and the reproduction of real sources near the image borders? What's the minimum execution time we can get away with?

I'd be happy to write some code to test this starting from your MS, but I'm away for a week starting tomorrow...

You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub [1]

*

Links:

[1] https://github.com/lofar-astron/factor/issues/40#issuecomment-207326390

[mhardcastle]

That's certainly the best solution in the long term, but can I ask on what timescale you plan to do it?

[Joshuaalbert]

We plan to work on it starting in May. Perhaps, end of April. There has already been some work in this direction so there are codes we can make use of. I think it should be useful second week of May.

On 2016-04-08 11:57, Martin Hardcastle wrote:

That's certainly the best solution in the long term, but can I ask on what timescale you plan to do it?

You are receiving this because you commented. Reply to this email directly or view it on GitHub [1]

*

Links:

[1] https://github.com/lofar-astron/factor/issues/40#issuecomment-207355792

[twshimwell]

Certainly a very nice thing would be to simulate right through the entire facet calibration pipeline but that is a big big project. Was that you had in mind with your simulations or were you intending to focus just on this aliasing issue?

I don't think it is worth waiting till May to further characterise this issue because essentially it seems that fiddling with gkernelsize and padding parameters can reduce it sufficiently to pretty much remove the aliasing and facilitate further factor commissioning.

[Joshuaalbert]

The intended idea is to take a model pass it through a phase screen and compute visibilities. Then running any calibration pipeline on the simulated measurements we can compare the products to the input model and see what is fake and what is real. Encompasses more that this aliasing issue clearly.

On 2016-04-08 12:21, twshimwell wrote:

Certainly a very nice thing would be to simulate right through the entire facet calibration pipeline but that is a big big project. Was that you had in mind with your simulations or were you intending to focus just on this aliasing issue?

I don't think it is worth waiting till May to further characterise this issue because essentially it seems that fiddling with gkernelsize and padding parameters can reduce it sufficiently to pretty much remove the aliasing and facilitate further factor commissioning.

You are receiving this because you commented. Reply to this email directly or view it on GitHub [1]

*

Links:

[1] https://github.com/lofar-astron/factor/issues/40#issuecomment-207362362

[mhardcastle]

Yep -- I agree with Tim, that should be done but is a big project and we should probably not delay the facet calibration testing while it happens.

A half-way house would be to inject numbers of simulated sources into a blank facet and then clean them back out again and look for aliasing effects. But even that may be more work (and CPU time) than we have time for right now.

[tammojan]

Please open a new ticket for this.

Tammo Jan

Op 8 apr. 2016 om 12:25 heeft Joshuaalbert notifications@github.com het volgende geschreven:

The intended idea is to take a model pass it through a phase screen and compute visibilities. Then running any calibration pipeline on the simulated measurements we can compare the products to the input model and see what is fake and what is real. Encompasses more that this aliasing issue clearly.

On 2016-04-08 12:21, twshimwell wrote:

Certainly a very nice thing would be to simulate right through the entire facet calibration pipeline but that is a big big project. Was that you had in mind with your simulations or were you intending to focus just on this aliasing issue?

I don't think it is worth waiting till May to further characterise this issue because essentially it seems that fiddling with gkernelsize and padding parameters can reduce it sufficiently to pretty much remove the aliasing and facilitate further factor commissioning.

You are receiving this because you commented. Reply to this email directly or view it on GitHub [1]

*

Links:

[1] https://github.com/lofar-astron/factor/issues/40#issuecomment-207362362

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub

[rvweeren]

Am I right in thinking that if you increase gkernelsize you would need a larger border round the image anyway? So we would need padding even if we increased gkernelsize?

If we increase gkernelsize we need less padding, but we always will need at least -some- padding.

[rvweeren]

To make sure we are on the same page. The aliasing we see is not a wsclean bug, it is just how the imaging works. (note that casa will also have these problems, they are just hidden as it internally pads by default, unlike wsclean). We just need to find a good combination of gkernelsize and trim (trim+imsize = padding).

For that reason I do not think we need to run a simulation pipeline (or wait for that) to deal with this.

I will ask Andre what he advices in terms of padding vs gkernelsize (i.e., what is better in terms of performance and accuracy). So we have already solved this problem (padding was added), it is just a matter what is most efficient in terms of processing time.

[rvweeren]

Answer from Andre below. So he suggests it is more efficient to use padding than increasing gkernelsize. (so what we have been doing, adding padding, is ok)

Increasing the gkernelsize should also make the noisy border smaller. The default size is 7 (equal to CASA). The number given needs to be odd -- maybe you specified an even number, which might not make it stronger (I've just added a check for 1.11 to throw an error if it's not odd). If you specify -gkernelsize 15 you should see a much smaller noisy border. However, it's way more (~4x) expensive if the nr of vis is large, so the trimming option might be better.

[rvweeren]

So after David increases the padding to 1.6 I think we can close this issue.

[mhardcastle]

Yeah, I guess even padding by 1.6 would not increase the execution time by that much. OK.

[darafferty]

OK, I've changed the default to 1.6 for imaging and left the default for the predict at 1.4.

[twshimwell]

I'm still getting aliasing issues, especially towards the edge of my facet and whilst the region outside the facet doesnt really matter I would still classify the aliasing in that region as pretty severe. Not sure if this is worth reopening the ticket for because the aliasing is so reduced from last time within the facet region. My reduction is all 244subbands, no extremely bright sources in the field (I think a few Jy max), good ionosphere and reaching a noise of ~100uJy/beam.

[darafferty]

I guess we need to understand how severe these issues are in the model predict (with its additional padding). Tim, can you identify any aliasing artifacts that appear inside a facet (from subtraction of earlier facets)?

[twshimwell]

This is the first facet. In the subtract pipeline images i did see pretty nasty aliasing (see https://github.com/lofar-astron/prefactor/issues/23) but this was in a region of sky which I have not yet facet calibrated. In the subtract pipeline images I cannot see any of aliasing features that I see within this image of the first facet.

[rvweeren]

Do not jude regions close to image borders as you expect aliasing there. That is not a bug, but reflects Andre's choice of not padding by default in WSClean. So -every- facet image made with wsclean should have aliasing, but -only- close to the image boarders (say within 20-30%). How far are these artifacts from the individual images boundaries ?

[twshimwell]

The aliasing I can see within the facet region (which is pretty faint and close to the edge of the facet) is about 80% of the way up the image. The closest one to the image centre I can see (which is outside the facet region) is about 75% of the way across the image. So in this case aliasing effects about half the image which agrees with the 20-30% boarders you mentioned. However, as there is aliasing within the facet region (only just within the region which is shown by the green boundary in the image I posted before) do we need to increase the padding further?

I guess for this facet, which is the first one, the aliasing is worst due to all the poorly subtracted structures.

[rvweeren]

Wondering what padding factor was used in your case? Initially we were doing 1.2, about a week ago, but then David increased it to 1.4 (as still saw signs if aliasing in the 1.2 factor padding). But maybe it needs to be increased even more. It is starting to become "painful" though in terms of CPU time, I already notice we spend much more time in wsclean now. Note that in the new reimage branch you can adjust the padding factors in the factor parset. So it would be interesting to increase it and run it again to see if it removes the aliasing.

[twshimwell]

In my logs I see:

padding to 10035 offset is 1433 size is 7168

So it looks like padding of 1.4 was used.

Yeh it does become very slow. I guess that if these artefacts do not get into the model and subtracted during the facet imaging then they will disappear in the reimaging step (as less residuals outside of facet and the padding can be tuned there anyway).

Maybe it is worth allowing for the padding to be set in the parset for the facet imaging (with a good default value of say 1.4)? Hopefully for most fields 1.4 would be fine but perhaps in special cases it can be increased or even decreased.

[rvweeren]

Maybe it is worth allowing for the padding to be set in the parset for the facet imaging (with a good default value of say 1.4)?

That option is available now (in the reimage branch)

[twshimwell]

Is that just for reimaging though? I was thinking of when the facets were first imaged.

[rvweeren]

No, not only for reimaging (that branch has quite a few new options unrelated to reimaging, for example spline amplitude smoothing, full correlation solve, etc., so the branch name is a bit misleading)

Padding factor for WSClean images (default = 1.6)

wsclean_image_padding = 1.6

Padding factor for WSClean models (default = 1.4)

wsclean_model_padding = 1.4

[twshimwell]

Oh excellent. Opps I missed that (I can see it in some text from David above now). I'll have a go with that branch. Thanks.

[rvweeren]

Note that the parset did change a bit because there are many new options (I think you cannot directly use your old parset).

(see the example in the example directory)