Extended Emission in calibrator image does not get subtracted

[AHorneffer]

I just noticed that in one of my facets there is a source that doesn't get fully subtracted. Which can be seen in the wsclean_pre and wsclean_post images below.

The wsclean_full_low image is from the low-resolution image of the facet, but with the clean-mask of the facet-calibrator as contours. Yes, I had to provide a hand-made region for the extended source at the right side. IMHO it is clear that the high-resolution image of the facet-calibrator did not pick up all the extended flux of that source.

My suggestion to solve this would be to also include the clean mask from the calibrator when looking for sources in the wsclean_full_low image.

[AHorneffer]

I did the fun exercise to print all *_post-image.fits of my already finished facets and concatenate them into an animation. The result is below. (The number at the start of the file name show the order of the facets.)

I guess it is fair to say that subtracting the sources in the facets doesn't work too well...

[AHorneffer]

And a similar animation with the facets overlay:

[rvweeren]

Nice overview of the problem. I suspect that the residuals are mainly the results of a sort of mismatch between the amplitude-scale of the core and remote stations, possible caused by the presence of diffuse emission (I've seen this happening before in other fields, usually in facets with relatively poor S/N for the facet calibrator).

The problem of a couple of negative sources that are appearing was probably caused by ignoring the negative clean components (that problem should have been fixed a few days ago).

[AHorneffer]

Sorry, yes, the negative sources are probably indeed mostly due to the fact that I run an older version of Factor.

But I'm pretty sure that the positive sources are not due to the difference in amplitude scale of core and remote stations: I see them also in the first (and thus strongest) facet, and I think that there are some fairly compact sources that don't get picked up by the mask. (I'll have to look into that more closely tomorrow.)

[rvweeren]

There is also the issue of interpolation and facet clean depth (because the facet images are made with only 6 blocks, that might also affect the accuracy of the subtraction, to what extent is unclear to me though....)

[darafferty]

The masking thresholds are indeed set fairly high and probably could be reduced. Also, the script that converts the model images to sky models (https://github.com/lofar-astron/factor/blob/master/factor/scripts/fits2sky.py) allows some residual flux to remain to reduce the number of clean components by ignoring the faintest ones (it allows up to 0.05 Jy per facet, which may be too high). So, we could try reducing the thresholds and the allowed max residual to see what effect they have.

[darafferty]

I adjusted the thresholds for the shallow facet images down quite a bit, so they should pick up more sources (and more extended emission). I also turned off multiscale clean for these images (it's still turned on as before for selfcal and the full-band images), since (as Reinout pointed out earlier), modeling extended emission with point sources doesn't work well (and gives a huge number of sources to predict). If we can get the Gaussians directly from WSClean, then we can re-enable the multiscale clean and get much better subtraction in these cases.

[AHorneffer]

After checking the code I noticed that the original issue (diffuse emission within the calibrator image) probably could have been solved if I had also given my hand-made mask for the full facet.

And somewhat belatedly I noticed that the more general problem was already mentioned in issue #164

I'll give the new version a try, and let you know how it works.

[darafferty]

I went ahead and also reduced the allowed amount of residual flux to 0, as it doesn't save much time in my tests (especially with the faster DPPP predict that's in the trunk as of last week).

[AHorneffer]

My Factor re-run shows that:

1. In general the post-image.fits looks much "cleaner" than the pre-image.fits.
2. Some negative features remain where negative clean-components were outside the mask.
3. When specifying a region-file for the clean mask for the full facet in the directions-file, then in the 'mask5' step the overlap with the calibrator-region is not removed.
4. (And my calibrator-radius is/was too small, so that part of the extended structure lay outside the calibrator region.)

In the image below there is:

• the last calibrator image (image42_iter1) as a raster-image (in greyscales)
• the mask from the calibrator imaging (mask4) in white contours
• the mask from the wsclean_image_full(mask5) in red contours
• the mask from the wsclean_image_full_low(mask6) in green contours
• the clean region that I supplied for the calibrator and full-facet in pink

The image is at the right edge of the calibrator region. The border of the "90%" region within which the calibrator should be is at about a third from the right side of the image.

The way I understand the logic, the white contours (mask4) should not contain the right part of the given clean region (for the calibrator) because the initsubtract sources were not added there. And the red contours (mask5) should not contain the left part of the given clean region (this time given for the full-facet) because that part has (or should have been) already been subtracted when imaging the calibrator. This latter part means that other, point-like sources (not shown here) are actually subtracted twice and show up as negative sources in the wsclean_image_full_low image. (Where they are then added again, so the end effect isn't too bad. But still...)

P.S. I'll re-do this facet next week when I'm a) not travelling and b) hopefully better rested.

[darafferty]

I rearranged the masking steps in make_clean_mask.py, so now the step that adjusts the mask for the calibrator and facet regions is after the step that adds the user-supplied region. This should fix the issues with mask4 and mask5 above.

[AHorneffer]

Thanks. I'll have a look at it next week.

[AHorneffer]

O.K. it looks good now. (There is still about 20 mJy flux from the extended source in the post image, but that seems to be the flux that wasn't cleaned away in the wsclean_image_full_low image.)

The only issue that I still see is the one I mentioned in #180 about the negative clean components outside the mask.