GRS1747-312 dataset

[Victoria-Samboco]

Greetings all As what was discussed by Prof. @IanHeywood (in #2 ), here are the images of the 1GC and the Self-calibrated data as well as the image of the Sun from this observation. We can see from the Sun image the very active regions, sunspots.

This first image is from prof Ian, from it we can see sidelobes most prominently on the right of the map as well as the Sgr A, the Galactic centre on top of it. Those sources in the circle are not real, they are actually sunspots being aliased into the image.

This are the 1GC and SSelfcal image that I got from the same dataset. Here is not possible to see the sunspots and the Sgr A probably it because of the size of the image it its not piking it, so I'm imaging it again with a bigger size (10240, hope it is enough :)).

This is the image of the Sun from this observation.

[o-smirnov]

Lovely stuff!

Now you could take the clean component list (run wsclean with -save-source-list, if you haven't already), use crystalball to predict it into a separate column (say, SUN_MODEL_DATA), and see if you can't use QuartiCal to peel it away? @IanHeywood what do you reckon, workable plan?

[Victoria-Samboco]

okay, I will do it.

[Victoria-Samboco]

I've already found a solution for the problem with the source list. I had to set niter to 1 instead of 0.

On Wed, 25 Jan 2023, 19:21 Oleg Smirnov, @.***> wrote:

Lovely stuff!

Now you could take the clean component list (run wsclean with -save-source-list, if you haven't already), use crystalball to predict it into a separate column (say, SUN_MODEL_DATA), and see if you can't use QuartiCal to peel it away? @IanHeywood https://github.com/IanHeywood what do you reckon, workable plan?

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/solarkat/issues/20#issuecomment-1403967787, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWYM2BPQJLCFIUD2UUEMS7DWUFOKBANCNFSM6AAAAAAUGRQSSA . You are receiving this because you authored the thread.Message ID: @.***>

[o-smirnov]

Well, you probably want to try some cleaning -- niter 1 will not clean very deeply at all!

[IanHeywood]

The images look great, however I still think the elongated sunspot complex is being aliased into your map at the very top right, and it probably ending up in the selfcal model. This prospect for aliasing is another nasty way the Sun could ruin things, and one I hadn't really considered when we were thinking about this project at the start.

@Victoria-Samboco what are your image size and cell size?

I agree that cleaning / peeling would be a good test. Maybe consider making a mask for the sunspots to constrain the deconvolution. Model image interpolation with smops might also be a good thing to try, it might be faster than crystalball if you end up with a lot of clean components.

[Victoria-Samboco]

The image size is 6076. Now Im running the image after the Sun peeling step to see what happens.

On Thu, Jan 26, 2023 at 2:37 PM IanHeywood @.***> wrote:

The images look great, however I still think the elongated sunspot complex is being aliased into your map at the very top right, and it probably ending up in the selfcal model. This prospect for aliasing is another nasty way the Sun could ruin things, and one I hadn't really considered when we were thinking about this project at the start.

@Victoria-Samboco https://github.com/Victoria-Samboco what are your image size and cell size?

I agree that cleaning / peeling would be a good test. Maybe consider making a mask for the sunspots to constrain the deconvolution. Model image interpolation with smops might also be a good thing to try, it might be faster than crystalball if you end up with a lot of clean components.

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/solarkat/issues/20#issuecomment-1404941871, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWYM2BKTRTIQMTPDUOF4EALWUJVXDANCNFSM6AAAAAAUGRQSSA . You are receiving this because you were mentioned.Message ID: @.***>

[landmanbester]

Are you seeing this level of aliasing with the wgridder enabled?

[Victoria-Samboco]

Are you seeing this level of aliasing with the wgridder enabled?

Well, no. This is what I ran to generate the image

 wsclean -name img_sun/sun -data-column CORRECTED_DATA -size 10240 10240 -channels-out 8 -niter 0 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -subtract-model -auto-threshold 1 -pol I -padding 1.3 -nwlayers-factor 1 -no-update-model-required -auto-mask 3 -temp-dir 1671435077_sdp_l0_1024ch_GRS1747-312.ms

I didn't have the wgridder enabled...

[landmanbester]

Try with -use-wgridder, that should get rid of the aliasing

[Victoria-Samboco]

Try with -use-wgridder, that should get rid of the aliasing

Oky

[Victoria-Samboco]

Try with -use-wgridder, that should get rid of the aliasing

@landmanbester Here are the images of the Sun before(left) and after(right) -use-wgridder enabled, doesn't seem to be showing any improvements in terms of aliasing.

[IanHeywood]

@Victoria-Samboco the aliasing occurs when the Sun is out of the field of view of the image that you are making but a ghost image of it appears inside the image away from its true position. You can see this in the very first post in this issue, highlighted by the circle, where the sunspot pattern is visible.

You can also see the sunspots aliased into the image that you made here:

The different image sizes and pixel sizes will cause the aliased image to move around.

@landmanbester note that the image I made at the very top of the thread was using the w-gridder, and the aliasing is still a big problem. I think image/cell sizes and padding parameters need tweaking to remove the aliasing.

[landmanbester]

That is a bit surprising. The w-gridder should compute the required padding and kernel support to suppress aliases up to the precision it is invoked with so we should report an issue if this is really the case. It may be that the default accuracy in wsclean is not sufficient. Let me see if I can reproduce with pfb-clean. Just to confirm, it's this one

/home/ianh/solarkat/1671435077_sdp_l0_1024ch_GRS1747-312.ms

on nash right? CORRECTED_DATA column? @IanHeywood can you recall the imaging parameters you used for the first image?

[IanHeywood]

That's the MS, I think the DATA column is the one to use. Here's the wsclean command:

wsclean -log-time -abs-mem 225 -parallel-reordering 8 -name /scratch3/users/ianh/sun/IMAGES/img_1671435077_sdp_l0_1024ch_GRS1747-312.ms_datablind -save-source-list -data-column DATA -field 0 -size 16384 16384 -scale 1.6asec -use-wgridder -no-update-model-required -weight briggs -0.3 -parallel-deconvolution 2560 -niter 80000 -gain 0.15 -mgain 0.9 -channels-out 8 -fit-spectral-pol 4 -join-channels -circular-beam -threshold 1e-06 1671435077_sdp_l0_1024ch_GRS1747-312.ms

[Victoria-Samboco]

@Victoria-Samboco the aliasing occurs when the Sun is out of the field of view of the image that you are making but a ghost image of it appears inside the image away from its true position. You can see this in the very first post in this issue, highlighted by the circle, where the sunspot pattern is visible.

You can also see the sunspots aliased into the image that you made here:

The different image sizes and pixel sizes will cause the aliased image to move around.

@landmanbester note that the image I made at the very top of the thread was using the w-gridder, and the aliasing is still a big problem. I think image/cell sizes and padding parameters need tweaking to remove the aliasing.

Oh okay, yes I can see it. But I'm wondering why the ghost image in circle does not appear in the images on top, but it appears in the images in the bottom. Basically the difference between them are the image sizes. 10000x10000 for the images on top and 6076x6076 for the images in the bottom.

I'm thinking that maybe the difference in image size could be a factor affecting the appearance of the ghost image in the circle. Knowing that larger images have more pixels and contain more information, which can result in increased resolution and clarity. In contrast, smaller images have fewer pixels and can result in lower resolution and decreased clarity. Analysing this images I think the ghost image in the circle might be more apparent in the smaller images because they contain less pixels, and therefore less information to obscure the ghost image.

[landmanbester]

I have some hope that the aliasing can be removed by upping the gridder accuracy but we'll have to wait for the deconvolution to finish to confirm. Here are MFS dirty images with pfb-clean (left) and wsclean (right)

The wsclean image I got using the command @IanHeywood posted above. I made the pfb-clean image by specifying a 3deg field of view with an oversampling factor of 2 w.r.t. Nyquist and a gridder accuracy of 1e-7. It's maybe a bit difficult to see from the naturally weighted image but even with the much smaller fov I don't see any aliasing. I will post the model when I have it and will also try with identical imaging parameters to confirm that's where it is coming from. @IanHeywood did you say there is a wgridder-accuracy parameter in wsclean now? I don't see it in the version I am using

[IanHeywood]

Allegedly there is, and you even guessed its name correctly :)

-wgridder-accuracy

https://wsclean.readthedocs.io/en/latest/wgridding.html

Did your pfb-clean run match both the image size and the pixel size of the wsclean image? I think changing either of those will change the position of aliased sources in the image.

[landmanbester]

Did your pfb-clean run match both the image size and the pixel size of the wsclean image? I think changing either of those will change the position of aliased sources in the image.

No, they didn't and yes the location of the aliased source will depend on them. I figured it would show up somewhere so let's see. I'll try with identical settings after I see the result of the deconvolution. I didn't want to wait for a 32kx32k PSF, the double sized PSF being a requirement of the deconvolution algorithm I am testing

[IanHeywood]

We could also cook up a residual set of visibilities that just includes the Sun (+ Sgr A?) to make these tests easier, save time on the deconvolution to go looking for the aliases.

[landmanbester]

I would say yes if I wasn't so sure they shouldn't be there if you do the gridding accurately enough. Maybe a good experiment to test what kind of accuracy we need. I actually didn't realise that the aliases showed up in the MFS dirty image already, otherwise I would have just made those from the start. Here is the MFS dirty with the same image and cell sizes

Not exactly equivalent because, despite my best efforts, I can't get the weighting to match up with wsclean's. I don't think there are any aliases in there though. Can you see anything? Here is a side by side

Not sure if that is a hint of a sunspot or a real source but we could always up the gridding accuracy to find out.

[landmanbester]

That blighter! I suspect we can get rid of them by using a very conservative form of Clark clean. I'm gonna have to give this a go. Might take a while though

[Victoria-Samboco]

Is the Sun the source on top of the image ?

On Mon, Feb 6, 2023 at 7:42 PM Landman Bester @.***> wrote:

[image: image] https://user-images.githubusercontent.com/16836765/217043349-ebcf7046-ebbc-4889-b550-42eb4f8997ed.png

That blighter! I suspect we can get rid of them by using a very conservative form of Clark clean. I'm gonna have to give this a go. Might take a while though

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/solarkat/issues/20#issuecomment-1419484164, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWYM2BOHUO6WKK2F4TK3JKDWWEZW5ANCNFSM6AAAAAAUGRQSSA . You are receiving this because you were mentioned.Message ID: @.***>

[IanHeywood]

Yes, and the other strong source about a third of the way towards the Sun is the Sgr A complex at the Galactic centre. The centre of the solar system and the centre of the galaxy in one glorious but slightly problematic MeerKAT observation!

[Victoria-Samboco]

The wgridder-accuracy is returning an error

2023-02-07 12:38:02 STIMELA ERROR: pre-validation of recipe 'boom' failed                                                                                       
──────────────────────────────────────────────────────────────── detailed error report follows ─────────────────────────────────────────────────────────────────
        ⚠ pre-validation of recipe 'boom' failed                                                                                                                
        └── recipe 'boom': 2 errors                                                                                                                             
            ├── step 'image-1' failed prevalidation                                                                                                             
            │   └── unknown parameter 'boom.image-1.wgridder-accuracy'                                                                                          
            └── step 'image-2' failed prevalidation                                                                                                             
                └── unknown parameter 'boom.image-2.wgridder-accuracy'

I'm running it from this recipe step, what could be the cause?

image-1:
      info: "auto-masked deep I clean"
      _use: lib.steps.wsclean.image
      params: 
        column: DATA
        niter: 100000
        temp_dir: temp_dir 
        save-source-list: false
        use-wgridder: true
        wgridder-accuracy: 1e-6

[IanHeywood]

Again there are large gaps in my stimela knowledge, but maybe the version of wsclean it's calling is too old...

The w-gridder is available since WSClean version 2.9, and was further improved in speed in versions 2.10 and 3.0.

[IanHeywood]

Do you know which container it's calling (if that is what it's doing)?

[landmanbester]

I think the -wgridder-accuracy parameter was added later. I think it's just calling the native wsclean which may be too old. We can probably ask the sys admins to upgrade the box you are working on @Victoria-Samboco. If you tell me which box you are working on I can add a request on the ratt-ru systems repo

[Victoria-Samboco]

I think the -wgridder-accuracy parameter was added later. I think it's just calling the native wsclean which may be too old. We can probably ask the sys admins to upgrade the box you are working on @Victoria-Samboco. If you tell me which box you are working on I can add a request on the ratt-ru systems repo

I'm working on garfunkel and my wsclean version is the version v3.1

[Victoria-Samboco]

Oh, wait. Inspecting the stimela output makes me think that the wsclean cab doesn't have the wgridder-accuracy parameter. Can you point me to your cab definition?

/home/samboco/lib/stimela/omstimelation/oms-cabs.yml on garfunkel

[Kincaidr]

Oh, wait. Inspecting the stimela output makes me think that the wsclean cab doesn't have the wgridder-accuracy parameter. Can you point me to your cab definition?

Yes i don't think this option is added

[Victoria-Samboco]

I know Oleg changed the syntax from _ to - , so you might want to use: use_wgridder instead of: use-wgridder for the stimlea2 version you are using.

oky, let me check...I updated wsclen to version 3.2 still returning the same error

[IanHeywood]

Can you just run it directly until the parameter is exposed to the stimela config?

[landmanbester]

/home/samboco/lib/stimela/omstimelation/oms-cabs.yml on garfunkel

Ok, I don't see the parameter. You can double check by running

$ stimela help /path/to/oms-cabs.yml wsclean

It's simple enough to add it. Or just run locally for now as @IanHeywood suggests

[Victoria-Samboco]

Can you just run it directly until the parameter is exposed to the stimela config?

samboco@garfunkel:~/solarkat/SUN_IMAGING_STEPS$ wsclean -name /obs2/im1/im1 -data-column DATA -size 16300 16300 -channels-out 8 -niter 100000 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -save-source-list  -fit-spectral-pol 4 -auto-threshold 20 -pol I -padding 1.3 -nwlayers-factor 1 -gridder -wgridder-accuracy 1e-6  -temp-dir temp_dir 1671435077_sdp_l0_1024ch_GRS1747-312-1.ms/
+ + + + + + + + + + + + + + + + + + +
+ An exception occured:
+ >>> Invalid gridder requested: '-wgridder-accuracy'
+ + + + + + + + + + + + + + + + + + +

It its not working even...

(stimela_env) samboco@garfunkel:~/solarkat/SUN_IMAGING_STEPS$ wsclean -name /obs2/im1/im1 -data-column DATA -size 16300 16300 -channels-out 8 -niter 100000 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -save-source-list  -fit-spectral-pol 4 -auto-threshold 20 -pol I -padding 1.3 -nwlayers-factor 1 -use-wgridder -wgridder-accuracy 1e-6 -auto-mask 3 -temp-dir temp_dir 1671435077_sdp_l0_1024ch_GRS1747-312-1.ms/                                    !!! WARNING: Parameter '-use-wgridder' is deprecated and will be removed in a future version of WSClean.
!!!          Use parameter '-gridder' instead.

WSClean version 3.2 (2022-10-21)
This software package is released under the GPL version 3.
Author: André Offringa (offringa@gmail.com).

+ + + + + + + + + + + + + + + + + + +

[Victoria-Samboco]

stimela help /path/to/oms-cabs.yml wsclean

Yes, the -wgridder-accuracy parameter is not there

[landmanbester]

The parameter names have probably changed in the latest version. You can probably find the correct name by looking at the output of

$ wsclean

[Kincaidr]

This is an unrelated problem, but I am getting this error in Wsclean that I have not encountered before:

# + + + + + + + + + + + + + + + + + + +                                                                                 

# + An exception occured:                                                                                               

# + >>> FilebufIO::readBlock - incorrect number of bytes read for file                                                  

/net/sinatra/vault-ike/kincaid/Zwcl2341_meerkat_2019.06.12/msdir/field1_fixvis-ZwCl2341_1_p_0000-corr.ms/table.dat

[IanHeywood]

+ + + + + + + + + + + + + + + + + + +
+ An exception occured:
+ >>> Invalid gridder requested: '-wgridder-accuracy'
+ + + + + + + + + + + + + + + + + + +

@Victoria-Samboco did you also enable -use-wgridder?

I have a container that I run stuff on nash with that has a recent version installed if you want to run it using singularity.

Singularity> wsclean | grep wgrid
-use-wgridder
-wgridder-accuracy <value>

[Victoria-Samboco]

+ + + + + + + + + + + + + + + + + + +
+ An exception occured:
+ >>> Invalid gridder requested: '-wgridder-accuracy'
+ + + + + + + + + + + + + + + + + + +

@Victoria-Samboco did you also enable -use-wgridder?

I have a container that I run stuff on nash with that has a recent version installed if you want to run it using singularity.

Singularity> wsclean | grep wgrid
-use-wgridder
-wgridder-accuracy <value>

I succeeded to run it directly from the command line using

 wsclean -name obs2/im1/im1 -data-column DATA -size 16300 16300 -channels-out 8 -niter 100000 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -save-source-list  -fit-spectral-pol 4 -auto-threshold 20 -pol I -padding 1.3 -nwlayers-factor 1 -gridder wgridder -wgridder-accuracy 1e-6  -temp-dir temp_dir 1671435077_sdp_l0_1024ch_GRS1747-312-1.ms/

Here is the first restored image.

It appears with the sunspots being aliased into the image, but now in a different position. These are images before selfcal.

[o-smirnov]

Folks, if the upstream cab is missing that parameter, you can add it yourself in your recipe temporarily, it's as simple as putting:

cabs:
  wsclean:
    inputs:
      wgridder-accuracy:
        dtype: float

at the top of your recipe, and that's it.

@Kincaidr I'm afraid your MS is corrupted. Sometimes that happens when a process writing to it is killed at just the wrong moment. I have never discovered a way to recover from this -- regenerating it is the only way I know.

@Victoria-Samboco the aliasing position will of course depend on the image size, if you think carefully about what the FFT does...

[IanHeywood]

Indeed, the imaging parameters are different from mine which explains the moving aliases. The main issue here is that @landmanbester thinks the alising should vanish when running the wgridder at its 1e-6 precision limit but there they are...

@Victoria-Samboco can you please check that using -nwlayers-factor 1 -gridder wgridder hasn't automatically invoked the default w-stacking gridder? My version of wsclean (3.1.0) doesn't have a -gridder option, to use the w-gridder the -use-wgridder switch must be provided.

[Victoria-Samboco]

Indeed, the imaging parameters are different from mine which explains the moving aliases. The main issue here is that @landmanbester thinks the alising should vanish when running the wgridder at its 1e-6 precision limit but there they are...

@Victoria-Samboco can you please check that using -nwlayers-factor 1 -gridder wgridder hasn't automatically invoked the default w-stacking gridder? My version of wsclean (3.1.0) doesn't have a -gridder option, to use the w-gridder the -use-wgridder switch must be provided.

According to what it says here https://wsclean.readthedocs.io/en/latest/wgridding.html ( Therefore, many parameters accepted by wsclean’s w-stacking gridder (e.g. -padding, -nwlayers*, -grid-mode, -kernel-size and -oversampling and -parallel-gridding) will be ignored in that mode), it doesn't seem to me that the default w-staking gridder has been invoked, it says that the parameters referring to it are automatically ignored when the -use-gridder/-gridder is enabled. .

[IanHeywood]

But wsclean doesn't have a -gridder parameter as far as I can see. From that page "W-gridding is enabled by the command-line flag -use-wgridder". So I don't know what the effect of specifying -gridder wgridder would be, and I'm worried it just uses the default.

[o-smirnov]

From the message I assume Andre wants us to switch to -gridder wgridder as opposed to -use-wgridder in the future, but the latter way is supposed to work for now.

-gridder -wgridder-accuracy as per @Victoria-Samboco's command line is incorrect either way...

[IanHeywood]

I missed the memo there. So -gridder wgridder -wgridder-accuracy 1e-6 should work?

If so then we're stuck with the aliasing with wsclean.

[landmanbester]

I'm pretty sure there aren't any aliases in the versions I am making with pfb-clean. These use a gridder accuracy of 1e-7 and double precision. I will test with an accuracy of 1e-6 using single precision to see if I can reproduce. There is also a double-precision-accumulation option in ducc, maybe check if that is exposed in wsclean?

[Victoria-Samboco]

From the message I assume Andre wants us to switch to -gridder wgridder as opposed to -use-wgridder in the future, but the latter way is supposed to work for now.

-gridder -wgridder-accuracy as per @Victoria-Samboco's command line is incorrect either way...

I've already added it to the recipe, I'ts working now

[landmanbester]

Well this is bizarre, I can't seem to reproduce the aliasing even with single precision gridding with an accuracy of 1e-4 (no double precision accumulation either) which should match what wsclean is doing. I've run the comparison using multiple robustness values and I consistently see aliases in wsclean's images (except when using natural weighting where the aliases are not evident in either) and no aliases in those produced by pfb-clean (the wgridder in pfb-clean anyway). I even tried switching off the small inversion in wsclean which should really mean we are comparing apples to apples. I suspect something weird is going on inside wsclean. @o-smirnov can you maybe run this past Andre while you have him there at BASP?

[o-smirnov]

Hmmm -- same gridder of course, same gridding kernel? Could you give me two side by side images I can show to Andre?

[landmanbester]

I'm having issues getting the weighting schemes to match up and since the aliases don't show up using natural weighting it's a bit difficult but these two are pretty close (they are saturated to the same level but not scale matched)

Left is wsclean briggs 0 and on the right is pfb-clean with briggs 0.3 and a gridding accuracy of 1e-4 (single precision and no double accumulation). You can see the alias in the south east quadrant in the image on the left