Open Victoria-Samboco opened 1 year ago
@Victoria-Samboco if you used QuartiCal for the calibration and asked it to write out residual visibilities you can use surfvis
https://github.com/ratt-ru/surfvis
to locate bad data. Have a look at the surfchi2 worker eg.
$ surfchi2 --help
Usage: surfchi2 [options] msname
Options:
-h, --help show this help message and exit
--rcol=RCOL Residual column (default = RESIDUAL)
--wcol=WCOL Weight column (default = WEIGHT_SPECTRUM). The special
value SIGMA_SPECTRUM can be passed to initialise the
weights as 1/sigma**2
--fcol=FCOL Flag column (default = FLAG)
--dataout=DATAOUT Output name of zarr dataset
--imagesout=IMAGESOUT
Output folder to place images. Saved in CWD/chi2 by
default.
--nthreads=NTHREADS Number of dask threads to use
--ntimes=NTIMES Number of unique times in each chunk.
--nfreqs=NFREQS Number of frequencies in a chunk.
--use-corrs=USE_CORRS
Comma seprated list of correlations to use (do not use
spaces). Default = diagonal correlations
It will produce a directory tree with plots like this one
The numbers on the left and bottom of the image on the left are antenna numbers and the histogram on the right is a histogram of the average chi-square. The outliers with large chi-square should probably be flagged (eg. in the above example you can see that there is bad data affecting the baseline defined by antenna 52 and 29. You can flag it with the flagchi2 worker
$ flagchi2 --help
Usage: flagchi2 [options] msname
Options:
-h, --help show this help message and exit
--rcol=RCOL Residual column (default = RESIDUAL)
--wcol=WCOL Weight column (default = WEIGHT_SPECTRUM). The special
value SIGMA_SPECTRUM can be passed to initialise the
weights as 1/sigma**2
--fcol=FCOL Flag column (default = FLAG)
--flag-above=FLAG_ABOVE
flag data with chisq above this value (default = 3)
--unflag-below=UNFLAG_BELOW
unflag data with chisq below this value (default =
1.15)
--nthreads=NTHREADS Number of dask threads to use
--nrows=NROWS Number of rows in each chunk (default=10000)
--nfreqs=NFREQS Number of frequencies in a chunk (default=128)
--use-corrs=USE_CORRS
Comma seprated list of correlations to use (do not use
spaces)
--respect-ants=RESPECT_ANTS
Comma seprated list of antennas to respect (do not use
spaces)
Let me know if you want to try it out then I can give you a hand
With the results that I obtained through the SELF-CALIBRATED data, I believe it is no longer necessary to go through the peeling process, judging by the images of the Sun obtained now (from the selfcal data) - (Please correct me if I'm wrong). But of course I'm open to experimenting to see what it can do.
I will plot the visibilities of the problematic scans on the post-peel data but now for the self-calibrated data for comparison.
Hi everyone, here are the notes for the today meeting:
We first show the flags from 1GC I had to redo (to reduce the percentage of flags), and the flagging have reduced significantly from
The flaggings were reduced changing the flagger from
~96%
to an average of30%
as we can see below:aoflagger
toTricolour
and usinggorbachev.yaml
strategy. Some flags as for 2 problematic scans (23
and45
) and a problematic channel ('*:1007.9~1100.0MHz'
) were added.Proceed to Selfcal and DD-cal - after reducing the percentage of flagged data, the selfcal solution proved to be better than the selfical before (with![Screenshot from 2022-10-05 18-41-12](https://user-images.githubusercontent.com/95472901/194341776-2f3bb070-fecb-4e99-a73b-6b0de861376b.png)
96%
flagged, which led to the need to do DD-cal). What made me think that in the current situation it is not necessary to do DD-cal (concluded after comparing selfcal and dd-cal images which proved to be worse than the self-calibrated image). Bellow are the Self-calibrated data with96%
of data flagged and now (with30%
of data flagged) respectively :Comparison between the actual self-calibrated and dd-calibrated data (as we can see bellow the selfcal image are better than the dd-cal data)![image](https://user-images.githubusercontent.com/95472901/194349151-82f6ac03-16d4-4550-8baa-11b9eabeb792.png)
Scan 7 Amplitude vs Frequency in time.![image](https://user-images.githubusercontent.com/95472901/194350896-b5045659-bf80-4039-9ac3-6d4660247150.png)
scabs 11-17
scans 19 - 25
Scan 25 Amplitude vs Frequency in time.![image](https://user-images.githubusercontent.com/95472901/194351068-09fd07b9-140e-4c9f-85a0-b3949c0539b1.png)
And scans 29 - 35![scan29t035](https://user-images.githubusercontent.com/95472901/194343727-e5d62dc9-669e-4764-aa67-9da6598fc036.png)
Next steps after what was discussed: