Current Status

[piyanatk]

Since we fork from https://github.com/ronniyjoseph/corrcal2, where parts of the codes have already been migrated to Python 3, we should ask @ronniyjoseph to give us an update on the current status before planing the next step and delegating the works.

[ronniyjoseph]

So corrcal2.py contains a whole load of corrcal related functions. Installing is fairly easy, all packackages required are on pip or conda. But that should go into the README

However the original example script only uses all functions related to:

• get_gradient
• get_chisq
• sparse_2level

In addition I have used all functions related:

• grid_data (can me moved to some utility module)

So part of the restructuring implies:

• What functions to do we need in corrcal?

• Which functions can be moved out of the module

• Make the c-routines more readable (luckily, the code is fairly short)

• I've not added documentation (sorry, but I can add descriptions)

• Add some standardized covariance matrix eigenvector decomposition (This is array specific and in principle only has to be done once, but it wouldn't hurt to standardize it)

• Build in a default sky model covariance function

• Repeat some of @Mdlalose's results with the python 3 version.

[piyanatk]

@Mdlalose @Zahra-Kader Do you have anything else to add? What kind of tests and usages you have done so far?

[ronniyjoseph]

I've only ran it on simulated data from an array with either an MWA-like lay-out or SKA-LOW1 to see if it converges at all, I have looked at amplitude solutions but didn't have the time to implement the phase offset metric mentioned in Sievers 2017.

[Zahra-Kader]

Hi, I think Ronniys suggestions on restructuring is really good. I don't have any new points to add, but I just want to add to the point on including a default sky model covariance model. @Mdlalose has told me that he uses bright point sources from the GLEAM catalogue to construct the sky covariance matrix. Maybe we could include implement what he has done directly into corrcal and use catalogues for different frequencies. GLEAM isn't in the right frequency range for HIRAX, so we would have to use another source catalogue. I haven't been using point sources, but have used a simple sky covariance matrix constructed from the 21 cm signal. I get this straight from 'driftscan' in Richard Shaw's radiocosmology packages.

[ronniyjoseph]

@Zahra-Kader, brilliant. Yes, a somewhat flexible point source sky model covariance matrix would be nice.

I do think that generating sky model visibilities itself would fall into somewhere in between the "core" of corrcal and something as a instrument plug-in, because it requires some instrumental model (lay-out, beam model). But something that ensures visibilities get into corrcal in the same format would be desirable.

Also @piyanatk, I was hoping to start adding some docstrings of what I have learned while playing around with corrcal. How would you like to go about that. They won't be of Radio Astronomy Software standard (yet)

[piyanatk]

somewhat flexible point source sky model covariance matrix would be nice. But something that ensures visibilities get into corrcal in the same format would be desirable.

I agree with @ronniyjoseph here. I think the most we should do is developing a utility tool or a plugin for a calculating covariance matrix from visibility. Obtaining the visibility should be responsibilities and choices of the users, but we should provide the calculation tool since the the covariance matrix is a major parameter of corrcal.

Also @piyanatk, I was hoping to start adding some docstrings of what I have learned while playing around with corrcal. How would you like to go about that.

@ronniyjoseph, see issue #5 for this

[Mdlalose]

Hi everyone, sorry I couldn't reply earlier. This what I have done so far: For my master thesis, We have run corral 8x8 array simulation using a Gaussian beam with FWHM deviated by 0.05 for each antenna and used GLEAM Catalogue to simulate visibility. Then are put a scatter of 20% for gain amplitude and phase. We computed an analytical form of covariance in Fourier space using the same Gaussian beam.  We run corral for the two cases a) with identity visibility noise, covariance, initial gain with scatter of 20%  and visibility data and on the second case, We weight all GLEAcatalogue point source fluxes bGaussianna beam and then select top ten top 10 to use compute outerproduct of the visibility, to add SS^T in effective noise covariance.  We then compare this corrcal gain with a traditional redundant baseline, with a solution find through minimizing linearized chi-square.  To tease out the difference between the two calibration schemes, we computed autocorrelation of gain in frequency axis, this will allow us to see if ignoring non-identities due to beam variation does add gain spectral errors which are on the scale of interest i.e ~10MHz where we expect 21 cm signal to correlated or regions where structure is expected to form during EOR.

Now, we working on test corrcal by simulating HERA-like array and looking at different non-identities such as feed displacement, feed disorientation and dish deformation and surface roughness, and finaly, see if these non-identity are properly mitigated how they go affected the linkage of noise to EOR window. 

[Mdlalose]

@ronniyjoseph I have covariance matrix ready that I using, but this in spherical space, it take a dictionary with grouped baselines are to redundant sets, actual antennas positions, antenna beams, and the c_l of the expexted signals at the time of observations.

[ronniyjoseph]

@Mdlalose, do you still have the code lying around that you used to generate this, it might provide as a nice starting point for a general HIRAX/HERA/..... sky model generating module

[piyanatk]

@Mdlalose, that is definitely something that we could implement in a utility module. My only opinion is that we should try to use pyuvdata as an interface of visibility, beam, and etc.