Copyright (C) 2013-2021 Cyril Tasse, l'Observatoire de Paris, SKA South Africa, Rhodes University
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
The Direction Dependent calibration problem (also known as third generation calibration) naturally arises in the Radio Interferometry Measurement Equation (RIME, Hamaker et al. 1994), but only became overwhelmingly problematic with the construction of the SKA precursors and pathfinders. Solving for the DDE calibration problems basically consists in inverting a number of non-linear equation, while the system is (i) very large and (ii) often subject to ill conditioning. killMS's name originates from the early LOFAR commissioning phases, when understanding the interferometric data in a Measurement Set was a real challenge.
killMS implements two very efficient algorithms for solving the Direction-Dependent calibration problem. The current status of the software and use cases (as well as connection with other softwares such as DDFacet) are summarised in
http://www.astron.nl/lowfrequencyobserving2017/Documents/Wednesday/LFO2017_Tasse.pdf
The two algorithms (CohJones and Kafka) are based on complex optimisation techniques. They use the properties of the complex ("Wirtinger") Jacobian to exploit algorithmic shortcuts. The basics of the application of the Wirtinger Jacobian and Hessians to the RIME, as well as the related implemented algorithms are described in
Tasse 2014: https://arxiv.org/abs/1410.8706
Smirnov & Tasse 2015: https://arxiv.org/abs/1502.06974
killMS also runs an extended Kalman filter that uses the Wirtinger (half) Jacobian (to be published, a similar approach is described in https://arxiv.org/abs/1403.6308)
Preliminary documentation is given below.
To build from source:
virtualenv myvenv
source myvenv/bin/activate
(myvenv)$ pip install DDFacet # installs the latest DDF release from PyPI
(myvenv)$ pip install <path to checked out killMS>
If you want to run in development mode:
(myvenv)$ pip install -e <path to checked out killMS>
(myvenv)$ cd <path to checked out killMS>
(myvenv)$ python setup.py build #this rebuilds the backend
The old build system is still in place - you can still do:
cd Predict
make
cd ../Array/Dot
make
cd ../../Gridder
make
Type
kMS.py -h
MakeModel.py -h
DDF.py -h
MakeMask.py -h
in a file .txt (here mslist.txt), put the path to your MSs, for example
/data/tasse/BootesObs/L374583/L374583_SB244_uv.dppp.pre-cal_127080C79t_121MHz.pre-cal.ms
/data/tasse/BootesObs/L374583/L374583_SB254_uv.dppp.pre-cal_127080C79t_123MHz.pre-cal.ms
/data/tasse/BootesObs/L374583/L374583_SB264_uv.dppp.pre-cal_127080C79t_125MHz.pre-cal.ms
/data/tasse/BootesObs/L374583/L374583_SB274_uv.dppp.pre-cal_127080C79t_127MHz.pre-cal.ms
In the following, we do
DDF.py --Output-Name=image_DI --Data-MS mslist.txt --Deconv-PeakFactor 0.001000 --Data-ColName DATA --Parallel-NCPU=40 --Image-Mode=Clean --Deconv-CycleFactor=0 --Deconv-MaxMajorIter=3 --Deconv-Mode SSD --Weight-Robust -0.15 --Image-NPix=10000 --CF-wmax 100000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 3 --Facets-NFacets=11 --SSDClean-NEnlargeData 0 --Freq-NDegridBand 1 --Beam-NBand 1 --Deconv-RMSFactor=3.000000 --Data-Sort 1 --Cache-Dir=. --Freq-NBand=2 --Mask-Auto=1 --Mask-SigTh=5.00 --Cache-Reset 0 --SSDClean-MinSizeInitHMP=10
MakeModel.py --BaseImageName image_DI --NCluster 10
-> creates a cluster nodes catalog: image_DI.npy.ClusterCat.npy
kMS.py --MSName mslist.txt --SolverType KAFCA --PolMode Scalar --BaseImageName image_DI --dt 1 --NCPU 40 --OutSolsName testKAFCA --NChanSols 1 --InCol DATA --OutCol DATA --Weighting Natural --NodesFile image_DI.npy.ClusterCat.npy --MaxFacetSize 1.5
--> creates solution files inside each
DDF.py --Output-Name=image_DD --Data-MS mslist.txt --Deconv-PeakFactor 0.001000 --Data-ColName DATA --Parallel-NCPU=40 --Image-Mode=Clean --Deconv-CycleFactor=0 --Deconv-MaxMajorIter=3 --Deconv-Mode SSD --Weight-Robust -0.15 --Image-NPix=10000 --CF-wmax 100000 --CF-Nw 100 --Output-Also onNeds --Image-Cell 3 --Facets-NFacets=11 --SSDClean-NEnlargeData 0 --Freq-NDegridBand 1 --Beam-NBand 1 --Deconv-RMSFactor=3.000000 --Data-Sort 1 --Cache-Dir=. --Freq-NBand=2 --Mask-Auto=1 --Mask-SigTh=5.00 --Cache-Reset 0 --SSDClean-MinSizeInitHMP=10 --DDESolutions-DDSols testKAFCA --Predict-InitDicoModel image_DI.DicoModel --Facets-DiamMax 1.5 --Facets-DiamMin 0.1