search

e-merlin / eMERLIN_CASA_pipeline

This is CASA eMERLIN pipeline to calibrate data from the e-MERLIN array. Please fork the repository before making any changes and read the Coding Practices page in the wiki. Please add issues with the pipeline in the issues tab.
GNU General Public License v3.0
14 stars 11 forks source link

Bandpass #38

Closed jmoldon closed 7 years ago

jmoldon commented 7 years ago

Ready and working. function em.initial_bp_cal runs on 1407+284 a 'p', 'ap' calibrations and then runs bandpass. It produces plots of all the calibration tables produced.

We may want to define sub functions to tasks that are repeated because the function is a bit long. We also need to think about a clear way to pass calibration tables from one task to the next, even when not all tasks are executed.

Bandpass table on one baseline: bandpass

Amplitude pre calibration amp_pre_calibration

Amplitude post calibration amp_post_calibration

Phase pre calibration phs_pre_calibration

Phase post calibration phs_post_calibration1

jradcliffe5 commented 7 years ago

'We also need to think about a clear way to pass calibration tables from one task to the next'

We either use the ms.writehistory to check what has been applied/derived for the data or we can look for the newest version of the table e.g. for bandpass, the initial table has a suffix B0, the next B1 etc like what the AIPS pipeline does implicitly. I think the first option may be more useful as it more scope for expansion in the future such as an automated mode of the pipeline where it can look at the history and decide the next steps for you without an input file or gui.

jmoldon commented 7 years ago

I agree, we need some kind of "status database" so any step can be tracked, repeated or continued from. The history file within the MS sounds logical, but is it possible to edit it? for example if we want to repeat a step with different parameters. Also, what happens if a calibration table is deleted by hand but according to the history it should be available?

Maybe the history file should contain only changes that actually modify the MS (uvfix, flag history, applycal), while we keep track of the steps in a different text file that also checks the available tables in the calib folder.

Another possibility is to create tables with a time stamp so we now exactly which ones were produced and, of those, which one is the best one that will be applied. There are many possibilities but we should keep it a simple as possible according to our needs. For the moment what I have inplemented is what you comment: tables bpass.G1, bpass.G2, bpass.B0, etc that are removed each time the task is executed.

I think we can open an issue to discuss this in particular, and take a decission in the next meeting with everyone.

jradcliffe5 commented 7 years ago

Hmm yeah, maybe we should have the pipeline check both the calib directory and the ms history file and then produce a warning about what is different. When the pipeline deletes previous tables it can write to the history that this table has been deleted which can be used to track pipeline generated tables etc. I've opened #39 for this now.