search

lmorabit / lofar-vlbi

GNU General Public License v3.0
16 stars 13 forks source link

Pipeline crashes at CrashesTargetListToMapfile (if doing single infield delay calibration) #64

Closed mooneyse closed 4 years ago

mooneyse commented 4 years ago

These issues aren't relevant if preparing the data for LB-Split-Calibrators.parset but I thought I'd document them here in case it is useful for someone.

Going off what is in PipelineStep_TargetListToMapfile.py, the pipeline step requires one edit, and a few of the arguments are unused. I think it can be changed from

# Initialise file with direction of delay calibrator
prep_dirs.control.kind                    = plugin
prep_dirs.control.type                    = TargetListToMapfile
prep_dirs.control.mapfile_dir             = input.output.mapfile_dir
prep_dirs.control.infile                  = dpppconcat.output.mapfile
prep_dirs.control.filename                = prep_dirs.mapfile
prep_dirs.control.wd                      = {{ job_directory }}
prep_dirs.control.nP                      = 3   # this is the default setting
prep_dirs.control.counter                 = 0
prep_dirs.control.manual                  = True ## 
prep_dirs.control.target_file             = find_delay_cal.output.calfile

to

# Initialise file with direction of delay calibrator
prep_dirs.control.kind        = plugin
prep_dirs.control.type        = TargetListToMapfile
prep_dirs.control.mapfile_dir = input.output.mapfile_dir
prep_dirs.control.mapfile_in  = dpppconcat.output.mapfile  # infile -> mapfile_in, I think
prep_dirs.control.filename    = prep_dirs.mapfile
prep_dirs.control.target_file = find_delay_cal.output.calfile

Also PipelineStep_TargetListToMapfile.py itself reads in primary_delay_calibrator.csv and expects a header on that CSV:

t = Table.read(target_file, format='csv')
RA_val = t['RA_LOTSS'].data[0]
DEC_val = t['DEC_LOTSS'].data[0]
Source_id = t['Source_id'].data[0]

But there is none in the file. I added one manually as a workaround.

Lastly, the mapfile written out by the prep_dirs step causes the next step, dppp_phaseup, to crash with a mapfile mismatch error. I manually edited it from

[{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_122MHz.msdpppconcat', 'skip': False}]

to

[{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_122MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_124MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_126MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_128MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_130MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_132MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_134MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_136MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_138MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_140MHz.msdpppconcat', 'skip': False}]

to get it working, but I don't know enough genericpipeline mapfile kung fu to patch it properly.

lmorabit commented 4 years ago

Hi Sean, I'm actively updating the pipeline now and haven't pushed all my changes yet. Can I reach you on skype or slack to have a chat about what you're running into?

lmorabit commented 4 years ago

This step no longer exists on the master branch. Do you need this to be updated on a different branch?

mooneyse commented 4 years ago

Nope, all good, thanks :+1: