Aimfast restored-image filename not updated to new naming style

[KshitijT]

Continuing my reduction travails from #561 , switched off plotting aimfast but kept aimfast itself on. Ran into this issue:

  File "/usr/local/bin/aimfast", line 5, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/aimfast/aimfast.py", line 1815, in main
    area_factor=args.factor)
  File "/usr/local/lib/python3.6/dist-packages/aimfast/aimfast.py", line 566, in image_dynamic_range
    fits_info = fitsInfo(fitsname)
  File "/usr/local/lib/python3.6/dist-packages/aimfast/aimfast.py", line 208, in fitsInfo
    hdu = fitsio.open(fitsname)
  File "/usr/local/lib/python3.6/dist-packages/astropy/io/fits/hdu/hdulist.py", line 151, in fitsopen
    lazy_load_hdus, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/astropy/io/fits/hdu/hdulist.py", line 390, in fromfile
    lazy_load_hdus=lazy_load_hdus, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/astropy/io/fits/hdu/hdulist.py", line 1039, in _readfrom
    fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
  File "/usr/local/lib/python3.6/dist-packages/astropy/utils/decorators.py", line 521, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/astropy/io/fits/file.py", line 178, in __init__
    self._open_filename(fileobj, mode, overwrite)
  File "/usr/local/lib/python3.6/dist-packages/astropy/io/fits/file.py", line 555, in _open_filename
    self._file = fileobj_open(self.name, IO_FITS_MODES[mode])
  File "/usr/local/lib/python3.6/dist-packages/astropy/io/fits/util.py", line 389, in fileobj_open
    return open(filename, mode, buffering=0)
FileNotFoundError: [Errno 2] No such file or directory: '/home/kthorat/output/meerkathi-pipeline_1-MFS-image.fits'

I think the culprit is this code in self-cal worker:

 if pipeline.enable_task(config, 'extract_sources'):
            aimfast_settings.update({"tigger-model"   : '{0:s}_{1:s}_{2:d}-pybdsm{3:s}.lsm.html:output'.format(
                prefix, field, num if num <= len(config['calibrate'].get('model', num))
                else len(config['calibrate'].get('model', num)),
                '-combined' if len(model.split('+')) >= 2 else '')})
        else:
            # Use the image
            if config['calibrate'].get('output_data')[num-1 if num <= len(config['calibrate'].get('output_data',[])) else -1] == "CORR_DATA":
                aimfast_settings.update({"restored-image" : '{0:s}_{1:d}{2:s}-image.fits:output'.format(
                                                                prefix, num, mfsprefix)})
            else:
                try:
                    im = config['calibrate'].get('output_data').index("CORR_RES") + 1
                except ValueError:
                    im = num
                aimfast_settings.update({"restored-image" : '{0:s}_{1:s}_{2:d}{3:s}-image.fits:output'.format(
                                                                prefix, field, im, mfsprefix)})

whereas, the image prefix in image_opts is defined as:

                  "prefix"    : '{0:s}_{1:s}_{2:d}'.format(prefix, field, num),

So the restored-image pattern should be:

aimfast_settings.update({"restored-image" : '{0:s}_{1:s}_{2:d}{3:s}-image.fits:output'.format(
                                                                prefix, field, im, mfsprefix)})

in both cases.

Pinging @Athanaseus here - let me know if this sounds sensible. I am happy to fix it in the worker if so. Switching off aimfast in the interim.

[Athanaseus]

Ah yeah, you are right it's missing the field (target) name. Please go ahead and make the fix while still at it.

[KshitijT]

Great, willfix.

[KshitijT]

Trying to fix this but running into the following error:

meerkathi - 2019-08-16 02:28:31,237 INFO - Running: aimfast --residual-image /scratch/output/continuum/image_1/mypipelinerun_IC5264_1-MFS-residual.fits --tigger-model /scratch/output/continuum/image_1/mypipelinerun_IC5264_1-pybdsm.lsm.html --area-factor 6 --normality-test normaltest --label mypipelinerun_IC5264_1
meerkathi - 2019-08-16 02:28:31,237 INFO - Loading /scratch/output/continuum/image_1/mypipelinerun_IC5264_1-pybdsm.lsm.html: Tigger sky model
meerkathi - 2019-08-16 02:28:31,238 CRITICAL - INFO:STIMELA-11:Runtime was 0:00:10.024797.
meerkathi - 2019-08-16 02:28:31,238 CRITICAL - INFO:STIMELA-11:Saving pipeline information in .last_self_calibration_loop.json
meerkathi - 2019-08-16 02:28:31,239 CRITICAL - INFO:STIMELA-11:Recipe executed successfully
meerkathi - 2019-08-16 02:28:31,240 ERROR - An unhandled exeption occured. If you think this is a bug please report it.
meerkathi - 2019-08-16 02:28:31,240 ERROR - Your logfile is here: /vault2-tina/kshitij/svw_tests/log-meerkathi.txt.
meerkathi - 2019-08-16 02:28:31,241 ERROR - You are running version: 0.1.0-1058-gacd8a75
meerkathi - 2019-08-16 02:28:31,242 ERROR - Traceback (most recent call last):
  File "/home/kshitij/meerkathi/meerkathi/__init__.py", line 210, in __run
    pipeline.run_workers()
  File "/home/kshitij/meerkathi/meerkathi/workers/worker_administrator.py", line 225, in run_workers
    worker.worker(self, recipe, config)
  File "/home/kshitij/meerkathi/meerkathi/workers/self_cal_worker.py", line 1350, in worker
    while quality_check(self_cal_iter_counter, field, enable=pipeline.enable_task(config, 'aimfast')):
  File "/home/kshitij/meerkathi/meerkathi/workers/self_cal_worker.py", line 1021, in quality_check
    fidelity_data = get_aimfast_data()
  File "/home/kshitij/meerkathi/meerkathi/workers/self_cal_worker.py", line 994, in get_aimfast_data
    with open(filename) as f:
IOError: [Errno 2] No such file or directory: 'output/fidelity_results.json'

Essentially, the fidelity_results.json file can't be found anywhere in the output folder. Pinging @Athanaseus and @dane-kleiner (since I am using the MMGA3 branch), could you please advise on where might I find fidelity_results.json? NB : This is with aimfast enabled, so not exactly the minimal config file.

[dane-kleiner]

The fidelity_results.json should be created and looked for in the output folder, I haven't changed where they go. So, does that file exist? If not, the issue is with the json file being created.

[KshitijT]

@dane-kleiner , no the file is nowhere to be found. Could you confirm you see it in your output folder (if you enable aimfast)?

[dane-kleiner]

Works on my local branch, the json file is created and found, though my local branch uses the old config file / schema.

Here's my config file, MMGA.yml

!!omap
- schema_version: 0.1.0

- general:
    data_path: msdir
    msdir: msdir
    input: input
    output: output
    prefix: pipeline-prefix
    init_pipeline: true

- get_data:
    dataid: [ngc1399_20MHz-avg]
    mvftoms:
      enable: false
      channel_range: 'null'
    untar:
      enable: false
    combine:
      enable: false

- observation_config:
    obsinfo:
      enable: false
    bpcal: auto
    fcal: auto
    gcal: auto
    xcal: auto
    target: auto
    reference_antenna: 'm019'
    primary_beam:
      enable: false

- prepare_data:
    enable: false
    manage_flags:
      enable: true
      add_bitflag_column: true
      init_legacy_flagset: true
      remove_flagsets: true
    fixvis:
      enable: false
    prepms:
      enable: false
    add_spectral_weights:
      enable: true

- flagging:
    enable: false
    label: ''
    autoflag_autocorr_powerspectra:
      enable: false
    flag_autocorr:
      enable: true
    quack_flagging:
      enable: false
      quackinterval: 10
      quackmode: beg
    flag_shadow:
      enable: false
    flag_spw:
      enable: true
      channels: '*:856~880MHz,*:1658~1800MHz,*:1419.8~1421.3MHz' #band-rolloffs and Milkyway
    flag_time:
      enable: false
      timerange: 'null' # "YYYY/MM/DD/HH:MM:SS-YYYY/MM/DD/HH:MM:SS"
    flag_scan:
      enable: false
      scans: 'null'
    flag_antennas:
      enable: false
      antennas: 'null'
    static_mask:
      enable: true
      mask: labelled_rfimask.pickle.npy
    autoflag_rfi:
      enable: true
      flagger: aoflagger
      column: DATA
      fields: fcal,bpcal,gcal
      strategy: firstpass_QV.rfis # Default for MeerKAT data
    flagging_summary:
      enable: true

- cross_cal:
    enable: false
    label: 1gc1
    uvrange: '>50'
    otfdelay: true
    clear_cal:
      enable: true
      field:
      - bpcal
      - fcal
      - target
      - gcal
      - xcal
      addmodel: true
    set_model:
      enable: true
      field:
      - fcal
      meerkathi_model: true
    delay_cal:
      enable: true
      combine: scan
      solint: inf
      field:
      - bpcal
      flag:
        enable: true
        mode: clip
        clipminmax: [-60, 60]
      plot:
        enable: true
    bp_cal:
      enable: true
      combine: scan
      solint: inf
      remove_ph_time_var: false
      field:
      - bpcal
      flag:
        enable: false
        mode: clip
        clipminmax: [0.1, 10]
      plot:
        enable: true
    gain_cal_flux:
      enable: true
      combine: ''
      field:
      - fcal
      flag:
        enable: false
        mode: clip
        clipminmax: [0.1, 10]
      plot:
        enable: true
    gain_cal_gain:
      enable: true
      combine: ''
      field:
      - gcal
      flag:
        enable: false
        mode: clip
        clipminmax: [0.1, 10]
      plot:
        enable: true
    transfer_fluxscale:
      enable: true
      reference:
      - fcal
      transfer:
      - gcal
      plot:
        enable: true
    apply_delay_cal:
      enable: true
      field:
      - bpcal
      applyto:
      - bpcal
      - gcal
      - xcal
      applymode: calflag
    apply_bp_cal:
      enable: true
      field:
      - bpcal
      applyto:
      - bpcal
      - gcal
      - xcal
      applymode: calflag
    apply_gain_cal_gain:
      enable: false
      field:
      - gcal
      applyto:
      - bpcal
      - gcal
      - xcal
      applymode: calflag
    apply_transfer_fluxscale:
      enable: true
      field:
      - gcal
      applyto:
      - bpcal
      - gcal
      - xcal
      applymode: calflag
    autoflag_closure_error:
      enable: false
      column: CORRECTED_DATA
      fields: auto
      calibrator_fields: auto
      baseline_to_group_threshold: 10
      scan_to_scan_threshold: 10
    flagging_summary:
      enable: true

# flag calibrated data
- flagging__2:
    enable: false
    label: ''
    autoflag_rfi:
      flagger: aoflagger
      enable: true
      column: CORRECTED_DATA
      fields: fcal,bpcal,gcal
      strategy: secondpass_QUV.rfis # This has to be in the pipeline input folder
    flagging_summary:
      enable: true

# recalibrate reflagged data to improve K and G solutions
- cross_cal__2:
    enable: false
    label: 1gc2
    uvrange: '>50'
    otfdelay: true
    clear_cal:
      enable: false
      field:
      - bpcal
      - fcal
      - target
      - gcal
      - xcal
      addmodel: true
    set_model:
      enable: false
      field:
      - fcal
      meerkathi_model: true
    delay_cal:
      enable: true
      combine: scan
      solint: inf
      field:
      - bpcal
      flag:
        enable: true
        mode: clip
        clipminmax: [-60, 60]
      plot:
        enable: true
    bp_cal:
      enable: true
      combine: scan
      solint: inf
      remove_ph_time_var: false
      field:
      - bpcal
      flag:
        enable: false
        mode: clip
        clipminmax: [0.1, 10]
      plot:
        enable: true
    gain_cal_flux:
      enable: true
      combine: ''
      field:
      - fcal
      flag:
        enable: false
        mode: clip
        clipminmax: [0.1, 10]
      plot:
        enable: true
    gain_cal_gain:
      enable: true
      combine: ''
      field:
      - gcal
      flag:
        enable: false
        mode: clip
        clipminmax: [0.1, 10]
      plot:
        enable: true
    transfer_fluxscale:
      enable: true
      reference:
      - fcal
      transfer:
      - gcal
      plot:
        enable: true
    apply_delay_cal:
      enable: true
      field:
      - bpcal
      applyto:
      - bpcal
      - gcal
      - xcal
      - target
      applymode: calflag
    apply_bp_cal:
      enable: true
      field:
      - bpcal
      applyto:
      - bpcal
      - gcal
      - xcal
      - target
      applymode: calflag
    apply_gain_cal_gain:
      enable: false
      field:
      - gcal
      applyto:
      - bpcal
      - gcal
      - xcal
      - target
      applymode: calflag
    apply_transfer_fluxscale:
      enable: true
      field:
      - gcal
      applyto:
      - bpcal
      - gcal
      - xcal
      - target
      applymode: calflag
    autoflag_closure_error:
      enable: false
      column: CORRECTED_DATA
      fields: auto
      calibrator_fields: auto
      baseline_to_group_threshold: 10
      scan_to_scan_threshold: 10
    flagging_summary:
      enable: true

- polcal:
    enable: false
    feed_angle_rotation: -90
    set_model:
      enable: true
      meerkathi_model: true
      threads: 24
    preaverage_time: 32s
    timesol_soltime: inf
    flagging_summary_crosshand_cal:
      enable: true
    do_solve_leakages: true
    preaverage_freq: 4
    label: crosshand_cal
    do_phaseup_crosshand_calibrator: true
    do_solve_crosshand_slope: true
    do_dump_postcalibration_leakage_reports: true
    do_solve_crosshand_phase: true
    timesol_solfreqsel: '' 
    solve_uvdist: 150~10000000000m
    do_apply_XD: true
    do_dump_precalibration_leakage_reports: true
    freqsol_soltime: inf

- inspect_data:
    enable: false
    label: 1gc2
    real_imag:
      enable: true
      fields:
      - gcal
      - bpcal
    amp_uvwave:
      enable: true
      fields:
      - gcal
      - bpcal
    amp_phase:
      enable: true
      fields:
      - gcal
      - bpcal
    amp_ant:
      enable: true
      fields:
      - gcal
      - bpcal
    phase_uvwave:
      enable: true
      fields:
      - gcal
      - bpcal
    amp_scan:
      enable: true
      fields:
      - gcal
      - bpcal
      - target

- split_target:
    enable: false
    label_in: ''
    label_out: avg
    split_target:
      enable: true
      column: corrected
      time_average: ''
      freq_average: 5
      spw: ''
      otfcal:
        enable: false
        callabel: 1gc2
        apply_delay_cal:
          enable: true
          field:
           - bpcal
        apply_bp_cal:
          enable: true
          field:
           - bpcal
        apply_gain_cal_gain:
          enable: false
          field:
           - gcal
        apply_transfer_fluxscale:
          enable: true
          field:
           - gcal
    init_legacy_flagset:
      enable: false
    changecentre:
      enable: false
      ra: ''
      dec: ''
    obsinfo:
      enable: true
      listobs: true
      summary_json: true

# flag only target data
- flagging__3:
    enable: false
    label: 'corr'
    autoflag_rfi:
      flagger: aoflagger
      enable: true
      column: DATA
      fields: target
      strategy: secondpass_QUV.rfis # This can be a bit aggressive !!
    rfinder:
      enable: true
      field: target
      polarization: Q
      spw_enable: true
      spw_width: 10
      time_enable: true
      time_step: 10
    flagging_summary:
      enable: true

- masking:
    enable: false
    cell_size: 2
    name_mask: my_mask.fits
    final_mask: 'final_mask.fits'
    centre_coord: ['03:18:0.000', '-37:27:0.000']
    mask_size: 3600
    extended_source_input: 'Fornaxa_vla.FITS'
    query_catalog:
      enable: true
      catalog: SUMSS
      width_image: 3d
    pb_correction:
      enable: true
      frequency: 1.37031054688e9
    make_mask:
      enable: true
      mask_with: sofia
      input_image: pbcorr
      thresh_lev: 7
    merge_with_extended:
      enable: true
      extended_source_input: Fornaxa_vla.FITS #specify ouptut name of extended source mask if present in /input
      mask_with: thresh
      thresh_lev: 8.e-4

- self_cal:
    enable: true
    label: avg
    undo_subtractmodelcol: false
    spwid: 0
    primary_beam: false
    img_pol: I
    img_npix: 5540
    img_cell: 1.3
    img_padding: 1.3
    img_weight: briggs
    img_robust: -1.5
    img_niter: 1000000
    img_facets: 11
    img_mgain: 0.9
    img_cleanborder: 0
    img_nchans: 3
    img_joinchannels: true
    img_fit_spectral_pol: 1
    calibrate_with: cubical
    cal_gain_amplitude_clip_low: 0.5
    cal_gain_amplitude_clip_high: 1.5
    cal_niter: 3
    start_at_iter: 1
    image:
      enable: false
      auto_mask: [40, 30, 10, 5, 3]
      auto_threshold: [0.5, 0.5, 0.5, 0.25, 0.25]
      column: [DATA, CORRECTED_DATA, CORRECTED_DATA, CORRECTED_DATA, CORRECTED_DATA]
      mask_from_sky: false
      multi_scale: false
      multi_scale_scales: [0, 40, 80, 150] #pixel
      local_rms: false
      no_update_model: false
      minuvw_m: 0
    sofia_mask:
      enable: false
      flag: false
      threshold: 5
      scale_noise_window: 51
    extract_sources:
      enable: false
      sourcefinder: 'pybdsm'
      spi: false
      thresh_pix: [10, 10, 10]
      thresh_isl: [8, 8, 5]
    aimfast:
      enable: true
      tolerance: 0.02
      plot: true
    calibrate:
      enable: false
      model: ['1', '2', '3', '4']
      output_data: [CORR_DATA]
      gain_matrix_type: [GainDiagPhase, GainDiag, GainDiag, GainDiag]
      Gsols_time: [90, 90, 90]
      Gsols_channel: [0, 0, 0]
      add_vis_model: true
      model_mode: vis_only
      ragavi_plot:
        enable: true
    restore_model:
      enable: false
      model: 2+3
      clean_model: '4'
    transfer_apply_gains:
      enable: false
      transfer_to_label: corr
    transfer_model:
      enable: false
      transfer_to_label: corr
      model: auto
    highfreqres_contim:
      enable: false
      chans: 12
      deconv_chans: 3
      fit_spectral_pol: 2
    flagging_summary:
      enable: false

- image_HI:
    enable: false
    label: corr
    restfreq: 1.420405752GHz
    npix: [1024]
    cell: 7
    weight: briggs
    robust: 0
    subtractmodelcol:
      enable: true
    mstransform:
      enable: true
      doppler: true
      telescope: meerkat
      mode: frequency
      outframe: bary
      uvlin: true
      fitspw:
      fitorder: 1
      column: corrected
    sunblocker:
      enable: false
      use_mstransform: true
      vampirisms: true
      uvmax: 2000
      threshold: 4
    wsclean_image:
      enable: true
      wscl_niter: 2
      tol: 0.01
      ownfitsmask: # or own mask '<mask-name>.fits:output' 
      rm_intcubes: true
      use_mstransform: true
      pol: I
      spwid: 0
      nchans: all
      firstchan: 0
      binchans: 1
      npix: [1024]
      padding: 1.2
      cell: 7
      weight: briggs
      robust: 2
      taper: 0
      niter: 10000000
      mgain: 1.0
      automask: 5
      autothreshold: 0.5
      cleanborder: 0
      no_update_mod: true
      make_cube: true
      multi_scale: false
      multi_scale_scales: [0, 40, 80, 150] #pixel
    casa_image:
      enable: false
      use_mstransform: true
      pol: I
      spwid: 0
      nchans: all
      startchan: 0
      npix: [1024]
      cell: 7
      weight: briggs
      robust: 2
      niter: 0
      threshold: 10mJy
      port2fits: true
    sofia:
      enable: false
      rmsMode: mad
      threshold: 4.0
      flag: false
      flagregion: []
      merge: true
      mergeX: 2
      mergeY: 2
      mergeZ: 3
      minSizeX: 3
      minSizeY: 3
      minSizeZ: 5
    freq_to_vel:
      enable: true
      reverse: false
    remove_stokes_axis:
      enable: true
    pb_cube:
      enable: false
    sharpener:
      enable: true
      catalog: PYBDSF
    flagging_summary:
      enable: false

[Athanaseus]

I reproduced the error using both singularity and uDocker. Somehow the fidelity_results.json is not written out. On the other hand Docker works just fine.

[KshitijT]

LoL, I am going to go the docker way then and wait for you to fix aimfast for singularity/udocker cases, @Athanaseus . :P But seriously, thanks for all the help with this, @Athanaseus and @dane-kleiner .

[KshitijT]

I think this is fixed - @SpheMakh could you confirm this was one of the singularity issues you fixed?

[KshitijT]

Checked with current version of meerkathi - runs through. Thanks to whoever fixed this in the interim.