18/05/2023 Meeting notes

[Victoria-Samboco]

The discussion of the meeting primarily focused on the issue of the Sun being subtracted while not rephased to the centre of the field, leading to unexpected results.

It was identified that the problem occurred because the MODEL_DATA_SUN was not rephased back to the original Measurement Set (MS) after subtraction.

To address this issue, the solution is to ensure that the MODEL_DATA_SUN is properly rephased to match the original MS.

Additionally, there was a recommendation to name the scans with their actual scan numbers (e.g., 3, 5, 7) instead of index numbers (0, 1, 2) to avoid confusion and improve clarity in the data and avoid the GLOB function problem.

Hopefully, the peeling step with the quartical will yield better results by implementing these suggestions.

[Victoria-Samboco]

When I try to copy the Model of the Sun after rephasing it returns this error

─────────────────────────────────────────────── boom.copycolumns ───────────────────────────────────────────────
2023-05-18 23:25:13 STIMELA.boom.copycolumns INFO: ### validated inputs
2023-05-18 23:25:13 STIMELA.boom.copycolumns INFO: cab find_sun_stimela.copy_columns:
2023-05-18 23:25:13 STIMELA.boom.copycolumns INFO:   from_ms = 
obs2/msdir/1671435077_sdp_l0_1024ch_GRS1747-312_concat.ms
2023-05-18 23:25:13 STIMELA.boom.copycolumns INFO:   to_ms = msdir/1671435077_sdp_l0_1024ch_GRS1747-312.ms
2023-05-18 23:25:13 STIMELA.boom.copycolumns INFO:   columns_to_copy = ['MODEL_DATA', 'MODEL_DATA_SUN']
### running copy_columns
# invoking callable find_sun_stimela.copy_columns() using external interpreter
# Successful readonly open of default-locked table obs2/msdir/1671435077_sdp_l0_1024ch_GRS1747-312_concat.ms: 27
columns, 437472 rows
# Successful read/write open of default-locked table msdir/1671435077_sdp_l0_1024ch_GRS1747-312.ms: 29 columns, 
218736 rows
0:03:02 boom.copycolumns copy_columns CPU 9.1%|RAM 189/504G|Load 9.5|R 12730 0.60G 16569ms|W 3133 0.02G 25056ms 
# Traceback (most recent call last):
#   File "<string>", line 17, in <module>
#   File "/home/samboco/solarkat/SUN_IMAGING_STEPS/pipeline/find_sun_stimela.py", line 446, in copy_columns
#     dest_table.putcol(column_name, src_data)
#   File "/home/samboco/Virtual_enviroments/stimela_env/lib/python3.8/site-packages/casacore/tables/table.py", 
line 1155, in putcol
#     self._putcol(columnname, startrow, nrow, rowincr, value)
# RuntimeError: ArrayColumn::putColumn - column MODEL_DATA has 218736, array has 437472 rows: Table array 
conformance error
2023-05-18 23:28:56 STIMELA.boom.copycolumns ERROR: step 'boom.copycolumns' has failed, aborting the recipe
──────────────────────────────────────── detailed error report follows ─────────────────────────────────────────
        ⚠ step 'boom.copycolumns' has failed, aborting the recipe
        └── error running cab 'find_sun_stimela.copy_columns'
            └── copy_columns returns error code 1 after 0:03:43

I don't understand why because I think rephasing the MS is not supposed to change the columns row number (please correct me if I'm wrong)

After rephasing the array of the concat MS is not the same as the original MS, and because when I add the columns in the original MS it uses its data description (DATA) with a different number of column rows it fails ( but this is not expected because I am rephasing back to the original phase centre).

The curious thing is that before when I was only rephrasing the CORRECTED_DATA and keeping the other columns it was running without any problem...

[o-smirnov]

Well clearly the number of rows is different:

# Successful readonly open of default-locked table obs2/msdir/1671435077_sdp_l0_1024ch_GRS1747-312_concat.ms: 27
columns, 437472 rows
# Successful read/write open of default-locked table msdir/1671435077_sdp_l0_1024ch_GRS1747-312.ms: 29 columns, 
218736 rows

You are right -- knowing how the software works, rephasing can't change the number of rows. However, when you split and re-concatenate, there's plenty more opportunities for something to go wrong there.

Remind me, why do you need to make a concatenated MS, anyway? Maybe I just forget the reason... seems like a lot of extra disk space to me, so it'd better be a good reason.

You could copy the Sun model over from the per-scan, rephased data sets directly, without concatenating:

• make an empty MODEL_SUN_DATA column in the original MS, of the same shape as MODEL_DATA

• per each scan, open the original MS as tab = table(msname, readonly=False).query(f"SCAN_NUMBER=={scan}"). This gives you a table object that refers to a subset of rows of the original MS

• do a getcol on the per-scan MS, and a putcol into the tab object.

[Victoria-Samboco]

Remind me, why do you need to make a concatenated MS, anyway? Maybe I just forget the reason... seems like a lot of extra disk space to me, so it'd better be a good reason.

To be able to compare images before and after the Sun subtraction, in this way, we have the contaminated data that allows me to restart if something goes wrong and the final corrected data without the Sun.

You could copy the Sun model over from the per-scan, rephased data sets directly, without concatenating:

• make an empty MODEL_SUN_DATA column in the original MS, of the same shape as MODEL_DATA
• per each scan, open the original MS as tab = table(msname, readonly=False).query(f"SCAN_NUMBER=={scan}"). This gives you a table object that refers to a subset of rows of the original MS
• do a getcol on the per-scan MS and a putcol into the tab object.

From what I understand (please correct me if not correct), this will be done after the prediction step following the order:

1. Predict model
2. rephase the scans to the old phase centre
3. Make an empty MODEL_SUN_DATA in the original MS and Pass the Model-data of the Sun from each scan into each scan in the original MS
4. Subtract the MODEL_SUN_DATA from the CORRECTED_DATA in the original MS (in this case, I will be subtracting directly from the original MS) and store the output in a column name different from CORRECTED_DATA and then perform peeling with quartical.

[Victoria-Samboco]

For the option of using the scan numbers as input for other steps to avoid the glob command...

I have defined the scan_numbers step in the recipe this way

    scan_numbers: 
      info: 'Extract a list of unique scan numbers from a MS file. The scans output parameter is a list of strings representing the unique scan numbers.'
      cab:
        command: |
          from casacore.tables import table
          import numpy
          scans=[]
          maintab = table(ms,ack=False)
          scan_no = list(numpy.unique(maintab.getcol('SCAN_NUMBER')))
          for scan in scan_no:
            scans.append(str(scan))
          maintab.close()
          print(scans)
        flavour: python-code
        inputs:
          ms:
            dtype: MS
        outputs:
          scans:
            dtype: List[str]
      params:
        ms: '{recipe.ms}'

and I use it as input for the split_ms_by_scan step. For it to work the two steps have to run in sequence, i.e. if I run only the split step separately the scan_numbers.scans are not recognized. In this case, is okay because they are steps that follow each other, but when I try to use the scan numbers from another step it is not recognized.

split_ms_by_scan: 
  info: ' Split the MS into scans using as input the scan_numbers'
  params:
    scan_list: =steps.scan_numbers.scans 
    ms: =steps.scan_numbers.ms
  recipe:
    inputs:
      scan_list:
        dtype: List[str]
      ms:
        dtype: MS
    for_loop:
      var: scan
      over: scan_list
    steps:         
      casa_split_scan:
        cab: splitms_scan
        params:
          vis: '{root.ms}'
          datacolumn: 'all'
          scan: '{recipe.scan}'
          outputvis: ="{root.dir_out}" + "/" + "{root.perscan_dir_out}" + "/" + STRIPEXT(BASENAME(recipe.ms)) + "_scan_{recipe.scan}.ms"

This is one of the cases where it happens

    make_mask:      
      info: 'Creating a fits mask with breizorro'
      params:
        image_list: =steps.sort_images.sorted_images #=GLOB("{recipe.dir_out}/{recipe.perscan_dir_out}/{recipe.sun_images_dir}/*-MFS-image*.fits")
        region_dir:  "{recipe.dir_out}/{recipe.regions_dir_out}"     
      recipe:
        inputs:
          region_dir:
            dtype: Directory 
          image_list:
            dtype: List[File]
        for_loop:
          var: image
          over: image_list
        steps:
          make_mask:
            cab: breizorro
            params:
              restored_image: =recipe.image
              threshold: 9999999999999999999999999999999
              merge: ="{recipe.region_dir}" + "/" + "scan_{root.steps.split_ms_by_scan.scan}_sun_region.reg"
              mask: ="{root.dir_out}" + "/" + "{root.masks_dir_out}" + "/" + STRIPEXT(BASENAME(recipe.image)) + "_mask.fits" 

───────────────────────── boom.make_mask.make_mask ─────────────────────────
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask INFO: ### validated
inputs
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask INFO: cab breizorro:
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask INFO:   threshold =
1e+31
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask INFO:   restored_image
=
obs2/ms_scans/sun_images/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12-MFS-im
age.fits
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask INFO:   merge =
Unresolved(scan_{{root.steps.split_ms_by_scan.scan}}_sun_region.reg: invalid
key scan)
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask INFO:   mask =
obs2/sun_masks/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12-MFS-image_mask.f
its
2023-05-24 22:44:27 STIMELA.boom.make_mask.make_mask ERROR: step 
'make_mask': invalid inputs: 'merge'
2023-05-24 22:44:27 STIMELA.boom.make_mask ERROR: step 'boom.make_mask' has 
failed, aborting the recipe
────────────────────── detailed error report follows ───────────────────────
        ⚠ step 'boom.make_mask' has failed, aborting the recipe
        └── step 'boom.make_mask.make_mask' has failed, aborting the recipe
            └── step 'make_mask': invalid inputs: 'merge'