Indexing Error For MS File With A Large Amount Of Spectral Windows

[Clark0907]

I am running into an issue when executing the "run_casa_pipeline_phangs-alma.py" part of the pipeline. There is another issue that references something similar, issue #180, and the fix seems to be setting "combinespw:True" in the config_definitions.txt key file.

I was wondering if there were any updates to this issue as I have multiple ms files that I am combing at once, other than the one that is throwing the error, and I do not want to combine the spws for all of them. Here is the error message below:

IndexError                                Traceback (most recent call last)
<ipython-input-10-c1bc4324497e> in <module>()
----> 1 execfile('/home/iyclark/Phangs-ALMATests/phangs_imaging_scripts-master/run_casa_pipeline_phangs-alma.py')

/home/iyclark/Phangs-ALMATests/phangs_imaging_scripts-master/run_casa_pipeline_phangs-alma.py in <module>()
    169     this_uvh.loop_stage_uvdata(do_copy=True, do_contsub=True,
    170                                do_extract_line=False, do_extract_cont=False,
--> 171                                do_remove_staging=False, overwrite=True)
    172 
    173     this_uvh.loop_stage_uvdata(do_copy=False, do_contsub=False,

/home/iyclark/Phangs-ALMATests/phangs_imaging_scripts-master/phangsPipeline/handlerVis.pyc in loop_stage_uvdata(self, do_all, do_copy, do_remove_staging, do_custom, do_contsub, do_extract_line, do_extract_cont, extra_ext, make_directories, statwt_line, statwt_cont, intent, timebin, just_projects, strict_config, require_full_line_coverage, require_full_cont_coverage, overwrite)
    152                         timebin=timebin,
    153                         require_full_line_coverage=require_full_line_coverage,
--> 154                         overwrite=overwrite,
    155                     )
    156 

/home/iyclark/Phangs-ALMATests/phangs_imaging_scripts-master/phangsPipeline/handlerVis.pyc in task_split(self, target, project, array_tag, obsnum, product, intent, extra_ext_out, do_statwt, timebin, use_symlink, require_full_line_coverage, overwrite)
    389                             max_chanwidth_kms=max_chanwidth_kms,
    390                             vsys_kms=vsys, vwidth_kms=vwidth,
--> 391                             require_full_line_coverage=require_full_line_coverage)
    392 
    393                     if spw is None or len(spw) == 0:

/home/iyclark/Phangs-ALMATests/phangs_imaging_scripts-master/phangsPipeline/casaVisRoutines.pyc in find_spws_for_line(infile, line, restfreq_ghz, vsys_kms, vwidth_kms, vlow_kms, vhigh_kms, max_chanwidth_kms, require_data, require_full_line_coverage, exit_on_error, as_list)
    549 
    550     logger.debug("... vm = au.ValueMapping(infile) ...")
--> 551     vm = au.ValueMapping(infile)
    552     logger.debug("... vm = au.ValueMapping(infile) done")
    553 

/home/iyclark/analysis_scripts/analysisUtils.pyc in __init__(self, inputMs)
   1547         """
   1548         self.inputMs = inputMs
-> 1549         self.setInputMs(self.inputMs)
   1550 
   1551     def getInputMs(self):

/home/iyclark/analysis_scripts/analysisUtils.pyc in setInputMs(self, inputMs)
   1575         self.doPolarizations()
   1576         self.doSpwAndFrequency()
-> 1577         self.doSpwScanMapping()     ;  self.doSpwFieldMapping()
   1578         self.doSpwIntentMapping()
   1579 

/home/iyclark/analysis_scripts/analysisUtils.pyc in doSpwScanMapping(self)
   1695         self.scansForSpw = {}
   1696         for i in self.spwForDataDescId :
-> 1697             spw = self.spwForDataDescId[i]
   1698             indices = np.where(self.dataDescId == i)
   1699             self.scansForSpw[spw] = np.unique(self.scans[indices])

IndexError: index 103 is out of bounds for axis 0 with size 103

[e-koch]

@Clark0907 -- I think this might actually be a bug in the analysisUtils package, but we should be able to find a workaround here. Would you mind posting the listobs log output here for the SPW table? I'm wondering what's causing there to be so many SPWs.

[Clark0907]

Hi @e-koch here is the listobs for the calibrated ms file of the tm1 configuration on ALMA that is causing us problems!

listobs_12m_tm1.pdf

[e-koch]

Hi @Clark0907 -- thank you! I'm thinking is possibly an edge-case in how the PHANGS pipeline expects the SPWs for measurement sets with the same config labeled.

would you mind running a quick test by labelling the current tm1 to tm1_1, tm1_2, and tm1_3, and checking whether combining those separately with the 12m tm2 an 7m works? you'll need to update the config_definitions.txt file with these, too. I think you could just run the staging step to save time.

[1054]

Hi @Clark0907 , thank you for finding the issue and providing the details. Hi @e-koch , thank you for the checking too! Sorry for my late reply.

I think this error is caused by the analysisUtils package which cannot make a ValueMapping object from the measurement set "calibrated_final.ms". A minimum reproducible code is probably like this (please confirm if you may):

import sys
sys.path.insert(1, '/your/path/to/analysis_scripts')
import analysisUtils as au
vm = au.ValueMapping('/your/path/to/calibrated_final.ms')
#--> The error about the index error should occur at this point

My guess is that when the calibration pipeline produces the "calibrated_final.ms", only science spws are extracted and non-science spws are discarded, therefore, the "DATA_DESCRIPTION" table does not contain the full list of "SPECTRAL_WINDOW_ID".

I guess one possible solution is to use the "calibrated.ms" if you have one, or, as what we have been doing, using the individual "uid___A002_*.ms" data (if not found then "uid___A002_*.ms.split.cal" data). Please have a try with the "calibrated.ms" first if you have that data in your calibrated directory. If it is not found, then please list all the "uid___A002_*.ms" data in the "keys/ms_file_key.txt".

Thanks for having a try! Please let us know if any solution could work! Good luck!

Best regards, Daizhong

[1054]

In addition, why combinespw works? I think this is because when combinespw is set to True, our imaging pipeline will call casaVisRoutines.find_spws_for_science instead of casaVisRoutines.find_spws_for_line. The later will call the au.ValueMapping function whereas the former will not call that function in default...

[Clark0907]

Hi everyone!

Thank you for getting back to me about this issue I appreciate it:) I am going to be out of the office for the next week so I will run these tests and provide updates when I return!

Thanks Again,

Ilyse Clark

On Sun, Jul 2, 2023 at 4:44 PM 1054 @.***> wrote:

In addition, why combinespw works? I think this is because when combinespw is set to True, our imaging pipeline will call casaVisRoutines.find_spws_for_science instead of casaVisRoutines.find_spws_for_line. The later will call the au.ValueMapping function whereas the former will not call that function in default...

— Reply to this email directly, view it on GitHub https://github.com/akleroy/phangs_imaging_scripts/issues/239#issuecomment-1616812641, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARDXNYKMV6XCLJVFYMDUBQTXOHMRXANCNFSM6AAAAAAZQZEE7Q . You are receiving this because you were mentioned.Message ID: @.***>

[Clark0907]

Hi @e-koch this approach to changing the array tags to 12_1, 12_2, and 12_3 in the ms key file and in the config key file where is asks for the array tags, along with using the "combinespw=True" in the config key file, worked on making a dirty image for all the ms files together which I was not able to do before!

We will run the full clean soon and update if this method still works:) Thank you so much for the help I appreciate it alot!

[e-koch]

Glad to hear! did you have a chance to check out @1054's suggestion if the MSs are the calibrated_final.ms delivered by ALMA?

[Clark0907]

Glad to hear! did you have a chance to check out @1054's suggestion if the MSs are the calibrated_final.ms delivered by ALMA?

Hi, sorry I forgot to respond to that message! All of the ms files we use from each configuration are the calibrated_final.ms products from ALMA except for the tm2 data which is in two separate ms files that we put into in the pipeline!