Execution Block ID uid://A001/X15a0/X10c Sgr_A_st_s_03_TM1

keflavich commented 2 years ago

Sgr_A_st_s_03_TM1 uid://A001/X15a0/X10c

[x] Observations completed?
[x] Delivered?
[x] Downloaded? (specify where)
- [x] hipergator
- [x] hipergator pipeline run
- Pipeline Run 20231219T050506
- Pipeline Run 20231218T175041
- Pipeline Run 20231218T191352
- Pipeline Run 20230225T055630
- Pipeline Run 20230225T050304
- Pipeline Run 20230214T215021
- Pipeline Run 20230214T201910
- Pipeline Run 20230114T050550
- Pipeline Run 20230114T061944
- Pipeline Run 20221203T062354
- Pipeline Run 20221203T050156
- Pipeline Run 20221204T054116
- Pipeline Run 20221028T040411
[x] Weblog unpacked
[ ] Weblog Quality Assessment?
Extra Weblog Sgr_A_st_s_03_TM1_0 -> pipeline-20221204T054116, Extra Weblog Sgr_A_st_s_03_TM1_1 -> pipeline-20221013T165317
[ ] Imaging: Continuum
[ ] Imaging: Lines
- [ ] SPW25 H13CN
- [ ] SPW27 H13CO+/SiO
- [ ] SPW29 HCO+
- [ ] SPW31 HNCO
- [ ] SPW33 Cont 1
- [ ] SPW35 Cont 2

Product Links:

Reprocessed Product Links:

keflavich commented 1 year ago

This is one of the C9 observations.

It failed on hifa_importdata:

2022-10-28 04:04:11     INFO    hifa_importdata::::casa ##########################################
2022-10-28 04:04:11     INFO    hifa_importdata::::casa ##### Begin Task: hifa_importdata    #####
2022-10-28 04:04:11     INFO    hifa_importdata::::casa hifa_importdata( vis=['../rawdata/uid___A002_Xfed4ee_X1e3', '../rawdata/uid___A002_Xfee03e_X2787'], session=['session_1', 'session_2'], pipelinemode='automatic', asis='SBSummary ExecBlock Antenna Station Receiver Source CalAtmosphere CalWVR CalPointing', pr
ocess_caldevice=False, overwrite=False, nocopy=False, bdfflags=True, asimaging=False, lazy=False, dbservice=False, ocorr_mode='ca', createmms='false', minparang=0.0, dryrun=False, acceptresults=True )
2022-10-28 04:04:11     INFO    hifa_importdata::pipeline.infrastructure.basetask::casa Starting execution for stage 1
2022-10-28 04:04:11     INFO    hifa_importdata::pipeline.infrastructure.basetask::casa Equivalent CASA call: hifa_importdata(minparang=0.0, createmms='false', ocorr_mode='ca', dbservice=False, lazy=False, asimaging=False, bdfflags=True, nocopy=False, overwrite=False, process_caldevice=False, asis='SBSummary Exe
cBlock Antenna Station Receiver Source CalAtmosphere CalWVR CalPointing', session=['session_1', 'session_2'], vis=['uid___A002_Xfed4ee_X1e3', 'uid___A002_Xfee03e_X2787'])
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa Error executing pipeline task hifa_importdata.
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa Traceback (most recent call last):
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            return self._handle_multiple_vis(dry_run, **parameters)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            single_result = self.execute(dry_run=dry_run, **parameters)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            result = method(self, *args, **kw)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            result = method(self, *args, **kwargs)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            result = method(self, *args, **kw)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 61, in finalise_pipeline_result
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            result = method(self, *args, **kw)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            result = self.prepare(**prepare_parameters)
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/importdata/importdata.py", line 174, in prepare
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+            raise TypeError('{!s} is of unhandled type'.format(vis))
2022-10-28 04:04:12     SEVERE  hifa_importdata::pipeline.infrastructure.basetask::casa+        TypeError: ../rawdata/uid___A002_Xfed4ee_X1e3 is of unhandled type
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa Task hifa_importdata raised an exception of class PipelineException with the following message: Traceback (most recent call last):
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            return self._handle_multiple_vis(dry_run, **parameters)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            single_result = self.execute(dry_run=dry_run, **parameters)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            result = method(self, *args, **kw)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            result = method(self, *args, **kwargs)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            result = method(self, *args, **kw)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 61, in finalise_pipeline_result
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            result = method(self, *args, **kw)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            result = self.prepare(**prepare_parameters)
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+          File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/importdata/importdata.py", line 174, in prepare
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+            raise TypeError('{!s} is of unhandled type'.format(vis))
2022-10-28 04:04:12     SEVERE  hifa_importdata::::casa+        TypeError: ../rawdata/uid___A002_Xfed4ee_X1e3 is of unhandled type
2022-10-28 04:04:12     INFO    hifa_importdata::::casa Task hifa_importdata complete. Start time: 2022-10-28 00:04:11.289983 End time: 2022-10-28 00:04:12.399441
2022-10-28 04:04:12     INFO    hifa_importdata::::casa ##### End Task: hifa_importdata      #####
2022-10-28 04:04:12     INFO    hifa_importdata::::casa ##########################################

This is a python error that indicates that hifa_importdata did not expect to receive a path string. That makes no sense at all.

keflavich commented 1 year ago

I believe these data are corrupted. However, we can't download them again until the ALMA archive allows logins again (the archive is up, but afaict, only works for public data)

keflavich commented 1 year ago

Nov 14, pipeline failed to run, but there was no error message. The log just shows that it spun up, then spun down. casa_log_mpi_pipeline_member.uid___A001_X15a0_X10c_51377685_2022-11-14_09_45_01.log for the record, though it's not accessible through globus (the logs live on a different drive)

keflavich commented 1 year ago

Removed calibrated/ again.

nbudaiev commented 1 year ago

December 4th pipeline run failed on mstransform with error message "Desired column (CORRECTED_DATA) not found in the input MS" Upon inspection, the corrected_data column is present in the data:

ms.open('/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated_failed_pre20230111/working/uid___A002_Xfee03e_X2787.ms')
ms.selectinit(2)
ms.getdata('CORRECTED_DATA')['corrected_data'].shape
Out[8]: (2, 1, 113400)

Full error output from casa_log_mpi_pipeline_imaging_member.uid___A001_X15a0_X10c_53087706_2022-12-04_00_40_53.log:

2022-12-04 06:12:32 INFO    mstransform::::casa ##########################################
2022-12-04 06:12:32 INFO    mstransform::::casa ##### Begin Task: mstransform        #####
2022-12-04 06:12:32 INFO    mstransform::::casa mstransform( vis='uid___A002_Xfee03e_X2787.ms', outputvis='uid___A002_Xfee03e_X2787_target.ms', createmms=False, separationaxis='auto', numsubms='auto', tileshape=[0], field='3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138', spw='25,27,29,31,33,35', scan='', antenna='', correlation='', timerange='', intent='OBSERVE_TARGET#ON_SOURCE', array='', uvrange='', observation='', feed='', datacolumn='corrected', realmodelcol=False, keepflags=True, usewtspectrum=False, combinespws=False, chanaverage=False, chanbin=1, hanning=False, regridms=False, mode='channel', nchan=-1, start=0, width=1, nspw=1, interpolation='linear', phasecenter='', restfreq='', outframe='', veltype='radio', preaverage=False, timeaverage=False, timebin='0s', timespan='', maxuvwdistance=0.0, docallib=False, callib='', douvcontsub=False, fitspw='', fitorder=0, want_cont=False, denoising_lib=True, nthreads=1, niter=1, disableparallel=False, ddistart=-1, taql='', monolithic_processing=False, reindex=False )
2022-12-04 06:12:32 INFO    MSTransformManager::parseMsSpecParams   Input file name is uid___A002_Xfee03e_X2787.ms
2022-12-04 06:12:32 INFO    MSTransformManager::parseMsSpecParams   Data column is CORRECTED
2022-12-04 06:12:32 INFO    MSTransformManager::parseMsSpecParams   Output file name is uid___A002_Xfee03e_X2787_target.ms
2022-12-04 06:12:32 INFO    MSTransformManager::parseMsSpecParams   Re-index is disabled 
2022-12-04 06:12:32 INFO    MSTransformManager::parseMsSpecParams   Tile shape is [0]
2022-12-04 06:12:32 INFO    MSTransformManager::parseDataSelParams  field selection is 3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138
2022-12-04 06:12:32 INFO    MSTransformManager::parseDataSelParams  spw selection is 25,27,29,31,33,35
2022-12-04 06:12:32 INFO    MSTransformManager::parseDataSelParams  scan intent selection is OBSERVE_TARGET#ON_SOURCE
2022-12-04 06:12:32 WARN    MSTransformManager::checkDataColumnsToFill  CORRECTED_DATA column requested but not available in input MS 
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams Selected SPWs Ids are Axis Lengths: [6, 4]  (NB: Matrix in Row/Column order)
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams+    [25, 0, 1919, 1
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams+     27, 0, 1919, 1
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams+     29, 0, 1919, 1
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams+     31, 0, 1919, 1
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams+     33, 0, 3839, 1
2022-12-04 06:12:32 INFO    MSTransformManager::initDataSelectionParams+     35, 0, 3839, 1]
2022-12-04 06:12:32 INFO    MSTransformManager::open    Select data
2022-12-04 06:12:32 INFO    MSTransformManager::createOutputMSStructure Create output MS structure
2022-12-04 06:12:32 SEVERE  mstransform::::casa Task mstransform raised an exception of class RuntimeError with the following message: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated/working/uid___A002_Xfee03e_X2787.ms).
2022-12-04 06:12:32 INFO    mstransform::::casa Task mstransform complete. Start time: 2022-12-04 01:12:31.563433 End time: 2022-12-04 01:12:31.768165
2022-12-04 06:12:32 INFO    mstransform::::casa ##### End Task: mstransform          #####
2022-12-04 06:12:32 INFO    mstransform::::casa ##########################################
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa Error executing pipeline task hif_mstransform.
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa Traceback (most recent call last):
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        return self._handle_multiple_vis(dry_run, **parameters)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        single_result = self.execute(dry_run=dry_run, **parameters)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kwargs)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 61, in finalise_pipeline_result
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = self.prepare(**prepare_parameters)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/tasks/mstransform/mstransform.py", line 167, in prepare
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        self._executor.execute(mstransform_job)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = job.execute(dry_run=self._dry_run, **kwargs)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        return self.fn(*self.args, **self.kw)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/mstransform.py", line 620, in __call__
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        task_result = _mstransform_t( _pc.document['vis'], _pc.document['outputvis'], _pc.document['createmms'], _pc.document['separationaxis'], _pc.document['numsubms'], _pc.document['tileshape'], _pc.document['field'], _pc.document['spw'], _pc.document['scan'], _pc.document['antenna'], _pc.document['correlation'], _pc.document['timerange'], _pc.document['intent'], _pc.document['array'], _pc.document['uvrange'], _pc.document['observation'], _pc.document['feed'], _pc.document['datacolumn'], _pc.document['realmodelcol'], _pc.document['keepflags'], _pc.document['usewtspectrum'], _pc.document['combinespws'], _pc.document['chanaverage'], _pc.document['chanbin'], _pc.document['hanning'], _pc.document['regridms'], _pc.document['mode'], _pc.document['nchan'], _pc.document['start'], _pc.document['width'], _pc.document['nspw'], _pc.document['interpolation'], _pc.document['phasecenter'], _pc.document['restfreq'], _pc.document['outframe'], _pc.document['veltype'], _pc.document['preaverage'], _pc.document['timeaverage'], _pc.document['timebin'], _pc.document['timespan'], _pc.document['maxuvwdistance'], _pc.document['docallib'], _pc.document['callib'], _pc.document['douvcontsub'], _pc.document['fitspw'], _pc.document['fitorder'], _pc.document['want_cont'], _pc.document['denoising_lib'], _pc.document['nthreads'], _pc.document['niter'], _pc.document['disableparallel'], _pc.document['ddistart'], _pc.document['taql'], _pc.document['monolithic_processing'], _pc.document['reindex'] )
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_mstransform.py", line 307, in mstransform
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        mtlocal.open()
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/mstransformer.py", line 44, in open
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        return self._swigobj.open()
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/mstransformer.py", line 176, in open
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        return _mstransformer.mstransformer_open(self)
2022-12-04 06:12:32 SEVERE  mstransform::pipeline.infrastructure.basetask::casa+    RuntimeError: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated/working/uid___A002_Xfee03e_X2787.ms).
2022-12-04 06:12:36 SEVERE  mstransform::::casa Task hif_mstransform raised an exception of class PipelineException with the following message: Traceback (most recent call last):
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        return self._handle_multiple_vis(dry_run, **parameters)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        single_result = self.execute(dry_run=dry_run, **parameters)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = method(self, *args, **kwargs)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 61, in finalise_pipeline_result
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = self.prepare(**prepare_parameters)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/tasks/mstransform/mstransform.py", line 167, in prepare
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        self._executor.execute(mstransform_job)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        result = job.execute(dry_run=self._dry_run, **kwargs)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        return self.fn(*self.args, **self.kw)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/mstransform.py", line 620, in __call__
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        task_result = _mstransform_t( _pc.document['vis'], _pc.document['outputvis'], _pc.document['createmms'], _pc.document['separationaxis'], _pc.document['numsubms'], _pc.document['tileshape'], _pc.document['field'], _pc.document['spw'], _pc.document['scan'], _pc.document['antenna'], _pc.document['correlation'], _pc.document['timerange'], _pc.document['intent'], _pc.document['array'], _pc.document['uvrange'], _pc.document['observation'], _pc.document['feed'], _pc.document['datacolumn'], _pc.document['realmodelcol'], _pc.document['keepflags'], _pc.document['usewtspectrum'], _pc.document['combinespws'], _pc.document['chanaverage'], _pc.document['chanbin'], _pc.document['hanning'], _pc.document['regridms'], _pc.document['mode'], _pc.document['nchan'], _pc.document['start'], _pc.document['width'], _pc.document['nspw'], _pc.document['interpolation'], _pc.document['phasecenter'], _pc.document['restfreq'], _pc.document['outframe'], _pc.document['veltype'], _pc.document['preaverage'], _pc.document['timeaverage'], _pc.document['timebin'], _pc.document['timespan'], _pc.document['maxuvwdistance'], _pc.document['docallib'], _pc.document['callib'], _pc.document['douvcontsub'], _pc.document['fitspw'], _pc.document['fitorder'], _pc.document['want_cont'], _pc.document['denoising_lib'], _pc.document['nthreads'], _pc.document['niter'], _pc.document['disableparallel'], _pc.document['ddistart'], _pc.document['taql'], _pc.document['monolithic_processing'], _pc.document['reindex'] )
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_mstransform.py", line 307, in mstransform
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        mtlocal.open()
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/mstransformer.py", line 44, in open
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        return self._swigobj.open()
2022-12-04 06:12:36 SEVERE  mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/mstransformer.py", line 176, in open
2022-12-04 06:12:36 SEVERE  mstransform::::casa+        return _mstransformer.mstransformer_open(self)
2022-12-04 06:12:36 SEVERE  mstransform::::casa+    RuntimeError: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated/working/uid___A002_Xfee03e_X2787.ms).
2022-12-04 06:12:36 INFO    hif_mstransform::::casa Task hif_mstransform complete. Start time: 2022-12-04 01:12:31.104067 End time: 2022-12-04 01:12:36.141056
2022-12-04 06:12:36 INFO    hif_mstransform::::casa ##### End Task: hif_mstransform      #####
2022-12-04 06:12:36 INFO    hif_mstransform::::casa ##########################################
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa Exception Reported: Error in hif_mstransform: Traceback (most recent call last):
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        return self._handle_multiple_vis(dry_run, **parameters)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        single_result = self.execute(dry_run=dry_run, **parameters)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = method(self, *args, **kwargs)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 61, in finalise_pipeline_result
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = self.prepare(**prepare_parameters)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/tasks/mstransform/mstransform.py", line 167, in prepare
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        self._executor.execute(mstransform_job)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        result = job.execute(dry_run=self._dry_run, **kwargs)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        return self.fn(*self.args, **self.kw)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/mstransform.py", line 620, in __call__
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        task_result = _mstransform_t( _pc.document['vis'], _pc.document['outputvis'], _pc.document['createmms'], _pc.document['separationaxis'], _pc.document['numsubms'], _pc.document['tileshape'], _pc.document['field'], _pc.document['spw'], _pc.document['scan'], _pc.document['antenna'], _pc.document['correlation'], _pc.document['timerange'], _pc.document['intent'], _pc.document['array'], _pc.document['uvrange'], _pc.document['observation'], _pc.document['feed'], _pc.document['datacolumn'], _pc.document['realmodelcol'], _pc.document['keepflags'], _pc.document['usewtspectrum'], _pc.document['combinespws'], _pc.document['chanaverage'], _pc.document['chanbin'], _pc.document['hanning'], _pc.document['regridms'], _pc.document['mode'], _pc.document['nchan'], _pc.document['start'], _pc.document['width'], _pc.document['nspw'], _pc.document['interpolation'], _pc.document['phasecenter'], _pc.document['restfreq'], _pc.document['outframe'], _pc.document['veltype'], _pc.document['preaverage'], _pc.document['timeaverage'], _pc.document['timebin'], _pc.document['timespan'], _pc.document['maxuvwdistance'], _pc.document['docallib'], _pc.document['callib'], _pc.document['douvcontsub'], _pc.document['fitspw'], _pc.document['fitorder'], _pc.document['want_cont'], _pc.document['denoising_lib'], _pc.document['nthreads'], _pc.document['niter'], _pc.document['disableparallel'], _pc.document['ddistart'], _pc.document['taql'], _pc.document['monolithic_processing'], _pc.document['reindex'] )
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_mstransform.py", line 307, in mstransform
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        mtlocal.open()
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/mstransformer.py", line 44, in open
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        return self._swigobj.open()
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/mstransformer.py", line 176, in open
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+        return _mstransformer.mstransformer_open(self)
2022-12-04 06:12:36 SEVERE  hif_mstransform::::casa+    RuntimeError: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated/working/uid___A002_Xfee03e_X2787.ms).
2022-12-04 06:12:36 INFO    hif_mstransform::::casa Traceback (most recent call last):
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/cli/gotasks/hif_mstransform.py", line 499, in __call__
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        _return_result_ = _hif_mstransform_t( _invocation_parameters['vis'],_invocation_parameters['outputvis'],_invocation_parameters['field'],_invocation_parameters['intent'],_invocation_parameters['spw'],_invocation_parameters['chanbin'],_invocation_parameters['timebin'],_invocation_parameters['pipelinemode'],_invocation_parameters['dryrun'],_invocation_parameters['acceptresults'] )
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/cli/hif_mstransform.py", line 95, in __call__
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        task_result = _hif_mstransform_t( _pc.document['vis'], _pc.document['outputvis'], _pc.document['field'], _pc.document['intent'], _pc.document['spw'], _pc.document['chanbin'], _pc.document['timebin'], _pc.document['pipelinemode'], _pc.document['dryrun'], _pc.document['acceptresults'] )
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/cli/private/task_hif_mstransform.py", line 29, in hif_mstransform
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        results = utils.execute_task(context, task_name, all_inputs)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        raise exceptions.PipelineException(previous_tracebacks_as_string)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+    pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        return self._handle_multiple_vis(dry_run, **parameters)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        single_result = self.execute(dry_run=dry_run, **parameters)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = method(self, *args, **kwargs)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 61, in finalise_pipeline_result
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = self.prepare(**prepare_parameters)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hif/tasks/mstransform/mstransform.py", line 167, in prepare
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        self._executor.execute(mstransform_job)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = method(self, *args, **kw)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        result = job.execute(dry_run=self._dry_run, **kwargs)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        return self.fn(*self.args, **self.kw)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/mstransform.py", line 620, in __call__
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        task_result = _mstransform_t( _pc.document['vis'], _pc.document['outputvis'], _pc.document['createmms'], _pc.document['separationaxis'], _pc.document['numsubms'], _pc.document['tileshape'], _pc.document['field'], _pc.document['spw'], _pc.document['scan'], _pc.document['antenna'], _pc.document['correlation'], _pc.document['timerange'], _pc.document['intent'], _pc.document['array'], _pc.document['uvrange'], _pc.document['observation'], _pc.document['feed'], _pc.document['datacolumn'], _pc.document['realmodelcol'], _pc.document['keepflags'], _pc.document['usewtspectrum'], _pc.document['combinespws'], _pc.document['chanaverage'], _pc.document['chanbin'], _pc.document['hanning'], _pc.document['regridms'], _pc.document['mode'], _pc.document['nchan'], _pc.document['start'], _pc.document['width'], _pc.document['nspw'], _pc.document['interpolation'], _pc.document['phasecenter'], _pc.document['restfreq'], _pc.document['outframe'], _pc.document['veltype'], _pc.document['preaverage'], _pc.document['timeaverage'], _pc.document['timebin'], _pc.document['timespan'], _pc.document['maxuvwdistance'], _pc.document['docallib'], _pc.document['callib'], _pc.document['douvcontsub'], _pc.document['fitspw'], _pc.document['fitorder'], _pc.document['want_cont'], _pc.document['denoising_lib'], _pc.document['nthreads'], _pc.document['niter'], _pc.document['disableparallel'], _pc.document['ddistart'], _pc.document['taql'], _pc.document['monolithic_processing'], _pc.document['reindex'] )
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_mstransform.py", line 307, in mstransform
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        mtlocal.open()
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/mstransformer.py", line 44, in open
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        return self._swigobj.open()
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/mstransformer.py", line 176, in open
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+        return _mstransformer.mstransformer_open(self)
2022-12-04 06:12:36 INFO    hif_mstransform::::casa+    RuntimeError: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated/working/uid___A002_Xfee03e_X2787.ms).
2022-12-04 06:12:36 INFO    hifa_imageprecheck::::casa  ##########################################

keflavich commented 1 year ago

Maybe the presence of CORRECTED_DATA is not enough of a check on its own? We should see if the data are calibrated.

I'm not sure the easiest way to do this. You might open the data in plotms and compare amp vs time between the DATA and CORRECTED_DATA columns. CORRECTED_DATA should be lower.

Or maybe we could do

ms.getdata('CORRECTED_DATA')['corrected_data'] / ms.getdata('DATA')['data']

and see what comes out...

keflavich commented 1 year ago

Yes, that works:

CASA <3>: datadict = ms.getdata(['DATA', 'CORRECTED_DATA'])

CASA <7>: (datadict['data'] / datadict['corrected_data']).max()
Out[7]: (1+0j)

CASA <8>: (datadict['data'] / datadict['corrected_data']).min()
Out[8]: (0.9999999999999999+0j)

the columns are identical. =(

keflavich commented 1 year ago

I tried the same on the latest version:


CASA <9>: ms.open('/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X10c/calibrated/working/uid___A002_Xfee03e_X2787.ms')
Out[9]: True

CASA <10>: ms.selectinit(2)
Out[10]: True

CASA <11>: datadict = ms.getdata(['DATA', 'CORRECTED_DATA'])
2023-01-18 13:44:45     WARN    ms::getdata     Cannot get requested column: corrected_data. Corrected column does not exist

and you see that there is no corrected_data column. So. Hrmph.

keflavich commented 1 year ago

https://data.rc.ufl.edu/secure/adamginsburg/ACES/weblogs-reimaging/member.uid___A001_X15a0_X10c/pipeline-20230114T050550/html/t2-4m.html?sidebar=sidebar_stage1&ms=all&subpage=t2-4m_details.html is an example failed pipeline run. https://data.rc.ufl.edu/secure/adamginsburg/ACES/weblogs-reimaging/member.uid___A001_X15a0_X10c/casa_log_mpi_pipeline_member.uid___A001_X15a0_X10c_55390308_2023-01-14_00_05_27.log is the command-line log.

keflavich commented 1 year ago

These are the ASDMs:

login4.ufhpc /orange/adamginsburg/ACES/data$ ls -lh *uid___A002_Xfed4ee_X1e3*
-rw-r--r-- 1 adamginsburg adamginsburg  60G Nov 15 22:40 2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 118G Oct 26 08:19 corrupt_2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
login4.ufhpc /orange/adamginsburg/ACES/data$ ls -lh *uid___A002_Xfee03e_X2787*
-rw-r--r-- 1 adamginsburg adamginsburg 60G Nov 16 02:37 2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 65G Oct 26 11:06 corrupt_2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar

showing that I re-downloaded in November. But they still don't work. So maybe it's not the ASDMs?

keflavich commented 1 year ago

/member.uid___A001_X15a0_X10c$ ls -lhrt raw/
total 8.0K
drwxr-xr-x+ 3 adamginsburg adamginsburg 4.0K Sep 24 20:57 uid___A002_Xfed4ee_X1e3.asdm.sdm
drwxr-xr-x+ 3 adamginsburg adamginsburg 4.0K Sep 26 19:27 uid___A002_Xfee03e_X2787.asdm.sdm

$ ls -lhrt *uid___A002_Xfee03e_X2787* *uid___A002_Xfed4ee_X1e3*
-rw-r--r-- 1 adamginsburg adamginsburg 62G Jan 18 14:07 2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 61G Jan 18 15:36 2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar

$ md5sum *uid___A002_Xfee03e_X2787* *uid___A002_Xfed4ee_X1e3*
0b7379e1373119e2000a4f8cf1c4819b  2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
a43ea10b31b9d132d17f0562207db93a  2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar

The cubes are junk:

$ ls -lhrtd calibrated/working/*.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Jan 18 02:07 calibrated/working/uid___A001_X15a0_X10c.s38_0.Sgr_A_star_sci.spw25.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Jan 18 02:07 calibrated/working/uid___A001_X15a0_X10c.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Jan 18 02:45 calibrated/working/uid___A001_X15a0_X10c.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Jan 18 03:16 calibrated/working/uid___A001_X15a0_X10c.s38_0.Sgr_A_star_sci.spw29.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Jan 18 22:55 calibrated/working/uid___A001_X15a0_X10c.s38_0.Sgr_A_star_sci.spw33.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Jan 18 23:57 calibrated/working/uid___A001_X15a0_X10c.s38_0.Sgr_A_star_sci.spw35.cube.I.iter1.image

I believe the asdm modification dates indicate that they were not re-extracted before re-running imaging, so that may be the explanation for the persistent corruption.

keflavich commented 1 year ago

Undergoing repipelining today. Not sure why it only triggered today as the last change I made was at least 5 days ago.

keflavich commented 1 year ago

Re-imaging resulted in junk. Removing everything and starting over again

keflavich commented 1 year ago

Restarting from:

$ ls -lhrt *uid___A002_Xfee03e_X2787* *uid___A002_Xfed4ee_X1e3* *X10c*
-rw-r--r-- 1 adamginsburg adamginsburg 3.5K Oct 25 17:13 member.uid___A001_X15a0_X10c.README.txt
-rw-r--r-- 1 adamginsburg adamginsburg 690M Jan 18 12:31 2021.1.00172.L_uid___A001_X15a0_X10c_auxiliary.tar
-rw-r--r-- 1 adamginsburg adamginsburg 147G Jan 18 12:31 2021.1.00172.L_uid___A001_X15a0_X10c_001_of_001.tar
-rw-r--r-- 1 adamginsburg adamginsburg  62G Jan 18 14:07 2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg  61G Jan 18 15:36 2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar

keflavich commented 1 year ago

Current state is below. Last imaging attempt failed. I removed the raw directory & calibrated again. Restarting.

$ ls -lh *_X10c_* *Xfed4ee_X1e3* *_X2787*
-rw-r--r-- 1 adamginsburg adamginsburg 147G Jan 18 12:31 2021.1.00172.L_uid___A001_X15a0_X10c_001_of_001.tar
-rw-r--r-- 1 adamginsburg adamginsburg 690M Jan 18 12:31 2021.1.00172.L_uid___A001_X15a0_X10c_auxiliary.tar
-rw-r--r-- 1 adamginsburg adamginsburg  60G Feb 16 01:48 2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg  60G Feb 16 04:03 2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar
$ md5sum *_X10c_* *Xfed4ee_X1e3* *_X2787*
4479a4daba3f62a49baefecab10e8284  2021.1.00172.L_uid___A001_X15a0_X10c_001_of_001.tar
448dcfcb48c1f59d8a7176dbb33476c9  2021.1.00172.L_uid___A001_X15a0_X10c_auxiliary.tar
7f699bcd2e75c1ed5737e75a720f98d0  2021.1.00172.L_uid___A002_Xfed4ee_X1e3.asdm.sdm.tar
61f181a22f41300c2f24c2811db11e77  2021.1.00172.L_uid___A002_Xfee03e_X2787.asdm.sdm.tar

keflavich commented 1 year ago

@qzhang99 can you try reimaging spw33?

pyhsiehATalma commented 1 year ago

@qzhang99 can you try reimaging spw33?

@qzhang99 I could work on reimaging.

keflavich commented 10 months ago

fix for divergence proposed in https://github.com/ACES-CMZ/reduction_ACES/pull/408/commits/8e68bc33dd4cc8b7230e87d0f5edb430a53d9fe3

keflavich commented 10 months ago

Apparently I decided to re-run the pipeline on this one two days ago. I'm not entirely sure why - I left a note saying I had done this on https://github.com/ACES-CMZ/reduction_ACES/issues/133, so maybe I reran on the wrong folder.

keflavich commented 10 months ago

No, more likely, I put the comment on the wrong issue: "The pipeline seems to have failed in the original run: there are no _target.ms files? I'm rerunning from scratch to see if that fixes it. Moved calibrated to broken_calibrated_20231216"

keflavich commented 10 months ago

The problem was that the .ms was incorrectly specified to be targets_line.ms, which is not produced by the pipeline.

pyhsiehATalma commented 8 months ago

截圖 2024-01-23 14 11 06

Hi, I am sorry for the slow progress, the re-imaging of spw 33 is running locally. It appears that the divergence could be fixed by changing some parameters of tclean.