Open keflavich opened 2 years ago
No major issues. Needs to be reimaged as the data were binned by a factor of 2, and SPW 33 was dropped, both for size mitigation. Once this has been done I think this is good-to-go.
The continuum looks neat. I think this is cloud C, so we can see the chain of bright continuum sources, the brightest of which is the HMSFR. To the left of that is a known HII region which is not associated with the cloud.
Were all the data binned by 2x? We need to add the corrected imaging parameters to https://github.com/ACES-CMZ/reduction_ACES/blob/main/pipeline_scripts/override_tclean_commands.json, then
This is like field ao
in that the reimaging was a total failure; it looks like uncalibrated data are being imaged.
This is one of the fields affected by the issue in which _target.ms
is never made.
@keflavich any updates on this one? Is anything currently running? I can see that in the most recent run, no _target.ms
files are made as you mentioned earlier. These should be calibrated, right, just no target splitting happening for some reason?
it finished and is still entirely broken
I think something is fundamentally wrong with the pipeline-delivered measurement set, but I haven't pinned it down.
I have a real mystery on hand. The weblog shows that applycal
was run:
2022-06-09 07:39:35 INFO applycal::::casa ##### Begin Task: applycal #####
2022-06-09 07:39:35 INFO applycal::::casa applycal( vis='uid___A002_Xf96bbc_Xbcc0.ms', field='28,52,122,11,62,87,120,125,130,66,117,147,12,145,23,58,82,119,137,5,17,110,16,33,84,37,97,9,J1924-2914,128,133,3,10,53,131,26,6,101,79,13,60,J1744-3116,100,30,95,127,89,4,88,50,19,104,47,135,140,43,121,108,34,92,113,44,40,112,48,116,132,93,38,59,36,68,64,39,45,69,14,143,74,126,77,103,42,115,91,18,75,24,102,134,124,114,31,106,72,55,27,141,76,138,78,96,20,7,8,94,32,142,123,70,41,136,63,109,105,99,57,129,80,81,29,86,61,146,54,83,98,56,111,15,49,46,35,118,71,25,144,22,90,65,85,73,139,51,21,107,148,67', spw='25,27,29,31,33,35', intent='OBSERVE_TARGET#ON_SOURCE,CALIBRATE_BANDPASS#ON_SOURCE,CALIBRATE_FLUX#ON_SOURCE,CALIBRATE_PHASE#ON_SOURCE', selectdata=True, timerange='', uvrange='', antenna='*&*', scan='', observation='', msselect='', docallib=True, callib='uid___A002_Xf96bbc_Xbcc0.ms.s19.2.callibrary', gaintable=[], gainfield=[], interp=[], spwmap=[], calwt=[True], parang=False, applymode='calflagstrict', flagbackup=True )
successfully, but that is evidently false:
>>>tb.open('uid___A002_Xf96bbc_Xbcc0.ms')
>>>corr2 = tb.getcol("CORRECTED", nrow=100, rowincr=1000)
>>> tb.close()
2022-09-30 16:04:28 SEVERE getcol::CORRECTED Exception Reported: TableProxy::getColumn: column CORRECTED does not exist
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Input In [18], in <cell line: 2>()
1 tb.open('uid___A002_Xf96bbc_Xbcc0.ms')
----> 2 corr2 = tb.getcol("CORRECTED", nrow=100, rowincr=1000)
3 tb.close()
File /orange/adamginsburg/miniconda3/envs/casapy38/lib/python3.8/site-packages/casatools/table.py:843, in table.getcol(self, columnname, startrow, nrow, rowincr)
832 def getcol(self, columnname, startrow=int(0), nrow=int(-1), rowincr=int(1)):
833 """The entire column (or part of it) is returned. Warning: it might be big!
834 The functions can only be used if all arrays in the column have the
835 same shape. That is guaranteed for columns containing scalars or fixed
(...)
841 shaped
842 """
--> 843 return _any_dc(self._swigobj.getcol(_str_ec(columnname), startrow, nrow, rowincr))
File /orange/adamginsburg/miniconda3/envs/casapy38/lib/python3.8/site-packages/casatools/__casac__/table.py:2154, in table.getcol(self, *args, **kwargs)
2115 def getcol(self, *args, **kwargs) -> "variant *":
2116 """
2117 getcol(self, _columnname, _startrow, _nrow, _rowincr) -> variant *
2118
(...)
2152
2153 """
-> 2154 return _table.table_getcol(self, *args, **kwargs)
RuntimeError: TableProxy::getColumn: column CORRECTED does not exist
This is the problem:
3454 2022-07-02 05:46:54 INFO hifa_restoredata::::casa ##########################################
3455 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
3456 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3457 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
3458 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
3459 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
3460 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
3461 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ self._executor.execute(task)
3462 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
3463 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ result = method(self, *args, **kw)
3464 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
3465 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ result = job.execute(dry_run=self._dry_run, **kwargs)
3466 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
3467 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ return self.fn(*self.args, **self.kw)
3468 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
3469 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
3470 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
3471 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ merge=merge)
3472 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
3473 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
3474 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
3475 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
3476 2022-07-02 05:46:54 SEVERE hifa_restoredata::::casa+ RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
3477 2022-07-02 05:46:54 INFO hifa_restoredata::::casa Traceback (most recent call last):
3478 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __call__
3479 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['rawdata_dir'],_invo
3480 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
3481 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bdfflags'], _pc.do
3482 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in hifa_restoredat
3483 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ results = utils.execute_task(context, task_name, all_inputs)
3484 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
3485 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ raise exceptions.PipelineException(previous_tracebacks_as_string)
3486 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
3487 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3488 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
3489 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
3490 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
3491 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
3492 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ self._executor.execute(task)
3493 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
3494 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ result = method(self, *args, **kw)
3495 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
3496 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ result = job.execute(dry_run=self._dry_run, **kwargs)
3497 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
3498 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ return self.fn(*self.args, **self.kw)
3499 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
3500 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
3501 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
3502 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ merge=merge)
3503 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
3504 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
3505 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
3506 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
3507 2022-07-02 05:46:54 INFO hifa_restoredata::::casa+ RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
Second attempt resulted in:
3438 2022-08-31 14:07:59 INFO hifa_restoredata::::casa Task hifa_restoredata complete. Start time: 2022-08-31 09:58:55.186016 End time: 2022-08-31 10:07:59.424530
3439 2022-08-31 14:07:59 INFO hifa_restoredata::::casa ##### End Task: hifa_restoredata #####
3440 2022-08-31 14:07:59 INFO hifa_restoredata::::casa ##########################################
3441 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
3442 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_restore_caltables
3443 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)[session])
3444 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ KeyError: 'session_2'
3445 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+
3446 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ During handling of the above exception, another exception occurred:
3447 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+
3448 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ Traceback (most recent call last):
3449 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3450 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
3451 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepare
3452 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
3453 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_restore_caltables
3454 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)['default'])
3455 2022-08-31 14:07:59 SEVERE hifa_restoredata::::casa+ KeyError: 'default'
3456 2022-08-31 14:07:59 INFO hifa_restoredata::::casa Traceback (most recent call last):
3457 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __call__
3458 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['rawdata_dir'],_invo
3459 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
3460 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bdfflags'], _pc.do
3461 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in hifa_restoredat
3462 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ results = utils.execute_task(context, task_name, all_inputs)
3463 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
3464 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ raise exceptions.PipelineException(previous_tracebacks_as_string)
3465 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
3466 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_restore_caltables
3467 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)[session])
3468 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ KeyError: 'session_2'
3469 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+
3470 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ During handling of the above exception, another exception occurred:
3471 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+
3472 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ Traceback (most recent call last):
3473 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3474 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
3475 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepare
3476 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
3477 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_restore_caltables
3478 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)['default'])
3479 2022-08-31 14:07:59 INFO hifa_restoredata::::casa+ KeyError: 'default'
3480 2022-08-31 14:07:59 INFO h_save::::casa ##########################################
I'm running a third attempt now.
Failed with same result:
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa Error executing pipeline task hifa_restoredata.
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa Traceback (most recent call last):
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata
6, in _do_restore_caltables
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ KeyError: 'session_2'
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ During handling of the above exception, another exception occurred:
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ Traceback (most recent call last):
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", li
ecute
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata
9, in prepare
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata
9, in _do_restore_caltables
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ KeyError: 'default'
2022-09-30 18:10:28 SEVERE flagmanager::::casa Task hifa_restoredata raised an exception of class PipelineException with the following message: Traceback (most recent call last):
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_restore_c
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ KeyError: 'session_2'
2022-09-30 18:10:28 SEVERE flagmanager::::casa+
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ During handling of the above exception, another exception occurred:
2022-09-30 18:10:28 SEVERE flagmanager::::casa+
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ Traceback (most recent call last):
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepare
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_restore_c
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28 SEVERE flagmanager::::casa+ KeyError: 'default'
2022-09-30 18:10:28 INFO hifa_restoredata::::casa Task hifa_restoredata complete. Start time: 2022-09-30 13:54:46.762822 End time: 2022-09-30 14:10:28.474123
2022-09-30 18:10:28 INFO hifa_restoredata::::casa ##### End Task: hifa_restoredata #####
2022-09-30 18:10:28 INFO hifa_restoredata::::casa ##########################################
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_r
les
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ KeyError: 'session_2'
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ During handling of the above exception, another exception occurred:
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ Traceback (most recent call last):
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepa
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_r
les
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28 SEVERE hifa_restoredata::::casa+ KeyError: 'default'
2022-09-30 18:10:28 INFO hifa_restoredata::::casa Traceback (most recent call last):
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __c
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['raw
nvocation_parameters['lazy'],_invocation_parameters['bdfflags'],_invocation_parameters['ocorr_mode'],_invocation_parameters['asis'],_invocation_parameters['pipelinemode'],_invocation_parameters['dryrun'],_invocation_parameters['acceptresults'] )
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bd
.document['ocorr_mode'], _pc.document['asis'], _pc.document['pipelinemode'], _pc.document['dryrun'], _pc.document['acceptresults'] )
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in
data
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ results = utils.execute_task(context, task_name, all_inputs)
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ raise exceptions.PipelineException(previous_tracebacks_as_string)
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_r
les
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ KeyError: 'session_2'
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ During handling of the above exception, another exception occurred:
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ Traceback (most recent call last):
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepa
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_r
les
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28 INFO hifa_restoredata::::casa+ KeyError: 'default'
@piposona @d-l-walker as people inside ALMA, is this a known error? Or should we try to raise a ticket about it?
I think this indicates a fundamental problem in the calibration tables.
Hi @keflavich , this is clearly not normal. We can either raise a ticket or as in previous cases, I can try to reproduce it if you want. My first guess, as usual is to blame the CASA version, which was CASA version: 6.2.1-7 for the delivered package. Will try to start downloading the data from globus, in case I can play with it.
Perfect, yes. For the record, these are the tarfiles:
-rw-r--r-- 1 adamginsburg adamginsburg 773M Jun 21 02:20 2021.1.00172.L_uid___A001_X15a0_X1a8_auxiliary.tar
-rw-r--r-- 1 adamginsburg adamginsburg 171G Jun 23 05:30 2021.1.00172.L_uid___A001_X15a0_X1a8_001_of_001.tar
-rw-r--r-- 1 adamginsburg adamginsburg 3.5K Aug 23 01:21 member.uid___A001_X15a0_X1a8.README.txt
I'll post an md5sum
when it's done.
Damn, the tarballs! I forgot, and I was downloading the directories!!! Ok, restarting the download. Will keep you posted.
fbd7ee831c76df3968a9f00954384344 2021.1.00172.L_uid___A001_X15a0_X1a8_001_of_001.tar
12ac397a25b68f82117f9ad1ec17ec09 2021.1.00172.L_uid___A001_X15a0_X1a8_auxiliary.tar
I run the in CASA and found a different problem
2022-10-06 16:30:17 INFO flagmanager::::casa ##########################################
2022-10-06 16:30:17 INFO flagmanager::::casa ##### Begin Task: flagmanager #####
2022-10-06 16:30:17 INFO flagmanager::::casa flagmanager( vis='uid___A002_Xf96bbc_Xbcc0.ms', mode='restore', versionname='Pipeline_Final', oldname='', comment='', merge='replace' )
2022-10-06 16:30:17 INFO flagmanager::AgentFlagger::open Table type is Measurement Set
2022-10-06 16:30:17 INFO flagmanager::::casa Restore flagversions Pipeline_Final
2022-10-06 16:30:17 SEVERE AgentFlagger::restoreFlagVersion (file casa-source/code/flagging/Flagging/AgentFlagger.cc, line 1001) Could not restore Flag Version : ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 SEVERE agentflagger:: (file binding/tools/agentflagger/agentflagger_cmpt.cc, line 35) Exception Reported: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 SEVERE flagmanager::::casa Task flagmanager raised an exception of class RuntimeError with the following message: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO flagmanager::::casa Task flagmanager complete. Start time: 2022-10-06 16:30:16.852123 End time: 2022-10-06 16:30:17.007405
2022-10-06 16:30:17 INFO flagmanager::::casa ##### End Task: flagmanager #####
2022-10-06 16:30:17 INFO flagmanager::::casa ##########################################
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.h.tasks.restoredata.restoredata::casa Application of final flags failed for uid___A002_Xf96bbc_Xbcc0.ms
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa Error executing pipeline task hifa_restoredata.
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa Traceback (most recent call last):
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ self._executor.execute(task)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ result = method(self, *args, **kw)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ merge=merge)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 SEVERE flagmanager::pipeline.infrastructure.basetask::casa+ RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 SEVERE flagmanager::::casa Task hifa_restoredata raised an exception of class PipelineException with the following message: Traceback (most recent call last):
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ self._executor.execute(task)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ result = method(self, *args, **kw)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ merge=merge)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 SEVERE flagmanager::::casa+ RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa Task hifa_restoredata complete. Start time: 2022-10-06 16:19:16.180129 End time: 2022-10-06 16:30:17.331255
2022-10-06 16:30:17 INFO hifa_restoredata::::casa ##### End Task: hifa_restoredata #####
2022-10-06 16:30:17 INFO hifa_restoredata::::casa ##########################################
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ self._executor.execute(task)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ result = method(self, *args, **kw)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ merge=merge)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 SEVERE hifa_restoredata::::casa+ RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa Traceback (most recent call last):
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __call__
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['rawdata_dir'],_invocation_parameters['lazy'],_invocation_parameters['bdfflags'],_invocation_parameters['ocorr_mode'],_invocation_parameters['asis'],_invocation_parameters['pipelinemode'],_invocation_parameters['dryrun'],_invocation_parameters['acceptresults'] )
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bdfflags'], _pc.document['ocorr_mode'], _pc.document['asis'], _pc.document['pipelinemode'], _pc.document['dryrun'], _pc.document['acceptresults'] )
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in hifa_restoredata
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ results = utils.execute_task(context, task_name, all_inputs)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ raise exceptions.PipelineException(previous_tracebacks_as_string)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ self._executor.execute(task)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ result = method(self, *args, **kw)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ merge=merge)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 INFO hifa_restoredata::::casa+ RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO h_save::::casa ##########################################
2022-10-06 16:30:17 INFO h_save::::casa ##### Begin Task: h_save #####
2022-10-06 16:30:17 INFO h_save::::casa h_save( filename='' )
2022-10-06 16:30:17 INFO h_save::pipeline.infrastructure.launcher::casa Saving context: pipeline-20221006T161916.context
2022-10-06 16:30:17 INFO h_save::::casa Task h_save complete. Start time: 2022-10-06 16:30:17.333957 End time: 2022-10-06 16:30:17.335131
2022-10-06 16:30:17 INFO h_save::::casa ##### End Task: h_save #####
2022-10-06 16:30:17 INFO h_save::::casa ##########################################
2022-10-06 16:30:17 INFO msmetadata_cmpt.cc::open Performing internal consistency checks on uid___A002_Xf96bbc_X41f2.ms...
2022-10-06 16:30:30 INFO msmetadata_cmpt.cc::open Performing internal consistency checks on uid___A002_Xf96bbc_Xbcc0.ms...
2022-10-06 16:30:41 INFO casa::scriptForPI::casa ALMA scriptForPI completed.
2022-10-06 17:25:56 INFO casa::::casa Terminating casaplotms, PID: 358491
2022-10-06 17:25:56 INFO casa::::casa Waiting for casaplotms to terminate. Poll return code: None, PID: 358491
2022-10-06 17:25:57 INFO casa::::casa Finished shutting down casaplotms. Poll return code: -15, PID: 358491
Just got this error so trying to figure out whether I downloaded the wrong data. These are the files I downloaded:
-rw-r--r-- 1 smartin science 183605459968 Oct 1 02:12 2021.1.00172.L_uid___A001_X15a0_X1a8_001_of_001.tar
-rw-r--r-- 1 smartin science 809844736 Oct 6 14:12 2021.1.00172.L_uid___A001_X15a0_X1a8_auxiliary.tar
-rw-r--r-- 1 smartin science 73580648448 Oct 6 14:19 2021.1.00172.L_uid___A002_Xf96bbc_X41f2.asdm.sdm.tar
-rw-r--r-- 1 smartin science 70536500224 Oct 6 14:08 2021.1.00172.L_uid___A002_Xf96bbc_Xbcc0.asdm.sdm.tar
which are the files you suggested @keflavich , plus the raw data files.
OK so these data are borked.
I think we need a helpdesk ticket on this one?
I can ask around here but I would say so. I can try with your pipeline version just to triple check, but there is something wrong about it, definitely.
Actually, not, the 6.4.3-2 pipeline version does not exist in my Lustre. You must have some connections to high powers to get this installations :D
Confirmed with the people here. Although I will consult one of the experts, on these kind of problems the fastest way is to open a helpdesk.
@snlongmore Could you raise this as a helpdesk ticket? In brief, we have a problem restoring the data for the SB listed in the header. We can provide more details later.
I have just raised a helpdesk ticket number 20676 and cc'd @keflavich and @ashleythomasbarnes.
Just contacted the right person now and he is looking into this. Still, the ticket was relevant.
Just some heads up, following the ticket with the responsible person here at JAO. Apparently the restoration worked ok for him but not for me...work in progress. He run CASA 6.4.1 while i did it in 6.2.1. I am currently running it again in 6.4.1. If it works...ok if it does not work I will check downloading the asdms again, in case there is a problem with the data themselves. Anyway, working on it, will keep reporting.
Ok here the summary of my tests:
So, somehow, the asdm data were corrupted in this case, causing the row mismatch error.
What should I do next? I could try to re-download the data from globus and try again, to triple-check if the problem is in the data stored there?
I am still puzzled by the fact that the error I got was not the same as the one @keflavich got, but in any case the error
KeyError: 'session_2'
might be well related to the pipeline version used. The DMG people here recommended me to use 6.4.1 since this is the version used and tested in production and they cannot guarantee any result coming from later versions.
Thanks @piposona. I'm totally removing all of the tarballs & extracted files and starting over.
Ok, for the time being I just started reprocessing to get the SPLIT datasets in case it is needed to get them uploaded.
These data are still undergoing reimaging, but the pipeline seems to have worked further - it crashed in a tclean step, which implies that the .ms was created correctly. Still need to inspect the data, but looks like there's a good chance this one is OK now.
@keflavich any update on this one? I'm trying to look through /member.uid___A001_X15a0_X1a8/calibrated/working/
, and I don't think I can see all of the expected products. It looks like most of the cubes are present, but I don't see any for SPW 33? There are also a load of IMAGING_WEIGHT_XYZ
files, which I'm guessing are either from a crashed run, or an ongoing one?
Ongoing; spw33 is still imaging:
56732164 as_TM1_spw33_cube astronomy-dept astronomy-dept-b RUNNING 4706 256G 64 108-07:07:44 1-16:36:41 4-00:00:00 c0704a-s26
ah, no, something has gone horribly awry.
2023-02-06 21:22:47 INFO task_tclean:::: Process 75822: waiting for write-lock on file /blue/adamginsburg/adamginsburg/ACES/workdir/as_spw33_cube_TM1_A001_X15a0_X1a8/uid___A001_X15a0_X1a8.s38_0.Sgr_A_star_sci.spw33.cube.I.iter1.model/table.lock
@keflavich just checking in for a status update on this one. Were you able to fix the issues with the write-locks on this?
Looking at the directory on globus, we still don't have SPW 33, plus SPWs 25 & 27 appear to be size-mitigated still (SPWs 29, 31, and 35 all seem fine, judging by the cube file sizes).
No, spw33 is still stuck on illegitimate write lock blocking. I think I'll have to do some major deletions and try again.
I've moved spw25 and spw27 to pre20230308
and triggered reimaging.
Latest: spw33 has been idling forever. I'm going to try to reduce the number of MPI tasks and rerun. If that fails, I'll go back to single-thread, but with more memory, and hope that has some effect. If that still fails, I might need to do some actual coding to solve this one. ðŸ˜
spw27 is advancing with errors like:
Exception: Error (Resource deadlock avoided) when acquiring lock on /blue/adamginsburg/adamginsburg/ACES/workdir/as_spw27_cube_TM1_A001_X15a0_X1a8/uid___A001_X15a0_X1a8.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.residual/table.lock
, but at least it seems to be doing something. It has 25 hours left on this run and is still doing major cycle 1, so let's see what happens.
@keflavich I'm still not seeing the products for SPWs 25, 27, or 33 ... assuming none of them completed successfully?
Correct. My recent PRs (#343, #344) were intended to try to fix or work around this problem, but I've determined significantly more work is necessary, or we need to jump to CASA v6.5, for those approaches to work.
We have a successful clean of spw33, but there is still low-level divergence resulting in striping:
In spw29, cleaning has artificially increased the flux in one channel. Here's an unmodeled channel:
and one with a model:
@keflavich are there any updates or to-do items based on the points you raised above? Just trying to gauge what the status of this SB is. I can see that all SPWs have now been imaged, but I'm not sure whether these other issues were addressed yet?
yeah, I'd argue that we need to reimage spw29 with different parameters.
@keflavich I've increased the cyclefactor in PR #375 for SPW 33, if you could re-run for that SPW? I haven't changed anything for SPW 29 yet, as I'm not sure how we're going to address those parameter changes.
Rerun triggered. Old files in 20230815_spw33
@keflavich any idea what the status of this is? I don't see any newly-updates files since you triggered the rerun.
Some subsets are still running:
7312991_2 as_TM1_spw33_cube_arr astronomy-dept astronomy-dept-b RUNNING 5097 128G 16 9-20:55:28 14:48:28 4-00:00:00 c33a-s36,c33b-s[23,+
7312991_3 as_TM1_spw33_cube_arr astronomy-dept astronomy-dept-b RUNNING 5097 128G 16 9-20:55:28 14:48:28 4-00:00:00 c34b-s[28,40],c35a-+
7312991_5 as_TM1_spw33_cube_arr astronomy-dept astronomy-dept-b RUNNING 5097 128G 16 9-20:55:28 14:48:28 4-00:00:00 c32a-s[3,19-20],c32+
7312991_7 as_TM1_spw33_cube_arr astronomy-dept astronomy-dept-b RUNNING 5097 128G 16 9-20:55:28 14:48:28 4-00:00:00 c29a-s41,c29b-s[23,+
7312991_9 as_TM1_spw33_cube_arr astronomy-dept astronomy-dept-b RUNNING 5097 128G 16 9-20:55:28 14:48:28 4-00:00:00 c32b-s[25-27,36]
7312991_26 as_TM1_spw33_cube_arr astronomy-dept astronomy-dept-b RUNNING 5097 128G 16 9-20:55:28 14:48:28 4-00:00:00 c24a-s[40-42]
HPG clean of SPW 33 has finished. @keflavich can you or someone at UF look at this one? It's ~ 75 GB and will take ages to download on my connection 🙃
@SpacialTree @nbudaiev @djeff1887 @abulatek who wants to have a look?
I managed to download SPW 33 in the end. As far as I can see, the divergence seems to be fixed, though I don't know exactly where @keflavich saw it previously.
Nice spectrum at the HMSFR in cloud C too:
I think the only outstanding issue now is the channel-to-channel variation in SPW 29 that Adam noticed.
Sgr_A_st_as_03_TM1 uid://A001/X15a0/X1a8
[x] Observations completed?
[x] Delivered?
[x] Downloaded? (specify where)
[ ] Weblog unpacked
[ ] Weblog Quality Assessment?
Extra Weblog Sgr_A_st_as_03_TM1_1 -> pipeline-20220704T125207, Extra Weblog Sgr_A_st_as_03_TM1_0 -> pipeline-20220608T153159, Extra Weblog Sgr_A_st_as_03_TM1_2 -> pipeline-20220704T125207
[x] Imaging: Continuum
[ ] Imaging: Lines (Images look good. Pending re-imaging due to size mitigation.)
Product Links:
Reprocessed Product Links: