Execution Block ID uid://A001/X15a0/X1a8 Sgr_A_st_as_03_TM1

keflavich commented 2 years ago

Sgr_A_st_as_03_TM1 uid://A001/X15a0/X1a8

[x] Observations completed?
[x] Delivered?
[x] Downloaded? (specify where)
- [x] hipergator
- [x] hipergator pipeline run
- Pipeline Run 20221019T162821
- Pipeline Run 20221020T043620
- Pipeline Run 20221019T180141
- Pipeline Run 20221001T040541
- Pipeline Run 20220930T225336
- Pipeline Run 20221001T002036
- Pipeline Run 20220704T125207
- Pipeline Run 20220702T053437
- Pipeline Run 20220702T040304
[ ] Weblog unpacked
[ ] Weblog Quality Assessment?
Extra Weblog Sgr_A_st_as_03_TM1_1 -> pipeline-20220704T125207, Extra Weblog Sgr_A_st_as_03_TM1_0 -> pipeline-20220608T153159, Extra Weblog Sgr_A_st_as_03_TM1_2 -> pipeline-20220704T125207
[x] Imaging: Continuum
[ ] Imaging: Lines (Images look good. Pending re-imaging due to size mitigation.)
- [ ] SPW25 H13CN
- [ ] SPW27 H13CO+/SiO
- [ ] SPW29 HCO+
- [ ] SPW31 HNCO
- [ ] SPW33 Cont 1
- [ ] SPW35 Cont 2

Product Links:

Reprocessed Product Links:

d-l-walker commented 2 years ago

No major issues. Needs to be reimaged as the data were binned by a factor of 2, and SPW 33 was dropped, both for size mitigation. Once this has been done I think this is good-to-go.

The continuum looks neat. I think this is cloud C, so we can see the chain of bright continuum sources, the brightest of which is the HMSFR. To the left of that is a known HII region which is not associated with the cloud. uid___A001_X15a0_X1a8 s36_0 Sgr_A_star_sci spw25_27_29_31_33_35 cont I iter1 image tt0 sky

keflavich commented 2 years ago

Were all the data binned by 2x? We need to add the corrected imaging parameters to https://github.com/ACES-CMZ/reduction_ACES/blob/main/pipeline_scripts/override_tclean_commands.json, then

keflavich commented 2 years ago

This is like field ao in that the reimaging was a total failure; it looks like uncalibrated data are being imaged.

keflavich commented 2 years ago

This is one of the fields affected by the issue in which _target.ms is never made.

d-l-walker commented 2 years ago

@keflavich any updates on this one? Is anything currently running? I can see that in the most recent run, no _target.ms files are made as you mentioned earlier. These should be calibrated, right, just no target splitting happening for some reason?

keflavich commented 2 years ago

it finished and is still entirely broken

keflavich commented 2 years ago

I think something is fundamentally wrong with the pipeline-delivered measurement set, but I haven't pinned it down.

keflavich commented 2 years ago

I have a real mystery on hand. The weblog shows that applycal was run:

2022-06-09 07:39:35 INFO    applycal::::casa    ##### Begin Task: applycal           #####
2022-06-09 07:39:35 INFO    applycal::::casa    applycal( vis='uid___A002_Xf96bbc_Xbcc0.ms', field='28,52,122,11,62,87,120,125,130,66,117,147,12,145,23,58,82,119,137,5,17,110,16,33,84,37,97,9,J1924-2914,128,133,3,10,53,131,26,6,101,79,13,60,J1744-3116,100,30,95,127,89,4,88,50,19,104,47,135,140,43,121,108,34,92,113,44,40,112,48,116,132,93,38,59,36,68,64,39,45,69,14,143,74,126,77,103,42,115,91,18,75,24,102,134,124,114,31,106,72,55,27,141,76,138,78,96,20,7,8,94,32,142,123,70,41,136,63,109,105,99,57,129,80,81,29,86,61,146,54,83,98,56,111,15,49,46,35,118,71,25,144,22,90,65,85,73,139,51,21,107,148,67', spw='25,27,29,31,33,35', intent='OBSERVE_TARGET#ON_SOURCE,CALIBRATE_BANDPASS#ON_SOURCE,CALIBRATE_FLUX#ON_SOURCE,CALIBRATE_PHASE#ON_SOURCE', selectdata=True, timerange='', uvrange='', antenna='*&*', scan='', observation='', msselect='', docallib=True, callib='uid___A002_Xf96bbc_Xbcc0.ms.s19.2.callibrary', gaintable=[], gainfield=[], interp=[], spwmap=[], calwt=[True], parang=False, applymode='calflagstrict', flagbackup=True )

successfully, but that is evidently false:

>>>tb.open('uid___A002_Xf96bbc_Xbcc0.ms')
>>>corr2 = tb.getcol("CORRECTED", nrow=100, rowincr=1000)
>>> tb.close()
2022-09-30 16:04:28 SEVERE  getcol::CORRECTED   Exception Reported: TableProxy::getColumn: column CORRECTED does not exist
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [18], in <cell line: 2>()
      1 tb.open('uid___A002_Xf96bbc_Xbcc0.ms')
----> 2 corr2 = tb.getcol("CORRECTED", nrow=100, rowincr=1000)
      3 tb.close()

File /orange/adamginsburg/miniconda3/envs/casapy38/lib/python3.8/site-packages/casatools/table.py:843, in table.getcol(self, columnname, startrow, nrow, rowincr)
    832 def getcol(self, columnname, startrow=int(0), nrow=int(-1), rowincr=int(1)):
    833     """The entire column (or part of it) is returned. Warning: it might be big!
    834     The functions can only be used if all arrays in the column have the
    835     same shape. That is guaranteed for columns containing scalars or fixed
   (...)
    841     shaped
    842     """
--> 843     return _any_dc(self._swigobj.getcol(_str_ec(columnname), startrow, nrow, rowincr))

File /orange/adamginsburg/miniconda3/envs/casapy38/lib/python3.8/site-packages/casatools/__casac__/table.py:2154, in table.getcol(self, *args, **kwargs)
   2115 def getcol(self, *args, **kwargs) -> "variant *":
   2116     """
   2117     getcol(self, _columnname, _startrow, _nrow, _rowincr) -> variant *
   2118 
   (...)
   2152 
   2153     """
-> 2154     return _table.table_getcol(self, *args, **kwargs)

RuntimeError: TableProxy::getColumn: column CORRECTED does not exist

keflavich commented 2 years ago

This is the problem:

3454 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa    ##########################################
3455 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa    Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
3456 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3457 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       result = self.prepare(**prepare_parameters)
3458 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
3459 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
3460 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
3461 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       self._executor.execute(task)
3462 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
3463 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       result = method(self, *args, **kw)
3464 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
3465 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       result = job.execute(dry_run=self._dry_run, **kwargs)
3466 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
3467 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       return self.fn(*self.args, **self.kw)
3468 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
3469 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
3470 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
3471 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       merge=merge)
3472 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
3473 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
3474 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
3475 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+       return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
3476 2022-07-02 05:46:54 SEVERE  hifa_restoredata::::casa+   RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
3477 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa    Traceback (most recent call last):
3478 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __call__
3479 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['rawdata_dir'],_invo
3480 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
3481 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bdfflags'], _pc.do
3482 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in hifa_restoredat
3483 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       results = utils.execute_task(context, task_name, all_inputs)
3484 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
3485 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       raise exceptions.PipelineException(previous_tracebacks_as_string)
3486 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+   pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
3487 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3488 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       result = self.prepare(**prepare_parameters)
3489 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
3490 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
3491 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
3492 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       self._executor.execute(task)
3493 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
3494 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       result = method(self, *args, **kw)
3495 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
3496 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       result = job.execute(dry_run=self._dry_run, **kwargs)
3497 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
3498 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       return self.fn(*self.args, **self.kw)
3499 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
3500 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
3501 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
3502 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       merge=merge)
3503 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
3504 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
3505 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
3506 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+       return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
3507 2022-07-02 05:46:54 INFO    hifa_restoredata::::casa+   RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)

keflavich commented 2 years ago

Second attempt resulted in:

3438 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa    Task hifa_restoredata complete. Start time: 2022-08-31 09:58:55.186016 End time: 2022-08-31 10:07:59.424530
3439 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa    ##### End Task: hifa_restoredata     #####
3440 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa    ##########################################
3441 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa    Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
3442 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_restore_caltables
3443 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+       pipemanifest.get_caltables(ouss)[session])
3444 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+   KeyError: 'session_2'
3445 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+
3446 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+   During handling of the above exception, another exception occurred:
3447 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+
3448 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+   Traceback (most recent call last):
3449 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3450 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+       result = self.prepare(**prepare_parameters)
3451 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepare
3452 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+       self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
3453 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_restore_caltables
3454 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+       pipemanifest.get_caltables(ouss)['default'])
3455 2022-08-31 14:07:59 SEVERE  hifa_restoredata::::casa+   KeyError: 'default'
3456 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa    Traceback (most recent call last):
3457 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __call__
3458 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['rawdata_dir'],_invo
3459 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
3460 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bdfflags'], _pc.do
3461 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in hifa_restoredat
3462 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       results = utils.execute_task(context, task_name, all_inputs)
3463 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
3464 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       raise exceptions.PipelineException(previous_tracebacks_as_string)
3465 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+   pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
3466 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_restore_caltables
3467 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       pipemanifest.get_caltables(ouss)[session])
3468 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+   KeyError: 'session_2'
3469 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+
3470 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+   During handling of the above exception, another exception occurred:
3471 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+
3472 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+   Traceback (most recent call last):
3473 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
3474 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       result = self.prepare(**prepare_parameters)
3475 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepare
3476 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
3477 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+     File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_restore_caltables
3478 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+       pipemanifest.get_caltables(ouss)['default'])
3479 2022-08-31 14:07:59 INFO    hifa_restoredata::::casa+   KeyError: 'default'
3480 2022-08-31 14:07:59 INFO    h_save::::casa  ##########################################

keflavich commented 2 years ago

I'm running a third attempt now.

keflavich commented 2 years ago

Failed with same result:

2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa     Error executing pipeline task hifa_restoredata.
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa     Traceback (most recent call last):
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata
6, in _do_restore_caltables
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+    KeyError: 'session_2'
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+    During handling of the above exception, another exception occurred:
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+    Traceback (most recent call last):
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", li
ecute
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata
9, in prepare
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata
9, in _do_restore_caltables
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28     SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+    KeyError: 'default'
2022-09-30 18:10:28     SEVERE  flagmanager::::casa     Task hifa_restoredata raised an exception of class PipelineException with the following message: Traceback (most recent call last):
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_restore_c
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+        pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+    KeyError: 'session_2'
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+    During handling of the above exception, another exception occurred:
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+    Traceback (most recent call last):
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+        result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepare
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+        self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_restore_c
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+        pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28     SEVERE  flagmanager::::casa+    KeyError: 'default'
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa        Task hifa_restoredata complete. Start time: 2022-09-30 13:54:46.762822 End time: 2022-09-30 14:10:28.474123
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa        ##### End Task: hifa_restoredata     #####
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa        ##########################################
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa        Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_r
les
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+           pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+       KeyError: 'session_2'
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+       During handling of the above exception, another exception occurred:
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+       Traceback (most recent call last):
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+           result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepa
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+           self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_r
les
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+           pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28     SEVERE  hifa_restoredata::::casa+       KeyError: 'default'
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa        Traceback (most recent call last):
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __c
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['raw
nvocation_parameters['lazy'],_invocation_parameters['bdfflags'],_invocation_parameters['ocorr_mode'],_invocation_parameters['asis'],_invocation_parameters['pipelinemode'],_invocation_parameters['dryrun'],_invocation_parameters['acceptresults'] )
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bd
.document['ocorr_mode'], _pc.document['asis'], _pc.document['pipelinemode'], _pc.document['dryrun'], _pc.document['acceptresults'] )
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in
data
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           results = utils.execute_task(context, task_name, all_inputs)
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           raise exceptions.PipelineException(previous_tracebacks_as_string)
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+       pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 536, in _do_r
les
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           pipemanifest.get_caltables(ouss)[session])
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+       KeyError: 'session_2'
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+       During handling of the above exception, another exception occurred:
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+       Traceback (most recent call last):
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           result = self.prepare(**prepare_parameters)
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 259, in prepa
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           self._do_restore_caltables(pipemanifest, session_names=session_names, session_vislists=session_vislists)
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+         File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 539, in _do_r
les
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+           pipemanifest.get_caltables(ouss)['default'])
2022-09-30 18:10:28     INFO    hifa_restoredata::::casa+       KeyError: 'default'

keflavich commented 2 years ago

@piposona @d-l-walker as people inside ALMA, is this a known error? Or should we try to raise a ticket about it?

I think this indicates a fundamental problem in the calibration tables.

piposona commented 2 years ago

Hi @keflavich , this is clearly not normal. We can either raise a ticket or as in previous cases, I can try to reproduce it if you want. My first guess, as usual is to blame the CASA version, which was CASA version: 6.2.1-7 for the delivered package. Will try to start downloading the data from globus, in case I can play with it.

keflavich commented 2 years ago

Perfect, yes. For the record, these are the tarfiles:

-rw-r--r-- 1 adamginsburg adamginsburg 773M Jun 21 02:20 2021.1.00172.L_uid___A001_X15a0_X1a8_auxiliary.tar
-rw-r--r-- 1 adamginsburg adamginsburg 171G Jun 23 05:30 2021.1.00172.L_uid___A001_X15a0_X1a8_001_of_001.tar
-rw-r--r-- 1 adamginsburg adamginsburg 3.5K Aug 23 01:21 member.uid___A001_X15a0_X1a8.README.txt

I'll post an md5sum when it's done.

piposona commented 2 years ago

Damn, the tarballs! I forgot, and I was downloading the directories!!! Ok, restarting the download. Will keep you posted.

keflavich commented 2 years ago

fbd7ee831c76df3968a9f00954384344  2021.1.00172.L_uid___A001_X15a0_X1a8_001_of_001.tar
12ac397a25b68f82117f9ad1ec17ec09  2021.1.00172.L_uid___A001_X15a0_X1a8_auxiliary.tar

piposona commented 2 years ago

I run the in CASA and found a different problem

2022-10-06 16:30:17 INFO    flagmanager::::casa ##########################################
2022-10-06 16:30:17 INFO    flagmanager::::casa ##### Begin Task: flagmanager        #####
2022-10-06 16:30:17 INFO    flagmanager::::casa flagmanager( vis='uid___A002_Xf96bbc_Xbcc0.ms', mode='restore', versionname='Pipeline_Final', oldname='', comment='', merge='replace' )
2022-10-06 16:30:17 INFO    flagmanager::AgentFlagger::open Table type is Measurement Set
2022-10-06 16:30:17 INFO    flagmanager::::casa Restore flagversions Pipeline_Final
2022-10-06 16:30:17 SEVERE  AgentFlagger::restoreFlagVersion (file casa-source/code/flagging/Flagging/AgentFlagger.cc, line 1001)   Could not restore Flag Version : ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 SEVERE  agentflagger:: (file binding/tools/agentflagger/agentflagger_cmpt.cc, line 35)  Exception Reported: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa Task flagmanager raised an exception of class RuntimeError with the following message: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO    flagmanager::::casa Task flagmanager complete. Start time: 2022-10-06 16:30:16.852123 End time: 2022-10-06 16:30:17.007405
2022-10-06 16:30:17 INFO    flagmanager::::casa ##### End Task: flagmanager          #####
2022-10-06 16:30:17 INFO    flagmanager::::casa ##########################################
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.h.tasks.restoredata.restoredata::casa Application of final flags failed for uid___A002_Xf96bbc_Xbcc0.ms
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa Error executing pipeline task hifa_restoredata.
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa Traceback (most recent call last):
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        self._executor.execute(task)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        merge=merge)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+        return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 SEVERE  flagmanager::pipeline.infrastructure.basetask::casa+    RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa Task hifa_restoredata raised an exception of class PipelineException with the following message: Traceback (most recent call last):
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        self._executor.execute(task)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        result = method(self, *args, **kw)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        merge=merge)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+      File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+        return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 SEVERE  flagmanager::::casa+    RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa    Task hifa_restoredata complete. Start time: 2022-10-06 16:19:16.180129 End time: 2022-10-06 16:30:17.331255
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa    ##### End Task: hifa_restoredata     #####
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa    ##########################################
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa    Exception Reported: Error in hifa_restoredata: Traceback (most recent call last):
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       self._executor.execute(task)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       result = method(self, *args, **kw)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       merge=merge)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+       return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 SEVERE  hifa_restoredata::::casa+   RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa    Traceback (most recent call last):
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/hifa/cli/gotasks/hifa_restoredata.py", line 571, in __call__
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       _return_result_ = _hifa_restoredata_t( _invocation_parameters['vis'],_invocation_parameters['session'],_invocation_parameters['products_dir'],_invocation_parameters['copytoraw'],_invocation_parameters['rawdata_dir'],_invocation_parameters['lazy'],_invocation_parameters['bdfflags'],_invocation_parameters['ocorr_mode'],_invocation_parameters['asis'],_invocation_parameters['pipelinemode'],_invocation_parameters['dryrun'],_invocation_parameters['acceptresults'] )
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/hifa/cli/hifa_restoredata.py", line 131, in __call__
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       task_result = _hifa_restoredata_t( _pc.document['vis'], _pc.document['session'], _pc.document['products_dir'], _pc.document['copytoraw'], _pc.document['rawdata_dir'], _pc.document['lazy'], _pc.document['bdfflags'], _pc.document['ocorr_mode'], _pc.document['asis'], _pc.document['pipelinemode'], _pc.document['dryrun'], _pc.document['acceptresults'] )
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/hifa/cli/private/task_hifa_restoredata.py", line 29, in hifa_restoredata
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       results = utils.execute_task(context, task_name, all_inputs)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/cli/utils.py", line 86, in execute_task
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       raise exceptions.PipelineException(previous_tracebacks_as_string)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+   pipeline.infrastructure.exceptions.PipelineException: Traceback (most recent call last):
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 660, in execute
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       result = self.prepare(**prepare_parameters)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 252, in prepare
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       self._do_restore_flags(pipemanifest, flag_version_name=flag_version_name)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/h/tasks/restoredata/restoredata.py", line 449, in _do_restore_flags
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       self._executor.execute(task)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       result = method(self, *args, **kw)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/basetask.py", line 818, in execute
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       result = job.execute(dry_run=self._dry_run, **kwargs)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/Pipeline-2021.2.0.128-py3.6.egg/pipeline/infrastructure/jobrequest.py", line 243, in execute
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       return self.fn(*self.args, **self.kw)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/flagmanager.py", line 96, in __call__
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       task_result = _flagmanager_t( _pc.document['vis'], _pc.document['mode'], _pc.document['versionname'], _pc.document['oldname'], _pc.document['comment'], _pc.document['merge'] )
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatasks/private/task_flagmanager.py", line 98, in flagmanager
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       merge=merge)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/agentflagger.py", line 83, in restoreflagversion
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       return self._swigobj.restoreflagversion(_str_ec(versionname), _str_ec(merge))
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+     File "/opt/casa_nrao/packages/RHEL7/release/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casatools/__casac__/agentflagger.py", line 365, in restoreflagversion
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+       return _agentflagger.agentflagger_restoreflagversion(self, *args, **kwargs)
2022-10-06 16:30:17 INFO    hifa_restoredata::::casa+   RuntimeError: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-10-06 16:30:17 INFO    h_save::::casa  ##########################################
2022-10-06 16:30:17 INFO    h_save::::casa  ##### Begin Task: h_save             #####
2022-10-06 16:30:17 INFO    h_save::::casa  h_save( filename='' )
2022-10-06 16:30:17 INFO    h_save::pipeline.infrastructure.launcher::casa  Saving context: pipeline-20221006T161916.context
2022-10-06 16:30:17 INFO    h_save::::casa  Task h_save complete. Start time: 2022-10-06 16:30:17.333957 End time: 2022-10-06 16:30:17.335131
2022-10-06 16:30:17 INFO    h_save::::casa  ##### End Task: h_save               #####
2022-10-06 16:30:17 INFO    h_save::::casa  ##########################################
2022-10-06 16:30:17 INFO    msmetadata_cmpt.cc::open    Performing internal consistency checks on uid___A002_Xf96bbc_X41f2.ms...
2022-10-06 16:30:30 INFO    msmetadata_cmpt.cc::open    Performing internal consistency checks on uid___A002_Xf96bbc_Xbcc0.ms...
2022-10-06 16:30:41 INFO    casa::scriptForPI::casa ALMA scriptForPI completed.
2022-10-06 17:25:56 INFO    casa::::casa    Terminating casaplotms, PID: 358491
2022-10-06 17:25:56 INFO    casa::::casa    Waiting for casaplotms to terminate. Poll return code: None, PID: 358491
2022-10-06 17:25:57 INFO    casa::::casa    Finished shutting down casaplotms. Poll return code: -15, PID: 358491

Just got this error so trying to figure out whether I downloaded the wrong data. These are the files I downloaded:

-rw-r--r-- 1 smartin science 183605459968 Oct  1 02:12 2021.1.00172.L_uid___A001_X15a0_X1a8_001_of_001.tar
-rw-r--r-- 1 smartin science    809844736 Oct  6 14:12 2021.1.00172.L_uid___A001_X15a0_X1a8_auxiliary.tar
-rw-r--r-- 1 smartin science  73580648448 Oct  6 14:19 2021.1.00172.L_uid___A002_Xf96bbc_X41f2.asdm.sdm.tar
-rw-r--r-- 1 smartin science  70536500224 Oct  6 14:08 2021.1.00172.L_uid___A002_Xf96bbc_Xbcc0.asdm.sdm.tar

which are the files you suggested @keflavich , plus the raw data files.

keflavich commented 2 years ago

OK so these data are borked.

I think we need a helpdesk ticket on this one?

piposona commented 2 years ago

I can ask around here but I would say so. I can try with your pipeline version just to triple check, but there is something wrong about it, definitely.

piposona commented 2 years ago

Actually, not, the 6.4.3-2 pipeline version does not exist in my Lustre. You must have some connections to high powers to get this installations :D

piposona commented 2 years ago

Confirmed with the people here. Although I will consult one of the experts, on these kind of problems the fastest way is to open a helpdesk.

keflavich commented 2 years ago

@snlongmore Could you raise this as a helpdesk ticket? In brief, we have a problem restoring the data for the SB listed in the header. We can provide more details later.

snlongmore commented 2 years ago

I have just raised a helpdesk ticket number 20676 and cc'd @keflavich and @ashleythomasbarnes.

piposona commented 2 years ago

Just contacted the right person now and he is looking into this. Still, the ticket was relevant.

piposona commented 2 years ago

Just some heads up, following the ticket with the responsible person here at JAO. Apparently the restoration worked ok for him but not for me...work in progress. He run CASA 6.4.1 while i did it in 6.2.1. I am currently running it again in 6.4.1. If it works...ok if it does not work I will check downloading the asdms again, in case there is a problem with the data themselves. Anyway, working on it, will keep reporting.

piposona commented 2 years ago

Ok here the summary of my tests:

Run pipelin in casa 6.2.1 did not work
Run pipeline in casa 6.4.1 did not work
Suggested by the person in charge of the filed ticket, i re-downloaded the ASDMs with asdmExport command which directly downloads the data from JAO archive. I then run the pipeline in casa 6.4.1 and IT DID WORK

So, somehow, the asdm data were corrupted in this case, causing the row mismatch error.

What should I do next? I could try to re-download the data from globus and try again, to triple-check if the problem is in the data stored there? I am still puzzled by the fact that the error I got was not the same as the one @keflavich got, but in any case the error KeyError: 'session_2' might be well related to the pipeline version used. The DMG people here recommended me to use 6.4.1 since this is the version used and tested in production and they cannot guarantee any result coming from later versions.

keflavich commented 2 years ago

Thanks @piposona. I'm totally removing all of the tarballs & extracted files and starting over.

piposona commented 2 years ago

Ok, for the time being I just started reprocessing to get the SPLIT datasets in case it is needed to get them uploaded.

keflavich commented 1 year ago

These data are still undergoing reimaging, but the pipeline seems to have worked further - it crashed in a tclean step, which implies that the .ms was created correctly. Still need to inspect the data, but looks like there's a good chance this one is OK now.

d-l-walker commented 1 year ago

@keflavich any update on this one? I'm trying to look through /member.uid___A001_X15a0_X1a8/calibrated/working/, and I don't think I can see all of the expected products. It looks like most of the cubes are present, but I don't see any for SPW 33? There are also a load of IMAGING_WEIGHT_XYZ files, which I'm guessing are either from a crashed run, or an ongoing one?

keflavich commented 1 year ago

Ongoing; spw33 is still imaging:

56732164                                 as_TM1_spw33_cube  astronomy-dept  astronomy-dept-b    RUNNING     4706     256G         64    108-07:07:44      1-16:36:41      4-00:00:00           c0704a-s26

keflavich commented 1 year ago

ah, no, something has gone horribly awry.

2023-02-06 21:22:47     INFO    task_tclean:::: Process 75822: waiting for write-lock on file /blue/adamginsburg/adamginsburg/ACES/workdir/as_spw33_cube_TM1_A001_X15a0_X1a8/uid___A001_X15a0_X1a8.s38_0.Sgr_A_star_sci.spw33.cube.I.iter1.model/table.lock

d-l-walker commented 1 year ago

@keflavich just checking in for a status update on this one. Were you able to fix the issues with the write-locks on this?

Looking at the directory on globus, we still don't have SPW 33, plus SPWs 25 & 27 appear to be size-mitigated still (SPWs 29, 31, and 35 all seem fine, judging by the cube file sizes).

keflavich commented 1 year ago

No, spw33 is still stuck on illegitimate write lock blocking. I think I'll have to do some major deletions and try again.

I've moved spw25 and spw27 to pre20230308 and triggered reimaging.

keflavich commented 1 year ago

Latest: spw33 has been idling forever. I'm going to try to reduce the number of MPI tasks and rerun. If that fails, I'll go back to single-thread, but with more memory, and hope that has some effect. If that still fails, I might need to do some actual coding to solve this one. 😠

keflavich commented 1 year ago

spw27 is advancing with errors like: Exception: Error (Resource deadlock avoided) when acquiring lock on /blue/adamginsburg/adamginsburg/ACES/workdir/as_spw27_cube_TM1_A001_X15a0_X1a8/uid___A001_X15a0_X1a8.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.residual/table.lock, but at least it seems to be doing something. It has 25 hours left on this run and is still doing major cycle 1, so let's see what happens.

d-l-walker commented 1 year ago

@keflavich I'm still not seeing the products for SPWs 25, 27, or 33 ... assuming none of them completed successfully?

keflavich commented 1 year ago

Correct. My recent PRs (#343, #344) were intended to try to fix or work around this problem, but I've determined significantly more work is necessary, or we need to jump to CASA v6.5, for those approaches to work.

keflavich commented 1 year ago

We have a successful clean of spw33, but there is still low-level divergence resulting in striping:

keflavich commented 1 year ago

In spw29, cleaning has artificially increased the flux in one channel. Here's an unmodeled channel:

and one with a model:

d-l-walker commented 1 year ago

@keflavich are there any updates or to-do items based on the points you raised above? Just trying to gauge what the status of this SB is. I can see that all SPWs have now been imaged, but I'm not sure whether these other issues were addressed yet?

keflavich commented 1 year ago

yeah, I'd argue that we need to reimage spw29 with different parameters.

d-l-walker commented 1 year ago

@keflavich I've increased the cyclefactor in PR #375 for SPW 33, if you could re-run for that SPW? I haven't changed anything for SPW 29 yet, as I'm not sure how we're going to address those parameter changes.

keflavich commented 1 year ago

Rerun triggered. Old files in 20230815_spw33

d-l-walker commented 1 year ago

@keflavich any idea what the status of this is? I don't see any newly-updates files since you triggered the rerun.

keflavich commented 1 year ago

Some subsets are still running:

7312991_2                            as_TM1_spw33_cube_arr  astronomy-dept  astronomy-dept-b    RUNNING     5097     128G         16      9-20:55:28        14:48:28      4-00:00:00 c33a-s36,c33b-s[23,+
7312991_3                            as_TM1_spw33_cube_arr  astronomy-dept  astronomy-dept-b    RUNNING     5097     128G         16      9-20:55:28        14:48:28      4-00:00:00 c34b-s[28,40],c35a-+
7312991_5                            as_TM1_spw33_cube_arr  astronomy-dept  astronomy-dept-b    RUNNING     5097     128G         16      9-20:55:28        14:48:28      4-00:00:00 c32a-s[3,19-20],c32+
7312991_7                            as_TM1_spw33_cube_arr  astronomy-dept  astronomy-dept-b    RUNNING     5097     128G         16      9-20:55:28        14:48:28      4-00:00:00 c29a-s41,c29b-s[23,+
7312991_9                            as_TM1_spw33_cube_arr  astronomy-dept  astronomy-dept-b    RUNNING     5097     128G         16      9-20:55:28        14:48:28      4-00:00:00     c32b-s[25-27,36]
7312991_26                           as_TM1_spw33_cube_arr  astronomy-dept  astronomy-dept-b    RUNNING     5097     128G         16      9-20:55:28        14:48:28      4-00:00:00        c24a-s[40-42]

d-l-walker commented 1 year ago

HPG clean of SPW 33 has finished. @keflavich can you or someone at UF look at this one? It's ~ 75 GB and will take ages to download on my connection 🙃

keflavich commented 1 year ago

@SpacialTree @nbudaiev @djeff1887 @abulatek who wants to have a look?

d-l-walker commented 1 year ago

I managed to download SPW 33 in the end. As far as I can see, the divergence seems to be fixed, though I don't know exactly where @keflavich saw it previously.

Nice spectrum at the HMSFR in cloud C too: Screenshot 2023-09-12 at 11 09 33

I think the only outstanding issue now is the channel-to-channel variation in SPW 29 that Adam noticed.

ACES-CMZ / reduction_ACES

Execution Block ID uid://A001/X15a0/X1a8 Sgr_A_st_as_03_TM1 #145

Product Links:

Reprocessed Product Links: