Execution Block ID uid://A001/X15a0/X154 Sgr_A_st_ae_03_TM1

keflavich commented 2 years ago

Sgr_A_st_ae_03_TM1 uid://A001/X15a0/X154

[x] Observations completed?
[x] Delivered?
[x] Downloaded? (specify where)
- [x] hipergator
- [x] hipergator pipeline run
- Pipeline Run 20230209T140206
- Pipeline Run 20230209T152149
- Pipeline Run 20230210T021804
- Pipeline Run 20230208T021256
- Pipeline Run 20230207T180115
- Pipeline Run 20230207T192651
- Pipeline Run 20221202T182226
- Pipeline Run 20221202T201323
- Pipeline Run 20221202T194101
- Pipeline Run 20220701T040328
- Pipeline Run 20220630T050014
- Pipeline Run 20220630T040406
[ ] Weblog unpacked
[x] Weblog Quality Assessment?
Extra Weblog Sgr_A_st_ae_03_TM1_1 -> pipeline-20220701T040328, Extra Weblog Sgr_A_st_ae_03_TM1_0 -> pipeline-20220603T155839
[x] Imaging: Continuum
[x] Imaging: Lines
- [x] SPW25 H13CN
- [x] SPW27 H13CO+/SiO
- [x] SPW29 HCO+
- [x] SPW31 HNCO
- [x] SPW33 Cont 1
- [x] SPW35 Cont 2

Product Links:

Reprocessed Product Links:

d-l-walker commented 2 years ago

Data look good. Once PL has been re-run to image SPW 33, this should be marked as 'Done'. A few minor points to note:

SPW 33 dropped for size mitigation.
There's some emission in the line-free moment 8 maps for SPW 25 & 29, but it's not too significant.
Some high phases on DA48 for EB X1947 (stage 17), but final calibration looks fine, so this isn't a problem.

d-l-walker commented 2 years ago

Looks like SPW 33 has been cleaned successfully now, so I'm setting this as 'Done'.

d-l-walker commented 2 years ago

Removed the 'Done' label from this. SPW 33 was cleaned, but the data look uncalibrated (see #279).

keflavich commented 2 years ago

I deleted the 'calibrated' directory and am re-running the pipeline.

keflavich commented 2 years ago

the failures included:

2022-07-01 04:31:05     SEVERE  mstransform::::casa     Task mstransform raised an exception of class RuntimeError with the following message: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X154/calibrated/working/uid___A002_Xf8f6a9_Xe40.ms).
2022-07-01 04:31:05     INFO    mstransform::::casa     Task mstransform complete. Start time: 2022-07-01 00:31:04.752545 End time: 2022-07-01 00:31:05.006818

2022-06-30 05:07:41     INFO    flagmanager::::casa     ##########################################
2022-06-30 05:07:41     INFO    flagmanager::::casa     ##### Begin Task: flagmanager        #####
2022-06-30 05:07:41     INFO    flagmanager::::casa     flagmanager( vis='uid___A002_Xf8f6a9_X1947.ms', mode='restore', versionname='Pipeline_Final', oldname='', comment='', merge='replace' )
2022-06-30 05:07:41     INFO    flagmanager::AgentFlagger::open Table type is Measurement Set
2022-06-30 05:07:41     INFO    flagmanager::::casa     Restore flagversions Pipeline_Final
2022-06-30 05:07:41     SEVERE  AgentFlagger::restoreFlagVersion (file src/code/flagging/Flagging/AgentFlagger.cc, line 1001)   Could not restore Flag Version : ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-06-30 05:07:42     SEVERE  agentflagger:: (file src/tools/agentflagger/agentflagger_cmpt.cc, line 35)      Exception Reported: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-06-30 05:07:42     SEVERE  flagmanager::::casa     Task flagmanager raised an exception of class RuntimeError with the following message: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2022-06-30 05:07:42     INFO    flagmanager::::casa     Task flagmanager complete. Start time: 2022-06-30 01:07:41.065736 End time: 2022-06-30 01:07:41.806284
2022-06-30 05:07:42     INFO    flagmanager::::casa     ##### End Task: flagmanager          #####
2022-06-30 05:07:42     INFO    flagmanager::::casa     ##########################################

Both of these indicate fundamentally corrupted data, so a rerun is necessary, and maybe a re-download. We'll see.

keflavich commented 1 year ago

Re-pipeline failure:

2022-12-02 20:48:39     INFO    mstransform::::casa     ##########################################
2022-12-02 20:48:39     INFO    mstransform::::casa     ##### Begin Task: mstransform        #####
2022-12-02 20:48:39     INFO    mstransform::::casa     mstransform( vis='uid___A002_Xf8f6a9_Xe40.ms', outputvis='uid___A002_Xf8f6a9_Xe40_target.ms', createmms=False, separationaxis='auto', numsubms='auto', t
ileshape=[0], field='3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67
,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127
,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146', spw='25,27,29,31,33,35', scan='', antenna='', correlation='', timerange='', intent='OBSERVE_TARGET#ON_SOURCE', array='', uvrange=
'', observation='', feed='', datacolumn='corrected', realmodelcol=False, keepflags=True, usewtspectrum=False, combinespws=False, chanaverage=False, chanbin=1, hanning=False, regridms=False, mode='channel', nc
han=-1, start=0, width=1, nspw=1, interpolation='linear', phasecenter='', restfreq='', outframe='', veltype='radio', preaverage=False, timeaverage=False, timebin='0s', timespan='', maxuvwdistance=0.0, docalli
b=False, callib='', douvcontsub=False, fitspw='', fitorder=0, want_cont=False, denoising_lib=True, nthreads=1, niter=1, disableparallel=False, ddistart=-1, taql='', monolithic_processing=False, reindex=False
)
2022-12-02 20:48:39     INFO    MSTransformManager::parseMsSpecParams   Input file name is uid___A002_Xf8f6a9_Xe40.ms
2022-12-02 20:48:39     INFO    MSTransformManager::parseMsSpecParams   Data column is CORRECTED
2022-12-02 20:48:39     INFO    MSTransformManager::parseMsSpecParams   Output file name is uid___A002_Xf8f6a9_Xe40_target.ms
2022-12-02 20:48:39     INFO    MSTransformManager::parseMsSpecParams   Re-index is disabled
2022-12-02 20:48:39     INFO    MSTransformManager::parseMsSpecParams   Tile shape is [0]
2022-12-02 20:48:39     INFO    MSTransformManager::parseDataSelParams  field selection is 3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,4
4,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,1
10,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146
2022-12-02 20:48:39     INFO    MSTransformManager::parseDataSelParams  spw selection is 25,27,29,31,33,35
2022-12-02 20:48:39     INFO    MSTransformManager::parseDataSelParams  scan intent selection is OBSERVE_TARGET#ON_SOURCE
2022-12-02 20:48:39     WARN    MSTransformManager::checkDataColumnsToFill      CORRECTED_DATA column requested but not available in input MS
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams     Selected SPWs Ids are Axis Lengths: [6, 4]  (NB: Matrix in Row/Column order)
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams+    [25, 0, 1919, 1
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams+     27, 0, 1919, 1
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams+     29, 0, 1919, 1
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams+     31, 0, 1919, 1
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams+     33, 0, 3839, 1
2022-12-02 20:48:39     INFO    MSTransformManager::initDataSelectionParams+     35, 0, 3839, 1]
2022-12-02 20:48:39     INFO    MSTransformManager::open        Select data
2022-12-02 20:48:39     INFO    MSTransformManager::createOutputMSStructure     Create output MS structure
2022-12-02 20:48:39     SEVERE  mstransform::::casa     Task mstransform raised an exception of class RuntimeError with the following message: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X154/calibrated/working/uid___A002_Xf8f6a9_Xe40.ms).
2022-12-02 20:48:39     INFO    mstransform::::casa     Task mstransform complete. Start time: 2022-12-02 15:48:39.055953 End time: 2022-12-02 15:48:39.424718
2022-12-02 20:48:39     INFO    mstransform::::casa     ##### End Task: mstransform          #####
2022-12-02 20:48:39     INFO    mstransform::::casa     ##########################################
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa     Error executing pipeline task hif_mstransform.
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa     Traceback (most recent call last):
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 640, in execute
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        return self._handle_multiple_vis(dry_run, **parameters)
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 762, in _handle_multiple_vis
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        single_result = self.execute(dry_run=dry_run, **parameters)
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 40, in attach_timestamp_to_results
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 120, in handle_matplotlibrc
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kwargs)
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+      File "/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-2-pipeline-2021.3.0.17/lib/py/lib/python3.6/site-packages/Pipeline-2021.3.0.17-py3.6.egg/pipeline/infrastructure/basetask.py", line 88, in capture
2022-12-02 20:48:40     SEVERE  mstransform::pipeline.infrastructure.basetask::casa+        result = method(self, *args, **kw)
...
2022-12-02 20:48:44     SEVERE  mstransform::::casa+        return _mstransformer.mstransformer_open(self)
2022-12-02 20:48:44     SEVERE  mstransform::::casa+    RuntimeError: Desired column (CORRECTED_DATA) not found in the input MS (/orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X154/calibrated/working/uid___A002_Xf8f6a9_Xe40.ms).
2022-12-02 20:48:44     INFO    hif_mstransform::::casa Task hif_mstransform complete. Start time: 2022-12-02 15:48:38.597923 End time: 2022-12-02 15:48:44.080979
2022-12-02 20:48:44     INFO    hif_mstransform::::casa ##### End Task: hif_mstransform      #####
2022-12-02 20:48:44     INFO    hif_mstransform::::casa ##########################################

keflavich commented 1 year ago

$ ls -lhrtd *X1947* *Xe40*
-rw-r--r-- 1 adamginsburg adamginsburg 60G Jul  6 00:21 2021.1.00172.L_uid___A002_Xf8f6a9_X1947.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 60G Jul  8 00:05 2021.1.00172.L_uid___A002_Xf8f6a9_Xe40.asdm.sdm.tar

keflavich commented 1 year ago

No obvious sign of corruption, but I'm trying to re-download these anyway.

$ md5sum *X1947* *Xe40*
a76b1579d9e1022970f05e12140a8b4e  corrupt_2021.1.00172.L_uid___A002_Xf8f6a9_X1947.asdm.sdm.tar
66dd0060900191ad110bb7e540fe2e7e  corrupt_2021.1.00172.L_uid___A002_Xf8f6a9_Xe40.asdm.sdm.tar

keflavich commented 1 year ago

There are differences in the md5sums:

login4.ufhpc /orange/adamginsburg/ACES/data$ ls -lhrtd *X1947* *Xe40*
-rw-r--r-- 1 adamginsburg adamginsburg 60G Jul  6  2022 corrupt_2021.1.00172.L_uid___A002_Xf8f6a9_X1947.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 60G Jul  8  2022 corrupt_2021.1.00172.L_uid___A002_Xf8f6a9_Xe40.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 60G Dec  6 23:08 2021.1.00172.L_uid___A002_Xf8f6a9_X1947.asdm.sdm.tar
-rw-r--r-- 1 adamginsburg adamginsburg 60G Dec  7 00:15 2021.1.00172.L_uid___A002_Xf8f6a9_Xe40.asdm.sdm.tar
login4.ufhpc /orange/adamginsburg/ACES/data$ md5sum *X1947* *Xe40*
9ebe237c08234a56429d6e2fde453f3f  2021.1.00172.L_uid___A002_Xf8f6a9_X1947.asdm.sdm.tar
a76b1579d9e1022970f05e12140a8b4e  corrupt_2021.1.00172.L_uid___A002_Xf8f6a9_X1947.asdm.sdm.tar
0710f4f373fb9aee0db02adec644c691  2021.1.00172.L_uid___A002_Xf8f6a9_Xe40.asdm.sdm.tar
66dd0060900191ad110bb7e540fe2e7e  corrupt_2021.1.00172.L_uid___A002_Xf8f6a9_Xe40.asdm.sdm.tar

so it seems likely these were corrupted downloads.

keflavich commented 1 year ago

The data cubes are junk as of today, but it looks like I didn't re-run the pipeline after re-downloading the ASDMs, so I'm now extracting the ASDMs and have removed the calibrated/ directory.

keflavich commented 1 year ago

We still have an error here:

2023-02-08 22:49:49     SEVERE  AgentFlagger::restoreFlagVersion (file src/code/flagging/Flagging/AgentFlagger.cc, line 1001)   Could not restore Flag Version : ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2023-02-08 22:49:49     SEVERE  agentflagger:: (file src/tools/agentflagger/agentflagger_cmpt.cc, line 35)      Exception Reported: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)
2023-02-08 22:49:49     SEVERE  flagmanager::::casa     Task flagmanager raised an exception of class RuntimeError with the following message: ScalarColumn::putColumn(Vector&): Table conformance error (#rows mismatch)

Maybe there's even more corrupted data? I'm inclined to remove this entire directory and start over.

keflavich commented 1 year ago

$ ls -lh *_X154_*
-rw-r--r-- 1 adamginsburg adamginsburg 149G Jun 24  2022 2021.1.00172.L_uid___A001_X15a0_X154_001_of_001.tar
-rw-r--r-- 1 adamginsburg adamginsburg 748M Jun 24  2022 2021.1.00172.L_uid___A001_X15a0_X154_auxiliary.tar
$ md5sum *_X154_*
d34647a536cc8b6e0877eae38e29a165  2021.1.00172.L_uid___A001_X15a0_X154_001_of_001.tar
e6eec2beba78a354cd30dc570115b7fa  2021.1.00172.L_uid___A001_X15a0_X154_auxiliary.tar

keflavich commented 1 year ago

Further diagnostics, comparing the "old, failed" version to the re-extracted version:

$ for fn in member.uid___A001_X15a0_X154/calibration/*; do md5sum $fn failed_${fn}; done
990b3812fbb539dbe30afbad49e73169  member.uid___A001_X15a0_X154/calibration/member.uid___A001_X15a0_X154.hifa_calimage_renorm.auxproducts.tgz
990b3812fbb539dbe30afbad49e73169  failed_member.uid___A001_X15a0_X154/calibration/member.uid___A001_X15a0_X154.hifa_calimage_renorm.auxproducts.tgz
9ec7f77560ce1475aaeb9964dd1c1d0e  member.uid___A001_X15a0_X154/calibration/member.uid___A001_X15a0_X154.session_1.caltables.tgz
9ec7f77560ce1475aaeb9964dd1c1d0e  failed_member.uid___A001_X15a0_X154/calibration/member.uid___A001_X15a0_X154.session_1.caltables.tgz
1bb2f7b2dafa784724b365ebe8993dab  member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_X1947.ms.calapply.txt
9ccd450826b31f279eb6524a6223d63a  failed_member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_X1947.ms.calapply.txt
8dabd84baf166f1fd61556aef5a19827  member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_X1947.ms.flagversions.tgz
479a941195bfc9c072f643861afe02eb  failed_member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_X1947.ms.flagversions.tgz
455f00656933dffba8480dd80dea4004  member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_Xe40.ms.calapply.txt
b8141f4bc7b9a925aeb835fac4f93e90  failed_member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_Xe40.ms.calapply.txt
0af3afc7c524c2938bbba3cd3f17a7a9  member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_Xe40.ms.flagversions.tgz
401e915eb6fdfe2675ab77e6ed93d1e1  failed_member.uid___A001_X15a0_X154/calibration/uid___A002_Xf8f6a9_Xe40.ms.flagversions.tgz

The applycal commands are different. So something was changed on the server, this wasn't as simple as a failed download - it was something legitimately changing.

new:

  # Apply file for uid___A002_Xf8f6a9_X1947.ms
  applycal(vis='uid___A002_Xf8f6a9_X1947.ms', field='3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,  92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146', intent='TARGET', spw='25,27,29,31,33,35', antenna='0~43', gaintable=['uid___A002_Xf8f6a9_X1947.    ms.h_tsyscal.s6_2.tsyscal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_wvrgcalflag.s9_5.sm6_048s.wvrcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_bandpass.s13_4.spw25_27_29_31_33_35.channel.solintinf.bcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_spwphaseup.s14_4.spw25_27_29_31_33_35.solintinf.gpcal.tbl',                'uid___A002_Xf8f6a9_X1947.ms.hifa_timegaincal.s17_3.spw25_27_29_31_33_35.solintinf.gpcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_timegaincal.s17_8.spw25_27_29_31_33_35.solintinf.gacal.tbl'], gainfield=['2', '', '', 'nearest', '', ''], spwmap=[[0, 19, 2, 3, 4, 5, 17, 19, 19, 9, 21, 11, 23, 17, 19, 21, 23, 17,    17, 19, 19, 21, 21, 23, 23, 17, 17, 17, 17, 19, 19, 19, 19, 21, 21, 23, 23], [], [], [], [], []], interp=['linear,linear', 'nearest', 'linearperobs,linearflag', 'linear,linear', 'linear,linear', 'linear,linear'], calwt=[True, False, True, False, False, True])
  applycal(vis='uid___A002_Xf8f6a9_X1947.ms', field='J1924-2914', intent='BANDPASS,AMPLITUDE', spw='25,27,29,31,33,35', antenna='0~43', gaintable=['uid___A002_Xf8f6a9_X1947.ms.h_tsyscal.s6_2.tsyscal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_wvrgcalflag.s9_5.sm6_048s.wvrcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.        hifa_bandpass.s13_4.spw25_27_29_31_33_35.channel.solintinf.bcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_spwphaseup.s14_4.spw25_27_29_31_33_35.solintinf.gpcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_timegaincal.s17_4.spw25_27_29_31_33_35.solintint.gpcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_timegaincal.s17_8.  spw25_27_29_31_33_35.solintinf.gacal.tbl'], gainfield=['J1924-2914', '', '', 'nearest', 'nearest', 'nearest'], spwmap=[[0, 19, 2, 3, 4, 5, 17, 19, 19, 9, 21, 11, 23, 17, 19, 21, 23, 17, 17, 19, 19, 21, 21, 23, 23, 17, 17, 17, 17, 19, 19, 19, 19, 21, 21, 23, 23], [], [], [], [], []], interp=['linear,linear',     'nearest', 'linearperobs,linearflag', 'linear,linear', 'linear,linear', 'nearest,linear'], calwt=[True, False, True, False, False, True])
  applycal(vis='uid___A002_Xf8f6a9_X1947.ms', field='J1744-3116', intent='PHASE', spw='25,27,29,31,33,35', antenna='0~43', gaintable=['uid___A002_Xf8f6a9_X1947.ms.h_tsyscal.s6_2.tsyscal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_wvrgcalflag.s9_5.sm6_048s.wvrcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_bandpass.       s13_4.spw25_27_29_31_33_35.channel.solintinf.bcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_spwphaseup.s14_4.spw25_27_29_31_33_35.solintinf.gpcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_timegaincal.s17_3.spw25_27_29_31_33_35.solintinf.gpcal.tbl', 'uid___A002_Xf8f6a9_X1947.ms.hifa_timegaincal.s17_8.                spw25_27_29_31_33_35.solintinf.gacal.tbl'], gainfield=['2', '', '', 'nearest', 'nearest', 'nearest'], spwmap=[[0, 19, 2, 3, 4, 5, 17, 19, 19, 9, 21, 11, 23, 17, 19, 21, 23, 17, 17, 19, 19, 21, 21, 23, 23, 17, 17, 17, 17, 19, 19, 19, 19, 21, 21, 23, 23], [], [], [], [], []], interp=['linear,linear', 'nearest',   'linearperobs,linearflag', 'linear,linear', 'linear,linear', 'nearest,linear'], calwt=[True, False, True, False, False, True])

old:

# Apply file for uid___A002_Xf8f6a9_X1947.ms
applycal(vis='uid___A002_Xf8f6a9_X1947.ms', field='3,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146', intent='TARGET', spw='25,27,29,31,33,35', antenna='0~43', gaintable='uid___A002_Xf8f6a9_X1947.ms.hif_uvcontfit.s5_2.Sgr_A_star.uvcont.tbl', gainfield='', spwmap=[], interp='', calwt=False)

The flagtables also changed.

d-l-walker commented 1 year ago

@keflavich any updates on this since your last comment? Just trying to get a feel for the status of this as I go through all of my assigned issues.

keflavich commented 1 year ago

$ ls -lhrtd *.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Feb 13 22:48 uid___A001_X15a0_X154.s38_0.Sgr_A_star_sci.spw25.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Feb 14 04:50 uid___A001_X15a0_X154.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Feb 17 22:04 uid___A001_X15a0_X154.s38_0.Sgr_A_star_sci.spw35.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Feb 17 23:27 uid___A001_X15a0_X154.s38_0.Sgr_A_star_sci.spw29.cube.I.iter1.image
drwxrwsr-x+ 4 adamginsburg adamginsburg 4.0K Feb 18 10:41 uid___A001_X15a0_X154.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image

Why is spw33 missing? Is it not in the override_tclean commands now? Might have to check for a regression (e.g., we accidentally deleted this one's spw33 commands at some point?)

d-l-walker commented 1 year ago

Yeah I just checked this myself, and SPW33 is in the override_tclean_commands file:

So I'm not sure why it hasn't been cleaned here ...

keflavich commented 1 year ago

Well, I see the message: ***** BROKEN spw spw33 ***** in the logs, so that's a clue.

keflavich commented 1 year ago

OK yeah this is legit, and a mistake on my part for not catching it earlier: The spw33 imaging override commands must include an imagename, since there is no default imagename. The parameters should include:

"imagename": "uid___A001_X15a0_X154.s38_0.Sgr_A_star_sci.spw33.cube.I.iter1",

and likely niter and some other parameters.

I'm gonna check through and see if there's an auto-update I can do... hold tight.

keflavich commented 1 year ago

see https://github.com/ACES-CMZ/reduction_ACES/pull/318; fixes going there

d-l-walker commented 1 year ago

This all looks good now, and SPW33 has been successfully imaged (CS peak intensity map shown below). Marking this one as done.

Screenshot 2023-03-08 at 10 39 27

djeff1887 commented 8 months ago

Cont QA (spw25_27,spw33_35): there appears to be a small amount of line contamination in both windows, notably in spw25_27.

djeff1887 commented 6 months ago

(upper 25_27, lower 33_35) This seems to be an improvement again, although there are still traces of line contamination in 33_35. There's also some odd rippling in the residual of 25_27 that wasn't present in the pipeline version.

keflavich commented 6 months ago

That rippling is strange! Could you put this in the agenda for discussion?

ACES-CMZ / reduction_ACES

Execution Block ID uid://A001/X15a0/X154 Sgr_A_st_ae_03_TM1 #132

Product Links:

Reprocessed Product Links: