Execution Block ID uid://A001/X15b4/X41 Sgr_A_st_m_03_TM1_updated

keflavich commented 2 years ago

Sgr_A_st_m_03_TM1_updated uid://A001/X15b4/X41

[x] Observations completed?
[x] Delivered?
[x] Downloaded? (specify where)
- [x] hipergator
- [x] hipergator pipeline run
- Pipeline Run 20231210T230154
- Pipeline Run 20231210T220431
- Pipeline Run 20231211T050300
- Pipeline Run 20220623T040333 [trash]
- Pipeline Run 20220623T053416 [trash]
- Pipeline Run 20220624T040519 [trash]
[ ] Weblog unpacked
[ ] Weblog Quality Assessment?
Extra Weblog Sgr_A_st_m_03_TM1_updated_0 -> pipeline-20220624T040519, Extra Weblog Sgr_A_st_m_03_TM1_updated_1 -> pipeline-20220602T193603
[ ] Imaging: Continuum
[ ] Imaging: Lines
- [ ] SPW25 H13CN
- [ ] SPW27 H13CO+/SiO
- [ ] SPW29 HCO+
- [ ] SPW31 HNCO
- [ ] SPW33 Cont 1
- [ ] SPW35 Cont 2

Product Links:

Reprocessed Product Links:

nbudaiev commented 2 years ago

SPW33

Size mitigated

List of other ~minor issues.

Second opinion on the importance of noted issues would be appreciated. (all issues are minor)

Contamination in Tsys data (7. hifa_tsysflag). Similar to issue #142 (the bump is outside of the spws)

tsys-uid___A002_Xf84513_X1134 ms-summary spw19

Issues with J1744 calibrator have not been flagged out properly everywhere.

uid___A002_Xf84513_X2e72 ms-J1744-3116-spw29-PHASE-amp_vs_freq-XX_YY

SPW25: compact emission visible in residual and mom8 map.

uid___A001_X15b4_X41 s38_0 Sgr_A_star_sci spw25 cube I iter1 residual sky

SPW27: some structure in residual and mom8 map.

uid___A001_X15b4_X41 s38_0 Sgr_A_star_sci spw27 cube I iter1 residual sky

SPW29: structure in residual and mom8 map.

uid___A001_X15b4_X41 s38_0 Sgr_A_star_sci spw29 cube I iter1 residual sky

SPW31: very minor structure in residual

SPW35: minor compact emission in residual and mom8

uid___A001_X15b4_X41 s38_0 Sgr_A_star_sci spw35 cube I iter1 residual sky

nbudaiev commented 2 years ago

SPW 33: Inspection by hand revealed divergence around channel 611. Needs recleaning. Screen Shot 2022-09-28 at 9 33 42 AM

nbudaiev commented 1 year ago

A test reclean of 20 channels around the diverging channel with "cyclefactor" changed from 2.5 to 3.0 was successful, no divergence. A lot of re-cleaning attempts end with "Error in making PSF : One or more of the cube section failed in de/gridding. Return values for the sections: [1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]". The attempted fix was to run the cleaning on the pipeline version of casa (casa-6.4.3-2-pipeline-2021.3.0.17) instead of the standard version (casa-6.4.4-31-py3.8), which seemed to work. However, Adam Ginsburg suggested that in his experience you just have to keep restarting the clean until this error doesn't show up so it might have been a coincidence.

A re-clean that ran till the end took about 12 days on UF's Hipergator. "cyclefactor" was changed from 2.5 to 3.0. The resulting clean still diverged in channels 609 and 613 (different than before). Not sure why the results are different from the small 20-channel version. I will try once again with a higher cyclefactor.

nbudaiev commented 1 year ago

Threshold = 0.02Jy, cyclefactor = 5.0 were the parameters to avoid [most] of the divergence. You can see one of the channels that has a hint of divergence on the screenshot (left: new parameters, right: original divergence). The threshold value had to be increased to avoid divergence and from my experience I don't think there is a way to reduce it further without running into divergence. Took about 3 weeks to run. I will be updating the cleaning parameters soon.

Screenshot 2023-02-10 at 2 07 59 PM

keflavich commented 1 year ago

Wow 3 weeks is a long time, good work persisting through that! I'd guess that the remaining artifacts may be poor calibration, not just cleaning. But this being Sgr B2, I could be wrong.

nbudaiev commented 1 year ago

Unfortunately, there is a contsub issue affecting the location of SgrB2 M:

Screenshot_2023-02-10_at_1 58 26_PM

keflavich commented 1 year ago

TODO:

[ ] Redo UVcontsub - possibly need to redo selection
[ ] Contsub with zero-order
[ ] ...then reimage again w/same parameters.

ashleythomasbarnes commented 1 year ago

As done here: https://github.com/ACES-CMZ/reduction_ACES/issues/28

nbudaiev commented 1 year ago

Discussion from WP1 meeting: this field is supposed to have 0th order UVcontsub, which would subtract a constant value. Instead, it appears that UVcontsub produced a slope, which is indicative of a 1st order. TODO: look at the logs to see what UVcontsub was used

nbudaiev commented 1 year ago

casa_log_mpi_pipeline_imaging_member.uid___A001_X15b4_X41_40424731_2022-06-24_00_04_56.log shows that 'hif_uvcontfit' ran with fitorder=0, which contradicts the pervious hypothesis.

Screenshot 2023-05-17 at 1 09 59 AM

keflavich commented 12 months ago

The continuum selection is awful:

97.7238649485~97.7414430735GHz, 99.0817751048~99.0925172923GHz, 99.3727907298~99.3752321360GHz

there are 3 tiny windows, and the 3rd one lands directly on a bright line

It's still a bit unclear how this happened with fitorder=0

keflavich commented 12 months ago

I still agree with Nazar here that this doesn't make sense - fitorder=0 is what's in the pipeline.

There are these commands in this log file (member.uid A001_X15b4_X41.hifa_calimage_renorm.casa_commands.log):

# hif_uvcontfit(pipelinemode="automatic")
#
# No comment registered for hif_uvcontfit
#
uvcontfit(vis='uid___A002_Xf84513_X1134_target.ms',
          caltable='uid___A002_Xf84513_X1134_target.ms.hif_uvcontfit.s32_1.Sgr_A_star.uvcont.tbl',
          field='3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69
9,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114',
          intent='OBSERVE_TARGET#ON_SOURCE',
          spw='25:85.9848356039~85.9882535726GHz;86.0055875570~86.0070524008GHz;86.0314664633~86.0336637289GHz;86.1149625570~86.1183805258GHz;86.1300992758~86.1337613851GHz;86.2209195883~86.2238492758GHz;86.397
6.6965665609~86.7226896078GHz;86.8789396078~86.8821134359GHz;86.9204435140~86.9333829672GHz;86.9761075765~86.9853849203GHz;87.0266446859~87.0312833578GHz;87.0608243734~87.0693692953GHz;87.1111173422~87.11477945
8578GHz,29:89.1784367563~89.1810307504GHz;89.2039494516~89.2051091196GHz;89.2102971078~89.2120671274GHz,31:87.9406437875~87.9449162485GHz;87.9547734262~87.9552006723GHz,33:97.7238649485~97.7414430735GHz;99.0817
27907298~99.3752321360GHz,35:99.7692672923~99.7868454173GHz;100.7155563548~100.7194626048GHz;100.7604782298~100.7697555735GHz;100.9679977610~100.9748336985GHz;100.9826461985~100.9870407298GHz;101.0007126048~101
~101.0578415110GHz;101.2043258860~101.2209274485GHz;101.2375290110~101.2468063548GHz',
          solint='int', fitorder=1, append=False)
uvcontfit(vis='uid___A002_Xf84513_X2e72_target.ms',
          caltable='uid___A002_Xf84513_X2e72_target.ms.hif_uvcontfit.s32_2.Sgr_A_star.uvcont.tbl',
          field='3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69
9,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114',
          intent='OBSERVE_TARGET#ON_SOURCE',
          spw='25:85.9847359340~85.9881539027GHz;86.0054878871~86.0069527308GHz;86.0313667933~86.0335640590GHz;86.1148628871~86.1182808558GHz;86.1299996058~86.1336617152GHz;86.2208199183~86.2237496058GHz;86.397
6.6964668910~86.7225899379GHz;86.8788399379~86.8820137660GHz;86.9203438441~86.9332832972GHz;86.9760079066~86.9852852504GHz;87.0265450160~87.0311836879GHz;87.0607247035~87.0692696254GHz;87.1110176722~87.11467978
1879GHz,29:89.1783347831~89.1809287773GHz;89.2038474784~89.2050071464GHz;89.2101951347~89.2119651542GHz,31:87.9405418144~87.9448142753GHz;87.9546714531~87.9550986991GHz,33:97.7237503110~97.7413284360GHz;99.0816
26760923~99.3751174985GHz,35:99.7691526548~99.7867307798GHz;100.7154417173~100.7193479673GHz;100.7603635923~100.7696409360GHz;100.9678831235~100.9747190610GHz;100.9825315610~100.9869260923GHz;101.0005979673~101
~101.0577268735GHz;101.2042112485~101.2208128110GHz;101.2374143735~101.2466917173GHz',
          solint='int', fitorder=1, append=False)

that indicate fitorder=1, but they should be fitting across all windows, not within one window, and therefore shouldn't be so badly affected. It's also not clear to me that this log is telling what actually ran; it looks more like a script than a log.

ashleythomasbarnes commented 12 months ago

Indeed, looking at the weblogs, it isn't obvious that the images we're seeing above are from fitorder=1 or fitorder=0. The lastest version Extra Weblog Sgr_A_st_m_03_TM1_updated_1 -> pipeline-20220602T193603 has fitorder=1 - could we check if this was rerun correctly?

Noting again here that https://github.com/ACES-CMZ/reduction_ACES/issues/28 showed (I think) setting to fitorder=0 solved the issue

If we confirm this was run correctly and still has the problem, I will look into the continuum ranges

keflavich commented 12 months ago

Huh, you're right, I must have been searching the wrong files. Specifically, this page shows fitorder=1

keflavich commented 11 months ago

Discussion today: Are we changing anything before re-running, or do we just re-run as is?

Are the data on disk uv-subtracted already? If so, we need to modify the MSes.

Some question remains about whether we need to update the continuum selection in addition to re-fitting the UV subtraction. Dan suggested that the data column might be uv-unsubtracted and the corrected is uv-subtracted.

d-l-walker commented 11 months ago

From the pipeline documentation:

the original continuum + line emission is contained in the DATA column of the input MS called *_targets.ms,
while the continuum subtracted data are written to the DATA column of the new *_targets_line.ms.

For the older versions of the pipeline (up to 6.2.1):

the *_target.ms created by the pipeline, have calibrated line+continuum data in the
DATA column, and continuum-subracted line data in the CORRECTED column.

keflavich commented 11 months ago

Thanks @d-l-walker . Do you know a quick way to confirm that's the case? I don't like to just trust documentation, so is there a good way short of loading and plotting all the data to determine whether the CORRECTED column is populated with the uvsub'd data?

In any case, we need to re-run contfit & uvsub in this SB.

keflavich commented 11 months ago

I'm going to move the calibrated/ directory to calibrated_20231206 to trigger a re-run of the pipeline that should use 0-order.

keflavich commented 11 months ago

Re-running the pipeline has apparently run the newer version and I ended up with a targets.ms instead of a target.ms file. No lines files have shown up yet though, so I guess it's still going. I'm not 100% sure what outcome to expect, but right now the job runner doesn't like the partially-completed pipeline run.

keflavich commented 11 months ago

The pipeline died: it got stuck on a fight-with-self-over-lock issue. It got so close to completing, but the second target.ms file never finished because it's in a perma-lock state, even though there are no live CASA jobs running:

2023-12-10 18:59:26     INFO    applycal::::    Process 3970752: waiting for write-lock on file /orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15b4_X41/calibrated/working/uid___A002_Xf84513_X1134_targets.ms/table.lock

keflavich commented 11 months ago

unfortunately, this means the second target.ms wasn't split out, so I can't even recover the mstransform command... the simplest thing is to just delete this and re-run the pipeline from scratch, which will take another few computer-days but no more of my time (I hope...)

keflavich commented 9 months ago

The continuum in this field is not in a good state. While there's no surprise that Sgr B2 N and M are going to be ugly until we give them a lot of dedicated attention, there are two bigger problems: Left: the all-spw continuum clearly includes line emission. Those ripply structures look like line features to me, anyway./ Right: The edge of the field diverged in the spw33+35 image

xinglunju commented 8 months ago

QA - Line contamination in continuum images from high/low frequencies

This field contains Sgr B2 N/M hot cores. Overall the new images look good. Possible divergence in the northern edge of the high freq image (spw33_35) as already mentioned.

Attached images: Top-left: spw33_35 Top-right: oldhigh_spw33_35 Bottom-left: spw25_27 Bottom-right: spw25_27_29_31_33_35

xinglunju commented 7 months ago

Updated continuum QA (20240403)

The images look good. The low-freq continuum (spw25_27, bottom left) seems to be diverging.

Attached images: Top-left: spw33_35 Top-right: old spw33_35 (v1.1) Bottom-left: spw25_27 Bottom-right: spw25_27_29_31_33_35

ACES-CMZ / reduction_ACES