search

ACES-CMZ / reduction_ACES

Reduction scripts and tools for ACES
https://worldwidetelescope.org/webclient/?wtml=https://data.rc.ufl.edu/pub/adamginsburg/ACES/mosaics/mosaics.wtml
15 stars 12 forks source link

Execution Block ID uid://A001/X15a0/Xfa Sgr_A_st_p_03_TM1 #149

Open keflavich opened 2 years ago

keflavich commented 2 years ago

Sgr_A_st_p_03_TM1 uid://A001/X15a0/Xfa

Product Links:

Reprocessed Product Links:

mpound commented 2 years ago

Overall the data look good.

Issues:

  1. Size mitigation: Setting nbin to 2 SPW 35,33,31,29,27,25.
  2. MakeImages (cube) - tclean divergence SPW 27,29,31. Mottled spot in lower left corner, most easily visible on SPW 27: uid___A001_X15a0_Xfa s38_0 Sgr_A_star_sci spw27 cube I iter1 image sky

This appears to come from channels ~350-378 where there is a lot of structure near the corner of the map, e.g. channel 364

spw27

mpound commented 2 years ago

I can't find a single cyclefactor that will fix the problem in all channels. Sometimes increasing cyclefactor improves it, sometimes it makes it worse. Here are some example channels from a 10-channel clean in SPW 27.

Left to right on all the images below: Cyclefactor =4, 3, 1.5 (original value) 1. Increasing cycle factor improves the clean but not enough.

cyclefactormakesitbetterbutnotenough

2. Increasing cycle factor makes it worse cyclefactormakesitworse

3. Increasing cycle factor has no effect cyclefactorhasnoeffect

keflavich commented 2 years ago

Try setting usemask='pb' and ... I forget the keyword now, but the pb mask limit - try setting that to 0.5 or something high, see if that works in conjunction with the high cyclefactor? (since this is coming in from the edge)

mpound commented 2 years ago

Although the latest FITS cubes indicate tclean ran after the merge above, the tclean does not appear to have used the updated parameters. The divergence in SPW27 is still present in channels 359-361 that had come out fine in my test. The FITS header says tclean for SPW27 ran on 2022-09-23 which was a week after merge #250. Yet here's channel 360: chan360bad

SPW31: tclean ran on 2022-09-15 There are also still divergent channels (channels 428-430)

SPW29: tclean ran on 2022-10-03 No Divergent Channels

@keflavich Is it possible that SPWs 27 and 31 ran with the old tclean parameters, but SPW29 had the new ones?

keflavich commented 2 years ago

yes, modification dates on those are old, so I'd believe it

-rw-r--r-- 1 adamginsburg adamginsburg  15G Sep 19 20:24 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image.pbcor.fits
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 19 20:24 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.pb
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 19 20:24 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image.pbcor
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 19 20:25 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.weight
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 19 20:25 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.model
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 19 20:25 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.residual
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 19 20:25 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.sumwt
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 19 20:26 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.psf
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 19 20:26 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 19 20:26 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.mask
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 23 11:42 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.sumwt
-rw-r--r-- 1 adamginsburg adamginsburg  15G Sep 23 11:42 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image.pbcor.fits
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 23 11:42 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.weight
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 23 11:43 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.mask
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 23 11:43 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.model
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 23 11:44 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image.pbcor
drwxr-sr-x 3 adamginsburg adamginsburg 4.0K Sep 23 11:44 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.psf
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 23 11:44 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 23 11:45 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.pb
drwxr-sr-x 4 adamginsburg adamginsburg 4.0K Sep 23 11:45 uid___A001_X15a0_Xfa.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.residual
keflavich commented 2 years ago

these are the parameters online: https://github.com/ACES-CMZ/reduction_ACES/blob/689c8f4830a9b3d5a68fa9294fea35f5301bd543/aces/pipeline_scripts/override_tclean_commands.json#L2473-L2478

https://github.com/ACES-CMZ/reduction_ACES/blob/689c8f4830a9b3d5a68fa9294fea35f5301bd543/aces/pipeline_scripts/override_tclean_commands.json#L2485-L2490

Are they right? Looks like they've been modified from default at least, by human hand, in https://github.com/ACES-CMZ/reduction_ACES/commit/0fccadc71e3c61d02c076395633c6dfa0506310f

mpound commented 2 years ago

Yep, those are the parameters I had tested locally which fixed the divergence issues when running on my machine. CASA 6.5.1-23 CASAtools:v1.0. Version on hipergator is CASA 6.4.3-2 CASAtools:v1.0.0

mpound commented 2 years ago

Aha -- the FITS header gives the answer. For SPW 29 HISTORY pbmask = 0.5 And for the others: HISTORY pbmask = 0.0

So 27 and 31 still need re-run with the new params

keflavich commented 2 years ago

they're running

50788606                                  p_TM1_spw27_cube  astronomy-dept  astronomy-dept-b    RUNNING     4553     128G          1        16:47:10        16:47:10      4-00:00:00           c0711a-s17
50788607                                  p_TM1_spw31_cube  astronomy-dept  astronomy-dept-b    RUNNING     4553     128G          1        16:47:10        16:47:10      4-00:00:00           c0711a-s20
mpound commented 2 years ago

New cubes look good - no divergent channels. Marking this as done.

d-l-walker commented 11 months ago

This is marked as done, but I think it still needs to be fixed for size mitigation. I see updated parameters in override_teclean_commands.json, but these relate to masking and cyclefactor, presumably related to the divergence issues discussed above.

@mpound can you please check this? If this does still need doing, could you then create a PR to update the channel parameters?

(Tagging #179 for tracking)

mpound commented 11 months ago

@d-l-walker Yes, these appear to still be size mitigated. Can you tell me why sometimes the fix for size mitigation sometimes is nchan=1912 (PR #392) and sometimes nchan=1914 (e.g. PR #302)? I want to make sure I update override_teclean_commands.json correctly.

keflavich commented 11 months ago

That likely has to do with how many EBs were executed and when they were executed. If there were a few weeks between observations, there might have been a channel-width shift between observations, and the final grid has to be expanded to accommodate them. I think the center frequency is always picked in the LSR frame, but there may be drift within the bounds of a channel, which would result in up to 2 pixel buffer being needed.

(but I hope Dan can confirm that)

d-l-walker commented 11 months ago

@mpound the way that I have been doing it is to look at the default_tclean_commands.json file to find the parameters used by the pipeline for the specific SB/SPW combination. Then just double the channel numbers / halve the channel width.

Edit: looks like @keflavich has responded with a plausible explanation for why this actually occurs, thanks ;)

mpound commented 11 months ago

Before uploading new clean parameters, I tested them on spw27 locally, all 1912 channels. It looks good, so I will do a PR with new tclean parameters.

keflavich commented 11 months ago

There are so many SiO masers in this field:

image
mpound commented 11 months ago

Wow, that's amazing. There are a couple compact 3mm sources in this field in the CARMA continuum survey as well as non-thermal filament SgrA-E ("The Thread")

xinglunju commented 9 months ago

QA - Line contamination in continuum images from high/low frequencies

The new spw33_35 image is misssing. The new spw25_27 image appears to be contaminated, with the large scale ripples.

Will update once the new spw33_35 image is available.

Attached images: Top-left: spw25_27 Top-right: oldhigh_spw33_35 Bottom-left: spw25_27_29_31_33_35

image
xinglunju commented 8 months ago

Updated continuum QA (20240403)

The new spw33_35 image is missing @keflavich

The new spw25_27 image looks better.

Attached images: Top-left: spw25_27 Top-right: old spw25_27 (v1.1) Bottom-left: old spw33_35 (v1) Bottom-right: old spw25_27_29_31_33_35 (v1)

image

keflavich commented 8 months ago

Reimaging to undo size mitigation is in the queue

d-l-walker commented 7 months ago

Products are now un-mitigated and ready for statcont @keflavich. Note that SPWs 25 & 33 are fine, the existing statcont cubes are unmitigated.

xinglunju commented 6 months ago

I downloaded the new spw33_35 continuum image on May 1. But the image is empty?

image

keflavich commented 6 months ago

Removed bad spw35: mv *.spw33_35.* failed_20240520/ mv *.spw25_27_29* failed_20240520/

I will restart with identical parameters and hope the redo fixes the problem. If it does not, we may need further investigation - possibly this is caused by an incorrect data column selection as we recently saw in aj & v (https://github.com/ACES-CMZ/reduction_ACES/issues/235#issuecomment-2120637880)

keflavich commented 6 months ago

The approach did not solve the issue. There is still no signal in the continuum images.