search

ACES-CMZ / reduction_ACES

Reduction scripts and tools for ACES
https://worldwidetelescope.org/webclient/?wtml=https://data.rc.ufl.edu/pub/adamginsburg/ACES/mosaics/mosaics.wtml
15 stars 12 forks source link

Array combination for HNCO (12m + 7m + TP) #192

Open d-l-walker opened 2 years ago

d-l-walker commented 2 years ago

This is general issue for tracking and discussing the 12m + 7m + TP combination for the spectral line data. HNCO data is being prioritised initially. Below is a checklist of regions that have been completed so far, along with links to download the combined products.

Note: the hyperlinks don't appear to be working, so for now you can just copy and past the link address, which should work. Products will soon be migrated to our Globus repo.

See the comments for this issue below for more info & images.

HNCO

d-l-walker commented 2 years ago

Peak HNCO intensity maps for Sgr_A_st_ao

Left: 12m + 7m + TP, Right: 12m + 7m (See #122 for more discussion on this region) Brick_12m_7m_TP_HNCO_clean_results

Peak HNCO intensity maps for Sgr_A_st_w. Lots of emission!

Left: 12m + 7m + TP, Right: 12m + 7m Sgr_A_st_w_12m_7m_TP_HNCO_clean_results

d-l-walker commented 2 years ago

For reference, here are the scripts that I'm currently using to do this: https://github.com/d-l-walker/ACES_personal/tree/main/joint_deconvolution

This is a work-in-progress and can almost certainly be improved, streamlined, packaged up properly, etc. This is just a first attempt. I'm already aware of a few fixes/exceptions etc. that I need to implement.

Right now I'm just running a master shell script to execute the relevant scripts in-turn. I just pass a region argument to the python scripts (e.g. _Sgr_A_stao), and the rest is automated.

The process is a bit slow at the moment, as I'm having to download the relevant datasets locally and set up the directory structures. I hope that soon we should have the large ACES NAS drives set up at the UK ARC Node, where we can mirror the data hosted on Globus. Once this is in place I should be able to set things up for that fixed directory structure, and better automate this to run on all regions for which we have 12m + 7m + TP data delivered.

keflavich commented 2 years ago

@d-l-walker could you put your joint_deconvolution directory/project into the main repository? It can sit around as a "draft" pull request for as long as you like, but this is a perfect case to be able to have code review.

d-l-walker commented 2 years ago

@keflavich Done: see #195

I'm happy for it to be merged if that would make the most sense. I'll just update with any changes as I continue to make improvements.

d-l-walker commented 2 years ago

The first fully-automated end-to-end run of the scripts just finished. This was for region Sgr_A_st_ai. Everything went smoothly and took just over 2 days to finish, but the 12m+7m cleaning diverged across a few channels due to bright edge emission.

The HNCO peak intensity of the 12m+7m+TP still looks decent (left), but you can see the divergence in the cube towards the centre of the lower edge (right). I was already running with cyclefactor=1.5 as my default. I've bumped this up to 2.5 to see whether that fixes things.

Screenshot 2022-08-04 at 15 14 19

ashleythomasbarnes commented 2 years ago

@d-l-walker - could you also produce a HNCO cube where you just feather the 12m with 7m and then feather 12m+7m with the tp? And then compare with your joint deconvolution?

d-l-walker commented 1 year ago

I've added 2 more completed regions to this issue. Peak intensity maps shown below. Link to download the FITS cubes can be found at the top of this issue.

Peak HNCO intensity maps for Sgr_A_st_a (#242)

Left: 12m + 7m + TP, Right: 12m + 7m Sgr_A_st_a_12m_7m_TP_HNCO_clean_results

Peak HNCO intensity maps for Sgr_A_st_aa (#251)

Left: 12m + 7m + TP, Right: 12m + 7m Sgr_A_st_aa_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Some more finished regions. The links to download the FITS cubes have been added to the main post for this issue.

Peak HNCO intensity maps for Sgr_A_st_ac (#232)

Left: 12m + 7m + TP, Right: 12m + 7m Sgr_A_st_ac_12m_7m_TP_HNCO_clean_results

Peak HNCO intensity maps for Sgr_A_st_d (#247)

Left: 12m + 7m + TP, Right: 12m + 7m Sgr_A_st_d_12m_7m_TP_HNCO_clean_results

Peak HNCO intensity maps for Sgr_A_st_ai (#130, this one was posted above noting divergence, which is now fixed)

Left: 12m + 7m + TP, Right: 12m + 7m Sgr_A_st_ai_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Another region finished: Sgr_A_st_ak (#120)

Sgr_A_st_ak_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Another region finished: Sgr_A_st_ap (#43)

Sgr_A_st_ap_12m_7m_HNCO_clean_results

d-l-walker commented 1 year ago

Another region finished: Sgr_A_st_c (#256). Download links have been added to the parent comment for this issue.

This field also includes the Brick. Probably worthwhile cleaning together with field ao (#41) at some point.

Sgr_A_st_c_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Region Sgr_A_st_ag (https://github.com/ACES-CMZ/reduction_ACES/issues/136) finished. Download links have been added to the parent comment for this issue.

Sgr_A_st_h_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Region Sgr_A_st_i (https://github.com/ACES-CMZ/reduction_ACES/issues/138) finished. The peak emission looks really nice in this region (which is just below Sgr B2 south). The TP data is also making a clear difference. Will add download links above shortly.

Sgr_A_st_i_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Region Sgr_A_st_j (https://github.com/ACES-CMZ/reduction_ACES/issues/117) finished. This is also just below Sgr B2 (below North/Main). Peak intensity maps look great, and again the TP data is adding a lot.

For now, I will only upload the 12m + 7m data, as the TP data we have in-hand has the wrong velocity shift. The updated SB has not yet been delivered. Once it has, I'll re-do the feathering and upload the fully combined cube.

Sgr_A_st_j_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

12m + 7m HNCO combination for region k (#152) diverged, despite using cyclefactor=3.0. It also looks like the imsize, which is pulled directly from the PL 12m parameters, is not suitable here. The image is wrapping in the vertical direction. Will try re-running with a higher cyclefactor and a larger image size.

Screenshot 2023-04-04 at 13 50 45

d-l-walker commented 1 year ago

Regions l (#137) and m (#118) have finished. The TP data don't appear to be adding much to region l, but the contribution for region m is more significant. (The latter is Sgr B2 N, hence the bright compact sources.) Download links added to to parent comment as always.

Sgr_A_st_l_12m_7m_TP_HNCO_clean_results Sgr_A_st_m_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Sgr_A_st_n (#231) finished. Emission looks fairly faint and fluffy here. This region is right at the lower left of the mosaic, so this is probably not too surprising. Adding download links now.

Sgr_A_st_n_12m_7m_TP_HNCO_clean_results

d-l-walker commented 1 year ago

Quick, dirty test following yesterday's discussion re: data combination.

Both look great, and the feathering-only approach is significantly faster, but there's a clear difference between the two, with the joint imaging + feathered image containing more (peak) intensity.

I just ran a quick, standalone test for this. Next step is to do this exactly as @pyhsiehATalma is doing for the CS data, following her sample code in #341 and #340.

Sgr_A_st_i_feather_testing

d-l-walker commented 1 year ago

A few more regions have finished.

Region p (#149) looks really nice. I think this may be ~ the 20 km/s cloud? TP data is making a big difference.

Sgr_A_st_p_12m_7m_TP_HNCO_clean_results

For the next few images, the TP combination has not finished yet, so apologies for the crappy screenshots ...

Region u (#241) looks interesting, though relatively diffuse. This region is at the bottom edge of our mosaic, just to the lower right of Sgr B2.

Screenshot 2023-06-07 at 13 33 12

Region v (#235) doesn't seem to show a whole lot of structure. This region is basically ~ 0.15 deg directly above Sgr A*, so maybe this isn't surprising given the latitude.

Screenshot 2023-06-07 at 13 35 54

Region q (#109) looks neat, but there's something funky going on at the edges. My guess is that it'll be something due to the additional channels in the 7m data causing there to be a handful of edge channels with no 12m data. I'll look into this and fix as appropriate. For context, I think this region corresponds to one of the "far side" clouds at negative longitudes (maybe the "wiggles"?).

Screenshot 2023-06-07 at 13 37 08

d-l-walker commented 1 year ago

Regions ah (#238) and g (#134) look super cool. They're adjacent field at the lower left of the mosaic.

Note that these are 12m + 7m + TP, all feathered together (see #350), not jointly imaged like the previous images in this issue.

Next step is to reproject these, plus all other combined HNCO peak intensity maps, into a single mosaic.

Screenshot 2023-06-19 at 15 56 53
d-l-walker commented 1 year ago

First attempt a combined peak HNCO intensity mosaic, and it's looking fantastic! (Open the image in a new tab to appreciate it fully). The whole Sgr B2++ region looks crazy.

HNCO_12m7mTP max_mosaic

There are a bunch of regions missing for various reasons:

Again, note that these are feathered-only maps. I will repeat this for the jointly-images data soon.

snlongmore commented 1 year ago

Wow!! Great work, that is truly spectacular!!!

d-l-walker commented 1 year ago

Updated with a couple more fields.

12m7mTP_HNCO_peak_mosaic

d-l-walker commented 1 year ago

Updated 12m+7m+TP peak HNCO mosaic. Only 3 regions missing now (r, y, af), we're getting there!

While we are waiting for these outstanding SBs to complete the mosaic, I'll work on incorporating region weighting as @pyhsiehATalma has done for the CS mosaic in order to suppress the individual mosaic edges.

12m7mTP_HNCO_peak_mosaic

ashleythomasbarnes commented 1 year ago

Quick comparison between the HNCO 12m+7m+tp (left, blue spec), tp (centre, orange spec), and mopra (right, red spec). Looks like we are recovering the total power flux pretty well, but there are some differences when compared to the mopra data. When looking through the mopra cube there are clearly baseline issues present. The baseline of the total power cube actually looks pretty nice, but I think there may be some overfitting from the pipeline going on here. Also worth pointing out that the mopra data doesn't exactly cover the lines we observed - e.g. the CS(2-1) isn't included... Anyway I still plan to try feather with the mopra data due to the higher angular resolution, as this could result in a better combination (let's see).

Screenshot 2023-08-10 at 09 54 16

keflavich commented 1 year ago

@pyhsiehATalma has CS 2-1 data from Nobeyama, though - we can compare to those: https://github.com/CentralMolecularZone/DataSets -> https://zenodo.org/record/7708322#.ZAiKsnZBy5c

ashleythomasbarnes commented 8 months ago

(copying some old stuff from Slack below)

ashleythomasbarnes commented 8 months ago

Ashley Barnes 2 months ago I downloaded the brick joint clean and it appears to have stronger negative bowls than the feathered image (which actually doesn’t look that bad.. Anyone else finding something similar?

Screenshot 2023-09-28 at 09 40 11

Daniel Walker 2 months ago I seem to remember the joint imaging of this field was very prone to diverging right in that lower left region, so it’s possible that something’s still dodgy with the jointly imaged cube here (edited)

Ashley Barnes 2 months ago Oh right, yep, probably not the easiest with the emission so close to the edge of the map

Ashley Barnes 2 months ago I’ll download another field to check

Daniel Walker 2 months ago I’ll try re-doing the joint imaging again with some tweaked parameters to hopefully address this. Thanks for catching it!

Ashley Barnes 2 months ago No rush on this one, maybe would be good to focus on the tests before rerunning :+1: 1

Ashley Barnes 2 months ago I’m downloading feild i now - looks like a good one with lots of extended emission

Ashley Barnes 2 months ago So it seems like where there is real emission, the joint imaged map does recover more of the larger scales and generally looks a bit nicer. However, it also appears to contain stronger and negative bowls around this emission compared to the feathered image… Screenshot 2023-09-28 at 12 06 11

Screenshot 2023-09-28 at 12 06 23

ashleythomasbarnes commented 8 months ago

Ashley Barnes 2 months ago When integrating over a large area comparable to the single dish resolution, I find that be fluxes are more or less the same, which makes me think this isn’t a problem with the TP feathering, but rather the cleaning of the 12m and 7m

Screenshot 2023-09-28 at 12 08 23

Ashley Barnes 2 months ago The negative bowls are not as prominent in the 12m only image…

Daniel Walker 2 months ago Yeah we’d noticed this in the past (https://docs.google.com/document/d/1kB1LJBfREhLHWzaw3kKR5E0tpE_T-pC8piQfAI-Injk/edit#bookmark=id.iad7sv37lpi) It seems that the negatives from the 7m are adding significant negatives into the jointly imaged map, but not when they’re all feathered.

Daniel Walker 2 months ago To be clear about how the joint imaging was done, all I’m doing is taking the exact tclean parameters that were used for the 12m, and just adding the 7m ms files in the vis parameter. The only other parameter I’ve tweaked is the cyclefactor when necessary.

Ashley Barnes 2 months ago Ooo I see… hmm okay

Ashley Barnes 2 months ago Also I’ve been playing with 1) including the MOPRA data to your cleaned 7m+12m images and it isn’t great, there are still a number of strong negative bowls in the image (I haven’t tested if its better than the TP); 2) increasing the weighting factor of the single dish in feather (attached show MOPRA+7m+12m; factors of 1, 2, 3, 4, and 5). You can see that upping the factor does solve the bowls, but some other artefacts are present (and the flux gets too high).

Screenshot 2023-09-28 at 18 18 12

Ashley Barnes 2 months ago How do the residuals in the 7m+12m cleaned images look @Daniel Walker ?

Daniel Walker 2 months ago Sorry @Ashley Barnes , I’ve been caught up with other things since last week and haven’t had a lot of time to spend on this. I’ve just had a quick look at some regions, and in general the residuals look decent. Not perfect, the automasking isn’t quite getting all of the fainter extended emission, but it’s doing a reasonable job considering the complexity. Screenshots show a single channel comparison, and the mean spectrum comparison. We’ll have to have a more detailed look at this at some point obviously, this is just at a glance.

Screenshot 2023-10-04 at 11 23 31 Screenshot 2023-10-04 at 11 53 53