Post HEAD 2023 updates

[zoghbi-a]

This PR has a few updates:

• Updating the NICER notebook to use the updated analysis tools (nicerl3-spect and nicerl3-lc).
• A new demo on how to use jdaviz for image visualization and region files in a notebook along with heasoftpy.
• A simple notebook on how to use pyvo to find data and then download it.

[abasuzych]

RE: jdaviz_demo notebook:

Excellent! This is really very cool! I'm adding my comments here:

• (minor)... at imviz.show() step might add a note that this take several seconds to load... I was getting worried and wasn't sure if I need to do anything and started clicking around on the window!
• I suggest more instructions for the selecting regions step. Creating subset 1 was easy enough but it wasn't obvious how to create a new region subset2 -- I kept overwriting subset1 somehow!
  • e.g. before the "We now proceed..." markdown add something like "In the imviz window above, select regions by clicking first for the source region and resizing (this will be designated Subset1) ... To create a new region for the background ... " and possibly add a screengrab (or mark in screengrab above, which is missing this part:) that includes the bar above with "Import data" etc to show that the red dot allows changing, deleting, adding subsets, etc.
• some suggestions for extensions: how can one display the previous region files on this viewer (i.e. next time load the viewer with the regions already marked? Can we use other region shapes e.g. annular/rectangular regions? If these suggestions are too specific, maybe provide links to the imviz page: https://jdaviz.readthedocs.io/en/latest/imviz/index.html. Can we save the region files in WCS coords? (I can see wanting to overlay the saved regions on other datasets and would want to know that this could be saved/converted reliably to WCS, etc.)

[abasuzych]

Are there better links for interacting with the viewer itself? For example, viewing the image in log scale, and some of the other points above are ones that I have based on my use, but thinking those with other ds9 preferences will have different questions so possibly best to refer to some link, if such is available? I couldn't find exactly what I wanted from the imviz link above nor jdaviz links in the notebook.

[abasuzych]

Comments RE: data_find_download notebook

• re: markdown "the search returned 12 entries": Note that there are 13 matches in numaster ngc 4151 today!🙂 Another case where might have to change to "several" and not specific number since this will likely change!
• [minor] might want to add slightly obvious note that the 6000111100* directories are now directly downloaded to their working directory and/or maybe change this to some defined "data" directory.
• re: data_access vs data_find_download: I like that there are separate notebooks. I think there might be a bit more connection and cross-referencing in both to the other but I like that the data_find_download is specific and simple. I think the data_access is more general. So in the data_find_download case, it might be useful to reference the data_access notebook to explore other ways to search and access the archive with notebooks. Conversely in the data_access notebook, it may be helpful to add (at the end) that the data_find_download shows how one might download the data directly once they have found the ones that they are interested in working with... etc.

[abasuzych]

Now I am wondering if organizing these cookbooks a bit more makes sense... some are generic to heasarc (data_find_download, data_access, jdaviz, ...) and others are mission-specific (rxte, ixpe, etc.). As we add more it seems that some org or structure would help people find relevant ones. Are there better ways to tag so users can find the "tools" within the notebooks since these are largely serving as examples to use when building their own use cases so wonder if the mission specific ones are really specific or show-casing some tool that might be useful in other cases also? Something we might want to think about and discuss?

[trjaffe]

My only objection to the download notebook is that to me it doesn't belong in the sciserver set. Part of the point is that the data are already available locally. Maybe change that part to say something like, "Here's how to find them on sciserver. Alternatively, if you are not on SciServer, here's how you can download them."

[zoghbi-a]

My only objection to the download notebook is that to me it doesn't belong in the sciserver set. Part of the point is that the data are already available locally.

This is a good point. When I didn't originally, it was not meant for sciserver, but we don't have another place to put it (NAVO notebooks?).

Maybe change that part to say something like, "Here's how to find them on sciserver. Alternatively, if you are not on SciServer, here's how you can download them."

I think there is a need to for something like this even in sciserver where the location of observational data is not always trivial (e.g. It is not always {mission}/{obsid}). I like the suggestion of including both options.

[abasuzych]

Ah... I see. Yes, I agree that it is slightly strange to download in sciserver but also really agree that finding within the HEASARC volume is difficult and not always obvious. Does this notebook belong outside sciserver and then might have the data_access notebook point to this link? Then I think the data_access notebook should have some "how to access within HEASARC volume"... hmmm. Tess' suggestion may be best for short term but I think brings up the good point that access from within sciserver to files in the HEASARC data volume should be easier!

[trjaffe]

(e.g. It is not always {mission}/{obsid})

But if you have a URL, then you can say the user can remove everything in front of the "/FTP" and it should work.

[zoghbi-a]

(e.g. It is not always {mission}/{obsid})

But if you have a URL, then you can say the user can remove everything in front of the "/FTP" and it should work.

yes. I will make changes that reflect that.

[zoghbi-a]

I think I addressed these in 5263f64

[zoghbi-a]

RE: jdaviz_demo notebook:

Excellent! This is really very cool! I'm adding my comments here:

• (minor)... at imviz.show() step might add a note that this take several seconds to load... I was getting worried and wasn't sure if I need to do anything and started clicking around on the window!

• I suggest more instructions for the selecting regions step. Creating subset 1 was easy enough but it wasn't obvious how to create a new region subset2 -- I kept overwriting subset1 somehow!

  • e.g. before the "We now proceed..." markdown add something like "In the imviz window above, select regions by clicking first for the source region and resizing (this will be designated Subset1) ... To create a new region for the background ... " and possibly add a screengrab (or mark in screengrab above, which is missing this part:) that includes the bar above with "Import data" etc to show that the red dot allows changing, deleting, adding subsets, etc.
• some suggestions for extensions: how can one display the previous region files on this viewer (i.e. next time load the viewer with the regions already marked? Can we use other region shapes e.g. annular/rectangular regions? If these suggestions are too specific, maybe provide links to the imviz page: https://jdaviz.readthedocs.io/en/latest/imviz/index.html. Can we save the region files in WCS coords? (I can see wanting to overlay the saved regions on other datasets and would want to know that this could be saved/converted reliably to WCS, etc.)

Thanks @abasuzych, I replace the static image with a GIF to show the steps, and added some text to explain how to select the background region.

My knowledge of jdaviz is limited, but I can see that it can do regions with different shapes. For WCS, I think it uses the region pakcage, so one has to use that and/or astropy to save the regions in WCS. We may do that separately. saving a region file in image coordinate should be enough for most of X-ray pipeline needs.

[mfacorcoran]

jdaviz_demo notebook comments:

• it seems to me that it might be good to make this more "heasarc-specific" by showing how to extract a spectrum from the nustar data using the defined regions using nuproducts (for example).

• The viewer object is defined but not used. Might be good to show how viewer options can be set by including a cell like:

viewer = imviz.default_viewer

viewer.zoom(1) viewer.zoom(2) viewer.set_colormap('Hot') viewer.stretch = 'sqrt' viewer.cuts = [.5, 30]

which displays the image using a "blackbody" colormap with a square root stretch with 0.5 < pixel val< 30

• it would also be good to show how we coulld find the chandra image of the field (assuming it exists) using pyvo and display it using imviz side by side with the nustar image - see https://www.youtube.com/watch?v=vZHiuDuKheQ

• I like displaying the viewer in a sidecar: imviz.show(loc='sidecar')

ª the output ds9 region file is written in pixel coordinates, which means the image binning is important when running nuproducts. I'm not sure if imviz supports writing regions in RA, Dec, radius.

• nuproducts allows the user to specify circular source and background regions as parameter inputs to nuproducts, so it's a bit redundant (& overkill?) to define these circular regions using imviz

[zoghbi-a]

Thanks @mfacorcoran, I added the viewer customization examples and a link to the NuSTAR analysis notebook to address the first point.

It was not immediately obvious to me how to save the regions in WCS. If anyone knows how to do it, please let me know.

[mfacorcoran]

I’m not sure saving in WCS is possible yet in imviz… at least I couldn’t find out how to do it…

[abasuzych]

I think I found a few options for WCS but haven't had time to test. I am putting ideas here in case anyone else wants to beat me to this. I will look into these more tmrw:

• https://jdaviz.readthedocs.io/en/v2.0.0/api/jdaviz.configs.imviz.Imviz.html (note the link_type='wcs' option).
• The other potential work-around is using pyregion to convert from one system to another (https://pyregion.readthedocs.io/en/latest/getting_started.html ).

[abasuzych]

Ok - this is a little brute force-y, but I think it works:

from astropy.wcs import WCS from astropy.coordinates import SkyCoord from regions import PixCoord

wcs = WCS('nu_image.fits') regions['Subset 1'].to_sky(wcs=wcs).write('source_wcs.reg', format='ds9', overwrite=True)

let's check the created region files:

!cat source_wcs.reg

For fun, checking that this works in reverse and it does!

load the image

imviz.load_data('nu_image.fits')

load the saved region file

imviz.load_regions_from_file("source_wcs.reg")

display the tool

imviz.show()

[zoghbi-a]

Thanks @abasuzych for figuring it out. That's great. f09a423 now includes that information.

[zoghbi-a]

This ready to be merged. If there are other comments, please let me know.