Image deconvolution with spacecraft attitude binning method

[hiyoneda]

I updated the image deconvolution code using the spacecraft attitude (scatt) binning method.

There are two new classes.

• exposure_table.py: it analyzes the orientation file and defines the indices of the scatt binning.
• coordsys_conversion_matrix.py: it produces the coordinate conversion matrix (ccm) using the exposure table.

I also uploaded two notebooks demonstrating the image deconvolution with the Crab data in DC2. First, you can run through Crab-DC2-ScAtt-DataReduction.ipynb in docs/tutorials/image_deconvolution/Crab/ScAttBinning. This notebook explains how to produce the binned data using the scatt binning method. Then, the image deconvolution with the produced datasets is presented on Crab-DC2-ScAtt-ImageDeconvolution.ipynb in the same directory.

[hiyoneda]

I added notebooks for 511 keV analysis. And the notebook named ***-DC2-ScAtt-Upsampling.ipynb shows the image deconvolution by better pixel resolution in the model space.

[hiyoneda]

I modified the code sometimes after I had opened this PR. Now it is ready to be reviewed.

[ckierans]

I ran through the Crab data reduction and imaging notebooks with my 16 GB memory computer and had no issues! The Crab image deconvolution took just over an hour. The results that I got were not identical to yours, especially noticeable in the spectrum and also in the likelihood ratio - see attached photos.

Not sure where the difference comes from, or if it's unexpected, but I did notice my response name was slightly different: SMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5 (can't remember what the good_chunks was all about).

Working my way through the 511 data reduction now.

[KeigoOkuma]

I ran the 511 keV image deconvolution and had no issues. There is a typo or something in the part of data reduction that downloads files from wasabi. Which is correct, 511_thin_disk_3months.fits.gz or 511keV_3months_unbinned_data.fits.gz?. Calculating the coordinate conversion matrix in Upsampling took 2 hours and 40 mins with my 32 GB memory M1Mac.

[KeigoOkuma]

I also ran GRB image deconvolution and had no issues.

[israelmcmc]

I haven't run this yet, but I just wanted to clarify that "good_chunks" refers to choosing optimal HDF5 chunks. Depending on how the code is accessing the response matrix, it could make the code run faster, but it shouldn't change the results.

[hiyoneda]

@ckierans Thank you for pointing this out. It may be caused by small changes in my code, e.g., very small but finite constant value to be added in the expected count calculation, a parameter in boosting EM algorithm. Actually, the responses in the old and new formats have a tiny difference between them (though it seems negligible). I run through the notebook again with the update code and the new response file, and the results are consistent with yours. I will update the figures.

[hiyoneda]

I updated the figures on the Grab notebooks and modified the types on the 511 keV notebooks.

[fieldrog]

I've been working through the the notebooks in docs/tutorials/image_deconvolution/511keV/ScAttBinning. All run in my environment.

Here are a few comments:

• it would be nice to use same data structure as on wasabi, with path_data pointing to some DC2 directory to be defined (to avoid having to re-define directories if you already downloaded the whole thing)
• in the 511keV-DC2-ScAtt-DataReduction nb, cell 5 has a call of undefined variables (nside, npix) though it doesn't seem to impact anything else; might be a typo. Edit, maybe this was already fixed since i pulled this PR
• would be nice to add a tqdm option to the spacecraft_attitude_binning_ccm function. This took 20 min on my system with use_averaged_pointing = True in the 511keV-DC2-ScAtt-DataReduction nb. In the 511keV-DC2-ScAtt-Upsampling nb, took 2:48, all of which seems to be consistent with expectation.

I have 16GB memory on my M1 mac.

[hiyoneda]

@fieldrog Thanks! I think the second one is already fixed. So, I will work on the first and third comments. Can I ask about the point of the third point? Even in the current code, the tqdm bar appears when running spacecraft_attitude_binning_ccm. Do you think it is better to add an option not to show it?

[fieldrog]

With the %%time magic option, i just get the following output after the cell is running:

CPU times: user 5.2 s, sys: 562 ms, total: 5.76 s Wall time: 6.3 s

I don't get any time bar as the code is running, it just says something like this:

0%| | 0/7979955 [00:00<?, ?it/s]

until the block finishes running.

Whereas with a tqdm option in the loop in the function, it could give an optional progress bar, in a notebook or in a script.

Unless something is wrong with my time magic!

[hiyoneda]

I see, but I already used tqdm in the loop in this function. In my case, the progress bar appears as normal, so it sounds strange. Did the bar appear when removing %%time?