Add the option to generate annular spectra with cross-ARFs

[DavidT3]

I realised that I never really documented this enhancement.

Currently sets of annular spectra have 'normal' ARFs generated for each spectrum of each annulus, without regard for PSF effects that can cause the scattering of photons of different energies from one annulus to another (bar the default annulus widths in XGA which are meant to help avoid PSF problems).

There is a more sophisticated treatment, built into arfgen, called cross-ARFs; this uses a detector map and a model of the PSF to calculate ARF corrections for PSF effects - though at the cost of compute time.

The option (though not the requirement) to use cross-arfs will be included in XGA (it's very simple to do).

[DavidT3]

I have deleted the original efforts I made on this issue, and I am starting again.

[DavidT3]

It is taking a very long time to run each at the moment, far too long to actually be practical to use at the scale we need (even with many cores) - I suspect this is because the det map I'm passing to arfgen at the moment is quite finely binned (I'm just using an existing one from a spectral generation run.

I need to make the detmap use in cross_arfs more robust, but also allow the user to make choices about the binning (both for this and for general spectral generation). Also the default needs to be coarser binning because this is intolerably slow.

[DavidT3]

My default detmap generation uses a binsize of 100, this must be increased for the default behaviour I think.

I will also add a specific common (internal) function that generates the commands for the detmap generation - this is because there is a whole process to determine which camera to use, because ideally you'll use a different camera to the one your calculating areas for (chip gaps are different), and it would be sloppy to just copy-paste that into the cross_arfs function (not that I haven't been sloppy before...)

[astrophysics-megan]

Using native Xmm binning is way way too many bins. Probably in both energy and pixel space. But I suspect this is energy/PHA space?

Megan

On May 9, 2023, at 4:40 PM, David Turner @.***> wrote:



My default detmap generation uses a binsize of 100, this must be increased for the default behaviour I think.

I will also add a specific common (internal) function that generates the commands for the detmap generation - this is because there is a whole process to determine which camera to use, because ideally you'll use a different camera to the one your calculating areas for (chip gaps are different), and it would be sloppy to just copy-paste that into the cross_arfs function (not that I haven't been sloppy before...)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/DavidT3/XGA/issues/917*issuecomment-1540259396__;Iw!!HXCxUKc!xR2vy0FQFbjLnJokq7xGyrRQ1_2iD4PnrTAIAjcMWgmBy0RpkA8D1KCGLGIa7O66rQnrMArjjxZwPmGgnP3DBpiRhQ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ASKK4P54XGED2HTLXKVATGDXFJJMRANCNFSM6AAAAAATZWCFXA__;!!HXCxUKc!xR2vy0FQFbjLnJokq7xGyrRQ1_2iD4PnrTAIAjcMWgmBy0RpkA8D1KCGLGIa7O66rQnrMArjjxZwPmGgnP1HmrE42Q$. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[DavidT3]

This refers to the spatial binning for detector maps used in arf generation - I wasn’t using native binning (I think my computer would explode if I did that for arf generation) - the previous XGA default was a bin size of 100, the new default is 200 which I think is more reasonable.

Spectra generated by XGA are re-binned to achieve a minimum number counts/SN set by the user.

I’m making more progress with this crack at cross-arfs than last time, I found an unintended behaviour in my AnnularSpectra class this time around which was causing all the issues I had last time.

Now my output folders are filling up with 100s of cross-arf files :D

On 9 May 2023, at 13:02, astrophysics-megan @.**@.>> wrote:

Using native Xmm binning is way way too many bins. Probably in both energy and pixel space. But I suspect this is energy/PHA space?

Megan

On May 9, 2023, at 4:40 PM, David Turner @.***> wrote:



My default detmap generation uses a binsize of 100, this must be increased for the default behaviour I think.

I will also add a specific common (internal) function that generates the commands for the detmap generation - this is because there is a whole process to determine which camera to use, because ideally you'll use a different camera to the one your calculating areas for (chip gaps are different), and it would be sloppy to just copy-paste that into the cross_arfs function (not that I haven't been sloppy before...)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/DavidT3/XGA/issues/917*issuecomment-1540259396__;Iw!!HXCxUKc!xR2vy0FQFbjLnJokq7xGyrRQ1_2iD4PnrTAIAjcMWgmBy0RpkA8D1KCGLGIa7O66rQnrMArjjxZwPmGgnP3DBpiRhQ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ASKK4P54XGED2HTLXKVATGDXFJJMRANCNFSM6AAAAAATZWCFXA__;!!HXCxUKc!xR2vy0FQFbjLnJokq7xGyrRQ1_2iD4PnrTAIAjcMWgmBy0RpkA8D1KCGLGIa7O66rQnrMArjjxZwPmGgnP1HmrE42Q$. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/DavidT3/XGA/issues/917*issuecomment-1540543693__;Iw!!HXCxUKc!zUJ3UrYeHMKxpXt3fB_RxIV97h9Qhmu43Kaa2KEgYKgGudDB4_MxbAf478Xpapd91nquutC0Rsjf6TMxytiQdUMFlg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AEHMFK7U4MEKZTVPLJKRZE3XFJ2AZANCNFSM6AAAAAATZWCFXA__;!!HXCxUKc!zUJ3UrYeHMKxpXt3fB_RxIV97h9Qhmu43Kaa2KEgYKgGudDB4_MxbAf478Xpapd91nquutC0Rsjf6TMxytjPFo_LUg$. You are receiving this because you were assigned.Message ID: @.***>

-- Dr David Turner (he/him) Research Associate - Michigan State University Astronomy Group GitHub - https://github.com/DavidT3 —

If I am sending emails outside of standard working hours, I may be working flexibly. Please be assured that I do not expect a response outside of your own working hours.

[DavidT3]

Note to self, need to make sure cross-arfs are read back in from the disk when a source/sample is re-declared and allowed to load already generated products. Currently they have to be regenerated each time, which is obviously daft.

[DavidT3]

Doing a manual practise of cross-arf XSPEC fitting, will update comment with lessons learned:

• The 'diag' responses used by examples are unitary matrices for residual SP models
• When adding more rmf and arfs to a spectrum group resp must be set first, then arf - makes sense in retrospect
• Example doesn't ever use a 'source number' more than once, though from what I can glean from XSPEC's docs source number are on a per-spectrum basis. Admittedly the XSPEC docs aren't explicit about that, but we'll see. [in fact the default source number is 1 so they must be per spectra - not sure why the example doesn't ever re-use them] correction - the source numbers do matter when you start declaring models that act as crosstalk between spectra

[DavidT3]

I think I've got a handle on this now - I used XGA to generate annular spectra and all the ancillary gubbins that go with it, as well as cross-arfs, and grabbed the paths and implemented an XSPEC manually with cross-contribution accounted for. The annular spectra have been generated for A907, the radial boundaries were pulled from thin air, and have no particular physical meaning (0-180-330-500 kpc, three annuli). 180kpc translates to ~65arcsec for A907, so really could have chosen better for a test of this but oh well.

A907 has three observations, but for the manual test I just used 0201903501, all three instruments, so each annulus has three spectra. Used a consttbabsapec model, frozen nH, redshift, metallicity - each annulus has kT and norm linked across the three spectra, the constant for each is not linked, though the constant for the 'first' spectrum in each annulus is frozen at 1.

Cross-contribution is taken into account with setting up a consttbabsapec model for each annulus's contribution to each other annulus (on an instrument by instrument level); i.e. there are 12 cross talk models for the three annuli with three spectra each.

The same setup, but without the cross contribution models, was used for a comparison fit (left is with cross arfs, right is without):

So it is certainly having something of an effect. Also attaching the xcm files for my future reference, as now I need to start the much harder task of implementing this generally in XGA, and the practise scripts are not a part of XGA.

no_crossarf_comp.txt practise_crossarf.txt

[DavidT3]

Given I can now easily generate and store cross-arfs, I will round off the current dev branch (feat/crossARFGenerationAndStorage), and start a new branch related to the fitting process.

[DavidT3]

Note to self, need to make sure cross-arfs are read back in from the disk when a source/sample is re-declared and allowed to load already generated products. Currently they have to be regenerated each time, which is obviously daft.

This note to self didn't help very much, I forgot to implement this before merging the generation and storage branch - I will make sure to do it in this fitting branch that I'm working on now.

[DavidT3]

Actually the model setup (for the other annulus contributions) isn't quite right yet, as the constants are just set to one. Really I need to ensure that each cross-arf model component is linked to its specific model parameters from the 'original' model.

[DavidT3]

Parameter linking has gone wonky for some reason - when I was testing with more annuli etc the fits are failing with nonsensical parameter values (some which are way higher than the allowed values). Then when I look at the model setup this is what I see:

The linking is happening between the complete wrong parameters, so no wonder I have mad parameter values and everything is breaking!

[DavidT3]

The cross-arf files are not being saved with the names I intended - the detector map information isn't in there for instance.

Some of the XSPEC script names being generated are longer than the file system character limit (we've been here before) so time for another inventory file system I guess. I am considering this a triage effort until I add a proper backend database to keep track of everything, I think it would definitely be better than this.