Add a DAXA wrapper for the reprocessing script

[DavidT3]

I have decided to go the same route I've taken with SAS, and Jess took with eSASS, wrapping the CIAO tools with a Popen call - that way all the machinery can basically stay the same, and I get the same logging etc.

Also means we can support installations of CIAO that haven't been installed into the current environment.

Nothing against their Python implementation though.

CIAO provides some ideas of how to implement it like we will (https://cxc.cfa.harvard.edu/ciao/scripting/runtool.html) but I've done this enough times that there isn't really anything new there.

They do explicitly say to watch out for colliding processes accessing pfiles - good to be told, we should be able to handle that.

[DavidT3]

We'll make a new PFILES directory and alter the environment variable. I was looking at their context manager for this 'new_pfiles_environment' - which is undoubtedly a nicer way to this, but I won't copy them for now.

They also copy over the 'ardlib.par' parameter file - I'm not gonna do this right now, but perhaps will if something is clearly awry

[DavidT3]

I am currently retrieving and storing the following files from the chandra_repro run:

• The evt2 file (of course) - this is still set up to assume there will only be one, so will not yet work with alternating exposure mode data (this is because I'm not yet sure how I will deal with data that are from the same exposure but shouldn't be added together in terms of the larger DAXA infrastructure).
• The flt1 GTI file - just going to keep this even though it should have been applied to the evt2 output file I think
• The new bad pixel file - very probably will be needed for things like generating new images (and also in XGA, spectra).
• The FOV file - not sure if it'll be useful but figured I would save it

There are other files I suspect I should keep, the asol and eph files for instance, but haven't yet done anything with them.

[DavidT3]

The chandra_repro function now works properly within DAXA infrastructure!!

[DavidT3]

I suspect there is still some fine-tuning that needs to be done (over and above the missing features mentioned in issue #333 ) - from what I've seen on the CIAO documentation site, HRC processing can produce some different files, and I haven't specifically tested it out yet.

[DavidT3]

Will now test the DAXA implementation of chandra_repro on a larger set of observations - finding relevant observations for the ACCEPT 1 sample springs to mind.

[DavidT3]

Distilling the notes I had in the chandra_repro docstring before I actually wrote it properly:

NOTES:

• We will always produce new bad-pixel files, as we don't currently allow the user to pass their own
• We will also always produce a new event file with 'process_events', so we don't allow any choice. We want a new calibrated level 1 (and from there level 2) event files in all cases.
• QUESTION FOR ME - set_ardlib controls whether the observation bad-pixel file is stored in the ardlib and (I think) doesn't have to be supplied to other analyses. This won't play well with multi-processing I think, and I do remember they mentioned ARDLIB in the multi-processing part of their docs. I'll either have to do it that way or see whether the bad pixel file can be passed manually for each analysis we might do (which honestly is probably the way because XGA is gonna be doing all that stuff). 'set_ardlib=FALSE' FOR NOW!!
• Following CIAO docs advice for the 'check_vf_pha' parameter and having it set to False by default (https://cxc.harvard.edu/ciao/why/aciscleanvf.html.)
• Don't fully understand 'pix_adj' yet, but should probably include if I don't understand it well enough to not be able to argue against it
• 'tg_zo_position' - this defines the coordinate of the target to be reduced in a grating observation, the 'zero point' - we run into the same problem as processing RGS that we just want to prep the data without making spectra, ARF, RMF, etc. - as this tool just prepares the data, and the user might not want to ultimately look at the brightest source in the field. NOT SURE WHAT TO DO YET - STARTED BY LEAVING IT ON THE DEFAULT BEHAVIOUR
• 'asol_update' - again don't know why you wouldn't want to do this, but we'll leave the choice in
• 'pi_filter' - for the grating spectra, a low-cost way of lowering the background it seems? I'll leave the choice and it'll be on by default
• I will initially set verbose to 5, to store the maximum amount of data for debugging
• The 'root' option, which I think essentially controls the prefix on the generated files, will be set to the ObsID + instrument + randomly generated ID I think, just to be totally sure which ones we made (also while I don't fully understand the behaviours of chandra_repro in practise I think it'll be a good way of tracking what it is doing

[DavidT3]

Incidentally, downloading and processing data relevant to the ACCEPT 1 sample went fine - approximately 1800 observations were identified and ran through the chandra_repro step with no issues.