Getting, applying metadata from FITS images

[ktfhale]

Since the pipeline needs to discriminate on the basis of a at least a FITS image's instrument and camera, we need to extract those cards from the header. The natural place to do this is in imaging_pipeline.py. It makes sense to write a new function in there. We'll have imaging pipeline(), which runs on every image, and which which calls run_cosmicx(), run_astrodrizzle etc, call it before it calls any of the other driver functions.

[ktfhale]

I've created a new branch, metadata, for this work. I've added a function, get_metadata(), which takes a path/filename of a FITS image and returns, in a tuple of strings, the instrument and the detector. WFPC2 images seem to have no DETECTOR card, so in that case the returned tuple is ('WFPC2', None).

EDIT: I'm changing the return format to a dictionary, with the keys 'instrument' and 'detector'.

Now we need to have the function imaging_pipeline() call get_metadata. It's still up in the air how/where we're going to put the logic for deciding how run_astrodrizzle() and run_cosmicx() operate based on these different variables.

[ktfhale]

At first glance a downside of this approach is a single extra time we need to open and close each FITS file with astropy.io. The alternative is, for instance, to have run_cosmicx() retrieve the necessary header information itself when it opens the FITS image.

But, we still need to have run_astrodrizzle() know what the header information, and run_astrodrizzle() itself never opens the FITS image like run_cosmicx() does. So we'd still need an extra open/close, but it would only be in run_astrodrizzle'. With this approach,run_astrodrizzle()andrun_cosmicx` would be getting the header information they need separately, but they would be using duplicate code to do it.

So, I think it's better to have something like get_metadata() run at the imaging_pipeline.py level, and add instrument and detector as input parameters to run_cosmicx() and run_astrodrizzle(). Inside them, they'll call a helper function that goes through the decision tree and retrieves (or sets the path to the file, in astrodrizzle's case) the necessary parameters from a configure file.

I'm planning on using settings.yaml and get_settings.py as a model for a WFPC2_cr_params.yaml settings file and a get_cr_params(). helper function. I figure get_cr_params() can go inside run_cosmicx.py.

[ktfhale]

At today's code review, it was suggested we set the gain and readnoise values for the cosmic ray rejection using information in the header. I've emailed Max about what keywords we should use, because it's not entirely clear.

WFPC2 has ATODGAIN, but nothing, so far as I can tell, about readnoise in the header. ACS and WFC3 headers have CCDGAIN, which is the "commanded" gain, but they also have four different ATODGN<A,B,C,D> keywords, and four READNSE<A,B,C,D> keywords, one for each of the four amplifiers, A, B, C, and D. I'll wait to hear back from Max about what to do with these before I actually implement grabbing them from the header, but I'll keep the need to do so in mind as I work on feeding cosmicx the detector-customized parameter sets.

Also, I just realized we don't need to know the instrument. The detector is sufficient to uniquely identify which parameter set we should use.

[ktfhale]

Last week I finished making the changes that let us specify cosmicx parameters in yaml files unique for each detector. These have been placed in a new folder at MTPipeline/cosmicx_cfg. In each file, it is necessary to specify the parameters for each individual science extension, even if the extensions are all to be treated identically. Note that, for PyYaml to interpret something as Python's None object, you need to use null or ~. Just putting None will result in the string 'None'.

In the course of testing this I also happened upon some raise exceptions in lacosmicx.py that were invalidated as of Python... 2.6, I think? They were inside if statments that tested whether in-mask was correctly formatted as either a NumPy array or as None. They tried to raise strings as exceptions, a feature that has been removed. I've corrected them to raise TypeException, accompanied by a string.

The pipeline successfully runs on some sample WFPC2 and UIVS data, and it passes the tests both locally and on Travis. It doesn't currently generate png's for UVIS data, however.

[ktfhale]

It can't generate the png's because it can't run astrodrizzle on the UVIS data yet, since we haven't written anything that lets 'astrodrizzle` run on anything other than WFPC2. But CR rejection seems to run just fine, although we don't have fine-tuned parameters for UVIS yet.

I think this branch is ready for a pull request.

[ktfhale]

Now that the merge is complete, (see Ticket #119), this issue can probably be closed. We will likely want to pull other metadata out of the FITs headers in the future (like the readnoise and gain, see Ticket #115), but we can decide that in their own tickets, now that we have the capability to do so.