rrdesi option to output model fits

[sbailey]

Add rrdesi --model filename.fits option to output pre-rendered models. A few choices that need to be made:

• just the best fit model, or all models (could get big)?
• output to separate file, or include in the redrock*.fits catalog file?
  • having it in the same file is convenient when you want the models, but would make those files considerably larger, which is less convenient if you just want the catalog.
• If the fit includes Legendre nuisance terms, should those be included directly in a single evaluated model array, or should we separate the template-part of the model from the Legendre-part of the model, storing them as different arrays?

[abhi0395]

@sbailey I think having at least the best-fit model stored somewhere, which should be available when data goes public, will be extremely useful not only from an outsider's point of view but also within the collaboration for people who are not very familiar with the data format.

Now, there can be a few choices:

1) Adding the best-fit model in the coadd*.fits file, where the spectra and the model can be extracted simultaneously. I understand this is a bit complicated as they are written before Redrock. Also, this makes the file size considerably larger.

2) A file (catalog) with TID, tile, petal, z, spectype, and model. This is less complex, but I don't know if that's a practical solution?

In any case, we should provide a small function to extract the model and spectra from any of these files?

Archetype+Legendre terms: Yes, this is an important point; if we want to include both the PCA and Archetype models, then (2) seems a better solution. But we will need surgery in the redrock to store those coefficients separately.

I am happy to help with this issue.

[sbailey]

@abhi0395 thanks for volunteering for this.

Let's not post-facto add it to the coadd files, since that would break the "write once read many" paradigm for pipeline dataflow and it would make those files even larger.

Let's keep the models in their own separate file, not adding to the existing redrock.fits files since that would make them much much larger to download. We could use a structure like the existing coadd files where there would be `_MODELHDUs matched to the_FLUX` HDUs of the coadd file. e.g. if Redrock was run on a coadd file with separate cameras B_FLUX, R_FLUX, Z_FLUX, then it would create an output file with matching image HDUs B_MODEL, R_MODEL, Z_MODEL. But if it was run on a coadd already coadded across cameras BRZ_FLUX, then it would output BRZ_MODEL. i.e. it always makes a model to match the structure of the input spectra, whatever that is. That model file could also include the REDSHIFTS and/or FIBERMAP (or fibermap subset?) HDUs from the original redrock.fits files to identify which rows are which. Note that we also need to support healpix-based spectra, i.e. handle the case where TILEID, PETAL, FIBER don't apply.

Let's start with the "small function to extract the model and spectra", regardless of the details of how we store that in a file. redrock.templates.eval_model() and redrock.archetypes.Archetype.eval are starting points for that.

I'll open a separate ticket about how to store PCA/NMF vs. Archetype vs. Legendre coeffs, which is a related but somewhat different question from how we store the models.

[abhi0395]

@sbailey Thanks for the suggestions. Sure, I will start looking into these small functions for doing such specific tasks.

[stephjuneau]

Hi @abhi0395 , adding a comment here to re-iterate that I'll be interested in trying out the function that you are writing and in particular testing it on spectra after they have been coadded across cameras. Please keep me posted!

[abhi0395]

@stephjuneau Thanks, sure I'll keep you posted.

[sbailey]

Implemented in PR #283. Closing.