save intermediates

[hdiamondlowe]

I continually find I would like to take a look inside of the intermediates data structure. As of now it looks like it exists only within Aperture.py. Useful data saved in the intermediates data structure are:

• The original data stamp that Aperture.py acts on when doing the extraction
• The estimated sky background that is subtracted from the stellar spectra.

Everything else of interest can be loaded from already saved data structures (e.g., calib, trace, etc.), or else simply derived from already saved data structures. Perhaps these two intermediates features can be added to the extracted data structure, which is saved for each exposure.

[zkbt]

Yeah, agreed -- it'd be nice to be able to look at these after the fact, and putting them in extracted would be the easiest way to do that. Do you want to implement that change?

[zkbt]

(Or have you already?!)

[hdiamondlowe]

Not yet! I'm working on my LHS 1140 paper, and that data is reduced already. I will be getting LDSS3C observations of LHS 3844b in a couple of weeks and I'm planning to make some updates to mosasaurus for that data set. I agree that sticking these in extracted would be simplest!

[hdiamondlowe]

Update: While it is indeed easy to save some of the properties from intermediates into extracted, it is incredibly redundant and increases the size of the extracted.npy files from order kB to order GB.

To address this I think the values should be saved as such:

• the original data stamp can be saved once per extracted file (it does not change for different extraction widths)
• the extractMask and extractSky are the same for every aperture, and do not change from one extraced file to the next, so these can be saved in their own .npy data structure in each aperture folder
• the sky estimate is unique to each extracted file and each extraction width, so this will have to be saved many times and there's no way around this

[hdiamondlowe]

I do not think that saving the original data stamp and the sky estimates is reasonable, given that it makes the files to big (on order MB; 'GB' from earlier was a typo, but still).

But I have modified mosasaurus to save extractMask and skyMask in a .npy file for each aperture, which does not take up that much space and may come in handy.

Finally, I have added a new diagnostic plot that takes cuts in the spatial direction across the science stamp for each aperture. This helps to inspect the profile, along with the sky and extraction regions, at several points in the spectrum. This was the main use I had for saving the sky estimate, but now it is a default plot. An example is shown here: diagnosticCuts_aperture_391_1133.pdf*

*Caveat! I am not well-versed in the many capabilities of craftroom's display.py. It is likely that this figure can be refined to be more in-line with other mosasaurus figures, but perhaps @zkbt will be more adept at doing this :)