justincely / lightcurve_pipeline

pipeline for high level science products of HST 13902
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

Ensure lightcurve products meet MAST HLSP requirements #19

Closed bourque closed 8 years ago

bourque commented 8 years ago

We need to make sure the composite lightcurves that we deliver to MAST meet their criteria listed here: https://archive.stsci.edu/hlsp/hlsp_guidelines.html

bourque commented 8 years ago

One thing we will have to change is the naming convention of the lightcurves. According to MAST:

The required naming scheme is:
hlsp_project_mission_instrument(-resolution)_field-name_filter_version_product.extension

hlsp - identifies the file as a community-contributed dataset.
project - An agreed upon name for the HLSP set. This name is also used as directory name and as a database keyword.
mission - Mission/missions acquiring the data. If two missions used they should be separated by hyphens; models/simulations can be indicated here.
instrument - Instrument (e.g. acs-wfc). When not applicable as for GALEX data, a description like imaging or spectra should be used. Resolution with units can be specified following a dash (e.g. 'asc-30mas' indicates ACS instrument with 30 milliarcsecond resolution)
field-name - Field name or target as designated by the team. Parts, counter numbers and epochs are allowed in this field and should be separated by hyphens. If using multiple epochs, please use suffix 'ep' followed by a number (e.g. m101-ep1, m101-ep2). Counters can be used when the same field is observered N times with same observing parameters. For parts and counters, we require leading zeros if more than 1 digit is needed (e.g. for 17 exposures of m101, each file has field-name as follows: m101-01, m101-02 ... m101-09 ... m101-17; similarly, for more than 9 epochs: m101-ep01 ... m101-ep19). Please describe your usage of field-name parts and counters within the README.
filter - Filter or filters used; or, grating(s) used. If HST this should be the full filter/grating name (e.g. f606w or ge230m). If more than one filter is used, separate the filters by hyphens (e.g. f606w-f814w).
version - Version Number, such as v#.# or v#. e.g. v2.4
product - Type of data as designated by the team (models/simulations can be indicated here). e.g. img, drz, sci, weight, wht, cat, modelmap, theory, sim, model, map and SED.
extension - Standard file extension. e.g. fits, txt, jpg

Use lower case characters only.
Use underscores ONLY to delineate between major fields (e.g. "goods_hst")
Within fields, dashes can be used as separators (e.g. "nicmos-nic3", "acs-wfc", "m101-01")
Version numbers can be specific to the project. We recommend teams use increased version numbers to indicate data which are superseded; we will NOT keep older versions of datasets unless the team explicitly requests this. Note that sometimes version number will denote single-EPOCH data as in GOODS; this sort of information must be documented well with deliveries.
We recommend data are re-delivered with higher version numbers (better products); we recommend that the newest deliveries contain both the re-processed data along with the single-epoch data associated with these products, if applicable.
We recommend that images from different filters that have the same field-name should have matching sizes and WCS coordinates. All images identified using the same field-name should cover the same part of the sky. If there are multiple images covering different parts of an object, they should have different field-names.
Examples:
hlsp_goods_hst_acs-wfc_north-sect13_f435w_v2.0_drz.fits
hlsp_angst_hst_wfpc2_11307-ngc0055-wide2_f606w-f814w_v1_gst.fits
hlsp_cosmos_galex_imaging_cosmos-01_fuv_v2_int.fits
hlsp_udf_hst_nicmos-nic3_treasury_f110w_v2_img.fits
hlsp_clash_hst_acs-30mas_a209_f435w_v1_drz.fits
hlsp_myproj_hst_stis_hd12345_ge230m_v1_sci.fits

So in our case, the files should be named hslp_hstlc_hst_<cos or stis>_<target>_<filter>_v1_sci.fits(?)

bourque commented 8 years ago

The following header keywords are missing from the composite lightcurves but are required:

TARGNAME
EXTNAME

The following are missing but are optional:

APERTURE
POSANGL
bourque commented 8 years ago

We should be careful about changing the composite lightcurve naming convention, as it might break other code like some of the plotting code. I'll double check to see if there are any dependencies on that.

bourque commented 8 years ago

Looks like there is a dependency in the make_exploratory_table function of make_hstlc_plots:

            targname = os.path.split(dataset)[1].split('_')[0]
            instrument = os.path.split(dataset)[1].split('_')[1]
            grating = os.path.split(dataset)[1].split('_')[2]
            cenwave = os.path.split(dataset)[1].split('_')[3]

If we keep the cenwave in the HLSP name, then this will be an easy fix:

hslp_hstlc_hst_intrument_targname_grating_cenwave_v1_sci.fits

bourque commented 8 years ago

See pull request #22 for code changes regarding MAST HLSPs.

bourque commented 8 years ago

See pull request #25 in the lightcurve package for changes regarding header keywords.

bourque commented 8 years ago

I believe our composite lightcurves now meet the MAST HLSP requirements.