PhangsTeam / pjpipe

Pipeline for processing JWST imaging data, tailored for nearby galaxies. Built for PHANGS
https://pjpipe.readthedocs.io/en/latest/
GNU General Public License v3.0
14 stars 1 forks source link

[FEATURE]: Add pipeline and DR version to file headers #44

Closed low-sky closed 10 months ago

low-sky commented 11 months ago

It's time to add pipeline version or DR version into the FITS headers of the results. Note that the “anchor” files have this versioning information:

FILENAME= 'ic5332_nircam_lv3_f360m_i2d_align.fits' / Name of the file
SDP_VER = '2023_2a '           / Data processing (DP) Software Version
PRD_VER = 'PRDOPSSOC-063'      / S&OC Project Reference Database (PRD) Version
OSS_VER = '8.7     '           / Observatory Scheduling Software (OSS) Version
GSC_VER = 'GSC2431 '           / Guide Star Catalog (GSC) Version
CAL_VER = '1.12.3.dev26+g9dfaa3724' / Calibration Software Version
CAL_VCS = '9dfaa3724'          / Calibration software version control sys number

The FILENAME should be shifted to anchor and we should add something like PIPE_VER = ‘1.0’ / PJPIPE version into the header and I certainly wouldn’t mind a PIPE_VCS github hash as well. Since there’s rapidly developing a notion of a data release versioning that is separate from the software, we might want a DR_VER keyword as well

thomaswilliamsastro commented 11 months ago

Agreed on all of this, I would suggest just:

PJPIPE_VER DR_VER

I think if people are using a github rather than pip-installed version, we can include the hash in the version, the STScI folk have this in the init which may just work for us:

import re

from importlib.metadata import version

__version__ = version(__name__)

_regex_git_hash = re.compile(r".*\+g(\w+)")
__version__commit__ = ""
if "+" in __version__:
    commit = _regex_git_hash.match(__version__).groups()
    if commit:
        __version_commit__ = commit[0]

I'm not sure why the name isn't updating, this may be due to some interface between using astropy.io.fits instead of stdatamodels.jwst.datamodels

low-sky commented 11 months ago

Great! I note that PJPIPE_VER is longer than the 8 characters allowed for FITS header cards but any 8 character string works.

thomaswilliamsastro commented 11 months ago

PJP_VER?

thomaswilliamsastro commented 10 months ago

This is addressed in #48

It's a little fiddly given how datamodels interfaces between metadata and fits headers. It also seems to delete things when you run through some of the steps. So, there are now two meta attributes in the ASDF, meta.pjpipe_version and meta.pjpipe_version which map to PJPIPE VER and DATA PROCESSING VER in the history at the bottom of the fits headers. There's no good way to add in fits keys, as far as I can tell