Conflicts with development

[stscicrawford]

The issue raised in revert #2733 I think highlights a problem with timing of development and data models. The MIRI team is trying to update the calibration files though this will be for post-Build 7.2 and at the same time, we are trying to finish off any updates for Build 7.2.

What's the best solution for this? Is it possible to have optional keywords so that we support data both in the current CRDS and the development of new reference files? Will this be something that needs to be supported more often? Or do we need to have a development release always available for the INS teams?

@hbushouse please comment if I've missed anything.

[sosey]

Aren't we tracking the versions of the pipeline and the CRDS context (pmaps) they require? I would be against using optional keywords that are slated for removal just to make the pipeline backwards or forwards compatible.

If the reference file formats are changing and that requires or depends on changes to the pipeline code, then the reference file delivery to CRDS should be coordinated with the merging of the updated software for broader use. Older CRDS files can be tested with different versions of the pipeline by installing the requisite version of the pipeline and setting the pmap back (by setting the crds context in the env).

If updates to be included with 7.2 continue on master and new code is sent to 0.13 then the teams can build the pipeline from the 0.13 branch if they need immediate testing of a new reference file with the code that supports it. If no code is needed to support the reference file changes, then they can test the new reference file locally, by sending that file into the requisite method instead of asking CRDS for it as well.

I might be missing some nuance here, but I've done the above myself in developing new code, and new reference files.

[stscicrawford]

Please add your suggestions and comments to this ticket.

Here are suggestions from the meeting for future discussion:

• Re-examining the workflow for introducing CRDS/Data models updates
• Keeping the development on master
• Adding a minimum software version to reference files

[jdavies-st]

Aren't we tracking the versions of the pipeline and the CRDS context (pmaps) they require?

We used to manually set the CRDS_CONTEXT version in our repository. That way, any particular snapshot of the repository had an associated CRDS_CONTEXT with it from which our then-current regression tests needed to pass.

We currently use the CRDS_CONTEXT=jwst-edit, which just points to the latest. This was to automate testing and make it easier to coordinate the updating context with our regression tests. But of course a side-effect of this is that we no longer have a snapshot of which context works at each commit in our repository.

Perhaps @jaytmiller and @hbushouse can comment further?

[sosey]

We currently use the CRDS_CONTEXT=jwst-edit, which just points to the latest.

Yes I know, but we should have a listing of crds context correspondence to jwst software version so that people know which context to revert to in order to test a specific version of the pipeline with delivered reference file.

[jdavies-st]

Agreed!

This can in principle be reconstructed. jwst.__version_date__ gives the datetime of any particular snapshot of the jwst repository. I believe there's also a 1-liner crds tool that gives the CRDS_CONTEXT on a given datetime for production.

[jaytmiller]

Definitely on build boundaries we know the exact context which corresponds to a build and list that in the formal SDN (software delivery notice).

The CRDS server maintains a dated history of all "operational" contexts which are basically the default used broadly for development unless overridden by setting CRDS_CONTEXT. That is visible on the top level web page at

https://jwst-crds.stsci.edu (last 4 contexts)

or digging deeper there's a table of all contexts:

https://jwst-crds.stsci.edu/display_context_history/

These one liners work with no configuration to get the near term contexts:

$ crds list --resolve-contexts --contexts jwst-edit jwst_0493.pmap

jwst-edit almost always corresponds to the last delivered .pmap.

And (sometimes a different answer) the default/blessed context:

$ crds list --resolve-contexts --contexts jwst-operational jwst_0493.pmap

Slightly more advanced, this requires a server connection to fetch "operational context at date":

$ export CRDS_SERVER_URL=https://jwst-crds.stsci.edu $ crds list --resolve-contexts --contexts jwst-2018-05-01T00:00:00 jwst_0454.pmap