Write steps and pipeline following JWST pipeline structure

[vandalt]

Currently, the pipeline and its various steps are similar to the JWST pipeline from the user's perspective, but they don't follow the same code structure and inherit the same classes. Following the JWST structure should remove some complexity from the code and make it easier to eventually make this an official JWST pipeline.

There is documentation here writing a pipeline and a step.

I'm not self-assigning for now in case someone else wants to work on this in the short term, but I'm happy to look at this in the next few weeks.

[kammerje]

I agree that this would be great. However, I think we should check with the pipeline folks first if they would be willing to add the KPI3Pipeline to the official JWST pipeline.

[vandalt]

Hi @kammerje I now have some experience with the JWST pipeline interface. I feel like, even if in the end this is not merge in the official pipeline, using the same building blocks would help a lot for reducing boilerplate code (related to saving outputs, skipping steps, etc) and would also simplify the inclusion of new/external pipeline steps (e.g. someone writes a step for another mode that would be useful for KPI, it can just be added to the pipeline and be run as is).

I think the main downside of this approach would be file I/O: pipeline steps expect Data models jwst.datamodels, which are handled a bit differently from FITS files. A temporary solution to avoid having to create a data model would be to shortcircuit the I/O with custom functions to write KPFITS files.

I have a somewhat clear idea of how this could be done and have started some tests on a dedicated branch. Let me know what you think @kammerje.

[kammerje]

I'm not sure if I understand this correctly, but the official pipeline also inputs and outputs FITS files. I'm guessing the FITS file handling is only wrapped into the pipeline stages and under the hood, all steps are called with datamodels. It is right that this will be an issue for the KPFITS file which doesn't have a corresponding datamodel.

[vandalt]

Exactly, once this FITS is read, everything else in pipeline code assumes datamodels. This is what I meant.

I'm working out a datamodel for KPFITS file right now. It's almost working, but I still have a few problems to fix. I'll keep you posted and probably open a draft PR once it works.

[kammerje]

@vandalt I just ran the Kpi3Pipeline (develop branch) with the new version of the JWST pipeline (1.13.4) and no FITS files are created anymore.

[vandalt]

I just ran an example script (NIRISS) with pipeline 1.14.0 and it works on my side.

Did you set pipe3.save_results = True ? Otherwise no FITS files will be saved, following the JWST pipeline default behavior. Then for each step you can also do, for e.g., pipe3.trim_frames.save_results = True to save an intermediate FITS file for that step.

[vandalt]

@kammerje does the suggestion from the comment above fix the issue on your side? For me this seems to work