stscijgbot
commented
5 years ago Comment by David Law: As discussed during the meeting yesterday, the mismatch between the archive structure (directories to organize data) and pipeline assumed structure (no directories, everything flat) suggests a few possible solutions:
1) MAST changes to deliver everything bundled together in one directory. That seems undesirable; with potentially thousands of files with long and complicated names it could be very challenging to tell what is what after downloading.
2) A script that converts the MAST download structure to a flat single directory for use with the pipeline. This doesn't seem to provide any advantages over (1), and might suffer from copying large quantities of files.
3) The JWST pipeline changes to allow for handling I/O in different directories. This seems much more compelling, flexible, and in line with other major data pipeline (e.g., SDSS).
This probably needs to be considered together with the planned change to cfg files. One option might be to have two top-level yaml-type files produced in the current working directory, a single config file that allows specification of overrides for any pipeline step (an example file that a user might modify could be produced by something like the collect_configs script) and a file list. The file list could be provided by MAST with any download bundle, and would specify a root input directory (presumably ./ by default), root output directory, and path to input files relative to the root input directory. Maybe with switches for whether to write exposure-level (e.g., DET1, SPEC2) into individual directories like the input structure or collect them elsewhere. Users could easily modify pipeline behaviour using these two files (instead of the current many cfg files) and have flexibility on where their data live.
Some potential use cases that would benefit from such a framework:
A) Instrument teams are reprocessing commissioning data between many team members, and frequently need to change pipeline parameters, override reference files, etc. This will be an extraordinarily common use case during commissioning. It would be great to download the uncal data from MAST once, with some sensible organization so that individual files can be inspected easily, and save them to a common location on central store so that all team members can work from the same raw data. The pipeline can pull inputs from the central store root directory and send outputs to a different directory in which I'm testing new reference file performance. I want to easily modify 15+ different pipeline parameters (skip this step, override this file, save results here, tweak this parameter there) and have a record of how I modified things.
B) Science user with strong opinions about how to defringe MRS data designs a program to get reference star observations at every dither location for their science target. They will download data from two different targets, run both through Spec2 with the fringing step turned off, and use their own tools to do something with the science and reference observations. For ease of organization they group science/reference data at each position into position-based directory structures so that they can find the files easily. When done, they want to run the Spec3 pipeline on the resulting changed files.
jdavies-st
Issue JP-739 was created by Alicia Canipe:
The May 21st DMS Associations meeting discussed some complications with the way data products are bundled in the MAST/Portal results table, and in downloaded zipped file bundles. The most significant issue was the mis-match between the file directory structure of the un-zipped downloads vs. the flat file organization currently expected by the CAL pipeline if users want to re-process data off-line. There are a few other issues too. See the [Association Meeting Notes|https://innerspace.stsci.edu/display/jwstdms/2019-05-21+Associations+Meeting+Notes] for details.
They would like for INS folks to offer their thoughts on this issue. It would be highly valuable to hear the “top 5” use cases for the circumstances where users will want to re-process JWST on their home machines. Some cases are easy to anticipate, such as spectral source extraction from MOS or slitless spectrograms, or subtracting PSFs from coronographic images. But knowing the most common cases could inform the choice of how best to package data.