Proposed directory structure for Run2.2i releases

[heather999]

As discussed in recent DESC DM/DA meetings, we have tentatively agreed to store our released processed data and all associated ancillary data under the shared area. Data transfers for ongoing processing at CC would continue to be stored under /global/cfs/cdirs/lsst/production/DC2_ImSim/ At NERSC, all released butler rerun areas and sim data would be moved from /global/cfs/cdirs/lsst/production/DC2_ImSim/Run2.2i to /global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i The proposed directory structure under shared would look like:

shared
  |_DC2_prod
       |_ Run2.2i
              |_ sim
                    |_ y1-wfd
                    |_ y2-wfd
                    |_ y3-wfd
                    |_ y4-wfd
                    |_ y5-wfd
              |_ desc_dm_drp
                    |_ v19.0.0
                        |_ calibrations
                        |_ CALIB
                        |_ raw
                        |_ ref_cats
                        |_ rerun
                            |_ run2.2i-calexp
                            |_ run2.2i-dr2-v1
                            |_ run2.2i-dr2-v1-grizy
                            |_ run2.2i-dr2-v1-u
                            |_ run2.2i-dr6-v2
                            |_ run2.2i-dr6-v2-grizy
                            |_ run2.2i-dr6-v2-u

Feedback welcome. It would be relatively quick to put this into place - but we do need to make sure we have agreement. I would additionally propose that we leave Run2.2i DR3 behind in the production area given the hole due to missing visits and leave the current Run2.2i DR6-v1 since it is superseded by DR6-v2.

[yymao]

Thanks, @heather999. I don't know about the structures within butler rerun areas to comment on that part, but the over plan sounds good. With this plan, the DC2-prod directory name is slightly misleading, since it's really DC2 releases (and productions are in ../production). I don't think we would want to change DC2-prod at this point, so I'd suggest adding a top-level README to indicate the intent for our future selves.

[heather999]

As part of this reorganization, the sim data has been moved to /global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim from its old location /global/cfs/cdirs/lsst/production/DC2_ImSim/Run2.2i/sim There is a symlink pointing to the new area in the old location. The butler ingest for that data is being updated for inclusion in the shared area, which users will directed to use as part of the DR2 and DR6-v2 releases. The existing ingest associated with the production area in /global/cfs/cdirs/lsst/production/DC2_ImSim/Run2.2i/desc_drp_dm/v19.0.0-v1 should still work with those symlinks in place.

[JoanneBogart]

This all sounds fine to me.

[katrinheitmann]

Jim suggested this is all done.