DC1 dataproduct retention

[TomGlanzman]

DC1 PhoSim generated 5 TB of simulated images DC1 DM pipeline read those images and generated 75 TB of output data

This is a big problem.

The PhoSim data were stored in DESC "project space", which now contains 65 TB of an 81 TB quota. The DM pipeline data were stored in global scratch space. Technically, scratch space is limited to 20 TB per user so we are way over quota (quotas do not seem to be enforced at present). In addition, scratch directories are purged after 12 weeks -- unless the data is accessed, which resets the clock. Purging has not been done up until now, but NERSC has recently warned it was about to start.

Here is a look at the DM pipeline directory, along with a summary of space consumed:

tony_j@cori17:/global/cscratch1/sd/descdm/DC1/DC1-imsim-dithered> du -hs *

Space used	Directory
4.0K	_mapper
28T	calexp
620K	config
21T	deepCoadd
188G	deepCoadd-results
4.9M	deep_assembleCoadd_metadata
125M	deep_makeCoaddTempExp_metadata
838M	eimage
28T	icExp
195G	icSrc
12G	processEimage_metadata
243M	ref_cats
17M	registry.sqlite
3 408K	schema
2.0T	src
2.6G	srcMatch

What to do with these data? One suggestion was to move it to HPSS (tape). But doing so does not look all that attractive. Such a move would:

1. require some detailed analysis of how to package the data to simultaneously satisfy the requirements of HPSS (bundle files into tar-balls of 10s to 100s of GB), and to make sensible restoration packages from the DM stack/analysis perspective
2. require writing a script to implement the above
3. require users to learn how to restore portions of these data when needed
4. more than consume our SRU (Storage Resource Unit) allocation for the year

Another suggestion is to prune the extraneous/unneeded/unwanted files. Can we agree on what to prune and how to do it?

[TomGlanzman]

At today's Twinkles meeting discussion on this topic, the following ideas were put forward:

• Write a scrub script
• Delete the icExp files straight away (no-one needs these) - saves 28TB
• Ask the DC1 science users if we can delete the deep coadd temp exposures - saves ~12TB
• Ask the DC1 science users if we can delete at least some of the calexps

[jchiang87]

Note that we have three datasets that would occupy ~75TB storage: phosim-DC1, imisim-DC1-dithered and imsim-DC1-undithered. imsim undithered will be somewhat smaller than the other two since it has only 150k sensor-visits vs ~190k each for phosim-DC1 and imsim-DC1-dithered.

[cwwalter]

We will talk about this tomorrow but how much space do we have and where can we keep at least the partial outputs for approximately the next year or so?

[jchiang87]

Here is the status of this issue:

• Mustafa is archiving the imsim-dithered, imsim-undithered and phosim Level 2 repos to HPSS. This takes some days to do, so it is still on-going I believe.
• Tony has copied the imsim-dithered repo to the new project space at NERSC:

  /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered

  This should be accessible to anyone in the lsst group.

[cwwalter]

If the HPSS backup is now finished, can we document where it is (i.e. how to retrieve it) and close this issue?