search

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

Long term storage of DC2_PhoSim Run1.2p #341

Closed heather999 closed 5 years ago

heather999 commented 5 years ago

We are rapidly running out of disk space on projecta at NERSC. We need to move more data to HPSS and remove it from /global/projecta/projectdirs/lsst/ The contents of /global/projecta/projectdirs/lsst/production/DC2_PhoSim/Run1.2p

./   CALIB/        desc_dm_drp/  eimages/       raw_files/
../  calib_raw_files/  dpdd/         instance_cat/  ref_cats/

The dpdd needs to remain on projecta. For now, I will be copying CALIB, eimages, calib_raw_files and instance_cat to HPSS and removing the copy on projecta. I understand that under eimages, there are SRS workflow logs associated with running phosim. Do we need to continue retaining those logs on projecta so that the SRS webpages can find them or does it suffice to know we have them available on HPSS if we need them? Pinging @TomGlanzman for his opinion.

TomGlanzman commented 5 years ago

The directory, /global/projecta/projectdirs/lsst/production/DC2_PhoSim/Run1.2p/eimages contains only symlinks so takes up almost no space (or inodes).

For reference, here is the current space consumption in /global/projecta/projectdirs/lsst/production/DC2_PhoSim/Run1.2p

143G    CALIB
2.5T    calib_raw_files
20T desc_dm_drp
851G    dpdd
13K eimages
14K instance_cat
5.6T    raw_files
458M    ref_cats

However, moving over to /global/projecta/projectdirs/lsst/production/DC2, the following are development/test areas for the cancelled DC2 Run 2.0p. Should that project resurface, these directories would be useful. In any event, on the scale of things, they are pretty small.

52G DC2-R2-0p-test-r/
5.6M    DC2-R2-0p-test-y/
1.0K    Run2.0p/
5.6M    DC2-R2-0p/

There are, in this directory, remnants of Run 1.0p and Run 1.1p which can likely be deleted, including:

drwxrws---+  4 descpho lsst 4096 Apr 18  2018 DC2-R1-2p-test-r/
drwxrws---+  3 descpho lsst 4096 Feb 15  2018 DC2-phoSim-2-r/
drwxrws---+  5 descpho lsst 4096 May 14  2018 DC2-phoSim-3-test/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-g/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-i/
drwxrws---+  4 descpho lsst 4096 Feb 19  2018 DC2-phoSim-3_WFD-r/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-u/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-y/
drwxrws---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-z/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_uDDF-g/
drwxrws---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-i/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-r/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-u/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-y/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-z/
drwxrws---+  3 descpho lsst 4096 Mar  6  2018 DC2-phoSim-test/
drwxrws---+  4 descpho lsst 4096 Mar 18  2018 DC2-phoSim-test_WFD-r/

(I do not have size estimates for these directories as the "du" command is taking a very long time to execute.)

TomGlanzman commented 5 years ago

GPFS finally provided a space summary of /global/projecta/projectdirs/lsst/production/DC2.

1.0K    DC2-R1-2p
300G    DC2-R1-2p-WFD-g
851G    DC2-R1-2p-WFD-i
950G    DC2-R1-2p-WFD-r
187G    DC2-R1-2p-WFD-u
978G    DC2-R1-2p-WFD-y
976G    DC2-R1-2p-WFD-z
125M    DC2-R1-2p-test-r
647G    DC2-R1-2p-uDDF-g
630G    DC2-R1-2p-uDDF-i
647G    DC2-R1-2p-uDDF-r
713G    DC2-R1-2p-uDDF-u
627G    DC2-R1-2p-uDDF-y
632G    DC2-R1-2p-uDDF-z
5.6M    DC2-R2-0p
52G DC2-R2-0p-test-r
5.6M    DC2-R2-0p-test-y
93M DC2-phoSim-2-r
3.8M    DC2-phoSim-3
3.8M    DC2-phoSim-3-test
201G    DC2-phoSim-3_WFD-g
809G    DC2-phoSim-3_WFD-i
884G    DC2-phoSim-3_WFD-r
189G    DC2-phoSim-3_WFD-u
529G    DC2-phoSim-3_WFD-y
851G    DC2-phoSim-3_WFD-z
140G    DC2-phoSim-3_uDDF-g
49G DC2-phoSim-3_uDDF-i
142G    DC2-phoSim-3_uDDF-r
334G    DC2-phoSim-3_uDDF-u
508G    DC2-phoSim-3_uDDF-y
566G    DC2-phoSim-3_uDDF-z
2.8M    DC2-phoSim-test
17G DC2-phoSim-test_WFD-r
7.6M    DC2_Repo
1.0K    Run1-2p
134G    Run1.0p
714G    Run1.2p
1.0K    Run2.0p
75M TGtemp
335M    flats
131K    gcr-catalogs-tests
32M obsolete
1.0K    test
14G test-DC2-phoSim-3_WFD-r
1.4G    test-WFD-y
heather999 commented 5 years ago

Ok based on the comments above, I will go ahead and remove (without backing up to tape) these directories - agreed @TomGlanzman? 

drwxrws---+  4 descpho lsst 4096 Apr 18  2018 DC2-R1-2p-test-r/
drwxrws---+  3 descpho lsst 4096 Feb 15  2018 DC2-phoSim-2-r/
drwxrws---+  5 descpho lsst 4096 May 14  2018 DC2-phoSim-3-test/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-g/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-i/
drwxrws---+  4 descpho lsst 4096 Feb 19  2018 DC2-phoSim-3_WFD-r/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-u/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-y/
drwxrws---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_WFD-z/
drwxr-s---+  4 descpho lsst 4096 Feb 20  2018 DC2-phoSim-3_uDDF-g/
drwxrws---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-i/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-r/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-u/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-y/
drwxr-s---+  4 descpho lsst 4096 Feb 21  2018 DC2-phoSim-3_uDDF-z/
drwxrws---+  3 descpho lsst 4096 Mar  6  2018 DC2-phoSim-test/
drwxrws---+  4 descpho lsst 4096 Mar 18  2018 DC2-phoSim-test_WFD-r/
TomGlanzman commented 5 years ago

Yes, @heather999 those directories should be safe to remove.

katrinheitmann commented 5 years ago

@heather999 Can this be closed now? Thanks!

katrinheitmann commented 5 years ago

closing this now ... @heather999 feel free to reopen if needed.