search

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

Request Run2.1i increased SCRATCH quota and turn off purging @NERSC #356

Closed heather999 closed 4 years ago

heather999 commented 5 years ago

We want to retain access to Run2.1i DM processed data on CSCRATCH at NERSC. It is preferred this data be on SCRATCH at NERSC to allow running batch jobs that utilize the DM Gen2 Butler (there are known locking issues that prevent the use of projecta in batch). Currently we have a quota of 100 TB on /global/cscratch1/sd/desc/. It is nearly full currently but much can be cleared out. In any case, this is likely not enough to handle both Run1.2i and Run2.1i DM processed data, even if we omit the calexps.

As a point of comparison here is a breakdown of the disk space used for coadd-dr1b-v1 which covers 6 months:

du -hs coadd-dr1b-v1
9.1T    coadd-dr1b-v1
du -hs coadd-dr1b-v1-grizy/
8.2T    coadd-dr1b-v1-grizy/
du -hs coadd-dr1b-v1-u     
1.6T    coadd-dr1b-v1-u

Before opening a ticket with NERSC, we need to:

katrinheitmann commented 4 years ago

Since Run 2.1i has been retired, I suppose this issue can be closed? @heather999?

heather999 commented 4 years ago

Yes, this can be closed, in fact there is no longer a need for data to be on CSCRATCH now that we know how to set up select files on CSCRATCH and leave the actual data on CFS.