update backup policy for bowery:/data/cgsb

[agarwal]

As of now, we have to tell the HPC team each sub-directory under bowery:/data/cgsb that we would like to have backed up. Thus, if we add a new sub-directory, we have to remember to tell the HPC team to add that to the backup script.

This is unsatisfactory because:

• We might add a sub-directory and forget.
• It is hard to get the HPC team to inform us of what directories are being backed up, so hard to validate what is being done.
• So far all sub-directories we've ever crated needed to be backed backed up. Thus, there is no reason for the extra complexity.

[agarwal]

From: Karl Ward kw1213@nyu.edu Date: Fri, Jul 26, 2013 at 12:46 PM Subject: Backup policy for /data/cgsb To: Lior Atar la466@nyu.edu, Sreedhar Manchu hpc@nyu.edu, Ashish Agarwal ashish.agarwal@nyu.edu

Back when we first started backing up /data/cgsb, we decided to only backup certain directories in /data/cgsb. Perhaps that decision should be reconsidered now. Ashish, take it away.

Karl Ward

[agarwal]

From: Lior Atar la466@nyu.edu Date: Fri, Jul 26, 2013 at 2:10 PM Subject: Re: Backup policy for /data/cgsb To: Karl Ward kw1213@nyu.edu Cc: Sreedhar Manchu hpc@nyu.edu, Ashish Agarwal ashish.agarwal@nyu.edu

Hello all,

Due to the nature of TSM (tivoli) configuration, we need explicitly list each directory to include. This is done for two reasons,

1. Otherwise we would have to then exclude every other directory on lustre ( major pain)
2. That way each directory under /data/cgsb then looks like another FS to tsm and then allows multi-threaded scans of those directories ( resulting in faster backups and queries on TSM end)

-lior

[agarwal]

From: Ashish Agarwal ashish.agarwal@nyu.edu Date: Fri, Jul 26, 2013 at 2:34 PM Subject: Re: Backup policy for /data/cgsb To: Lior Atar la466@nyu.edu Cc: Karl Ward kw1213@nyu.edu, Sreedhar Manchu hpc@nyu.edu

On Fri, Jul 26, 2013 at 2:10 PM, Lior Atar la466@nyu.edu wrote:

1. Otherwise we would have to then exclude every other directory on lustre ( major pain)

I don't get this. Right now you can say "backup /scratch/cgsb/aa144, /scratch/cgsb/bonneau, etc." It should be reducible and simpler to say "backup /scratch/cgsb". In both cases you are specifying one or more sub-directories of /scratch to include.

1. That way each directory under /data/cgsb then looks like another FS to tsm and then allows multi-threaded scans of those directories ( resulting in faster backups and queries on TSM end)

This will be resolved once we tar up the gencore raw directories, but that's held up until I get an answer to my previous email below.

-Ashish

[agarwal]

From: Ashish Agarwal ashish.agarwal@nyu.edu Date: Mon, Aug 26, 2013 at 6:05 PM Subject: Re: Backup policy for /data/cgsb To: Lior Atar la466@nyu.edu Cc: Karl Ward kw1213@nyu.edu, Sreedhar Manchu hpc@nyu.edu, Sebastien Mondet sebastien.mondet@nyu.edu

Hi. I'd like to revisit the request below now that the file count has been drastically reduced. Can you please institute a full backup of all of /data/cgsb. This would be much better and give us confidence we aren't accidentally omitting directories as they get added.