search

cgsb / hitscore

NYU CGSB Genomics Core Facility
1 stars 0 forks source link

transfer OLB output for run 130220_SN911_0123_AD1TC4ACXX #33

Closed agarwal closed 11 years ago

agarwal commented 11 years ago

In #9, we transferred the hiseq raw data directory, but due to #5 we also need to transfer the Bustard directory generated by OLB. @smondet created this output at rabbot:/scratch/sm4431/gencore_olb/D1TC4ACXX/. Thus, need to do:

rsync -av --delete \
  /scratch/sm4431/gencore_olb/D1TC4ACXX/130220_SN911_0123_AD1TC4ACXX/Data/Intensities/Bustard1.9.4_22-04-2013_sm4431 \
  bowery3:/data/cgsb/gencore-raw/HiSeq/130220_SN911_0123_AD1TC4ACXX/Data/Intensities/

Above command put in a PBS script: 192.rabbotcluster.bio.nyu.edu -- FAILED 195.rabbotcluster.bio.nyu.edu -- FAILED 205.rabbotcluster.bio.nyu.edu -- FAILED

agarwal commented 11 years ago

rsync command has failed 3 times now without transferring a single file. Let's try scp instead. Run the following in a PBS script.

scp -pr \
  /scratch/sm4431/gencore_olb/D1TC4ACXX/130220_SN911_0123_AD1TC4ACXX/Data/Intensities/Bustard1.9.4_22-04-2013_sm4431 \
  bowery3:/data/cgsb/gencore-raw/HiSeq/130220_SN911_0123_AD1TC4ACXX/Data/Intensities/

206.rabbotcluster.bio.nyu.edu -- FAILED

agarwal commented 11 years ago

Darn... problem all along was that immutable bit had been set on destination side. Running rsync command again:

207.rabbotcluster.bio.nyu.edu -- DONE

agarwal commented 11 years ago

Now need to secure this data.

bowery-0-1$ screen -S chattr-123 $ sudo /usr/bin/chattr +i /data/cgsb/gencore-raw/HiSeq/130220_SN911_0123_AD1TC4ACXX/Data/Intensities/ $ sudo /usr/bin/chattr -R +i /data/cgsb/gencore-raw/HiSeq/130220_SN911_0123_AD1TC4ACXX/Data/Intensities/Bustard1.9.4_22-04-2013_sm4431/ -- DONE

agarwal commented 11 years ago

Also need to give Sebastien permissions on this additional directory, but just going to redo for the whole directory.

$ qsub -v DIR=130220_SN911_0123_AD1TC4ACXX /data/cgsb/gencore-raw/bin/seb-perm.pbs 1602525.crunch.local -- DONE

agarwal commented 11 years ago

Normally would validate transfer of this directory by running dircmp. However, we have never had a single case where dircmp finds an error. In other words, if rsync exits with 0 then you can really be confident the transfer is correct. Thus, effective now, we will no longer run dircmp.