search

cgsb / hitscore

NYU CGSB Genomics Core Facility
1 stars 0 forks source link

NYUAD run D26YCACXX, D2BBLACXX #130

Closed smondet closed 10 years ago

smondet commented 10 years ago
smondet commented 10 years ago 
[sm4431@ADUAE119-LAP archive]$ nohup ./do_md5.sh 130917_SN7001341_0044_AD26YCACXX.tar.gz &
Ok

[sm4431@ADUAE119-LAP archive]$ nohup ./do_md5.sh 130917_SN7001341_0045_BD2BBLACXX.tar.gz &
Ok

In /share/archive:

03cc3c4cf77aae63ef24a29bed317ec3  130917_SN7001341_0044_AD26YCACXX.tar.gz
aad9089343fcc5ec21310b1ba3dbfbfa  130917_SN7001341_0045_BD2BBLACXX.tar.gz

$ qsub -v DIR=130917_SN7001341_0044_AD26YCACXX rsync-tar.pbs
927742.hpc0.abudhabi.nyu.edu

PBS Job Id: 927742.hpc0.abudhabi.nyu.edu
Job Name:   tar-hiseq-rsync
Exec host:  compute-3-5/6
Execution terminated
Exit_status=0
resources_used.cput=01:07:01
resources_used.mem=14004kb
resources_used.vmem=342300kb
resources_used.walltime=07:14:01

 $ qsub -v DIR=130917_SN7001341_0045_BD2BBLACXX rsync-tar.pbs
927743.hpc0.abudhabi.nyu.edu

PBS Job Id: 927743.hpc0.abudhabi.nyu.edu
Job Name:   tar-hiseq-rsync
Exec host:  compute-3-10/6
Execution terminated
Exit_status=0
resources_used.cput=01:04:48
resources_used.mem=14012kb
resources_used.vmem=341408kb
resources_used.walltime=07:43:38
$ qsub -v DIR=130917_SN7001341_0044_AD26YCACXX untar.pbs 
928902.hpc0.abudhabi.nyu.edu

PBS Job Id: 928902.hpc0.abudhabi.nyu.edu
Job Name:   untar_$DIR
Exec host:  compute-3-4/3
Execution terminated
Exit_status=0
resources_used.cput=01:22:01
resources_used.mem=4760kb
resources_used.vmem=345212kb
resources_used.walltime=08:47:03

$ qsub -v DIR=130917_SN7001341_0045_BD2BBLACXX untar.pbs 
928903.hpc0.abudhabi.nyu.edu

PBS Job Id: 928903.hpc0.abudhabi.nyu.edu
Job Name:   untar_$DIR
Exec host:  compute-3-4/10
Execution terminated
Exit_status=0
resources_used.cput=01:29:25
resources_used.mem=4764kb
resources_used.vmem=345212kb
resources_used.walltime=08:54:29

Archival

 $ qsub  -v DIR=130917_SN7001341_0045_BD2BBLACXX bin/cp-to-archive.pbs
931107.hpc0.abudhabi.nyu.edu

PBS Job Id: 931107.hpc0.abudhabi.nyu.edu
Job Name:   archive-tar
Exec host:  compute-3-4/2
Execution terminated
Exit_status=0
resources_used.cput=00:06:52
resources_used.mem=5796kb
resources_used.vmem=330072kb
resources_used.walltime=02:09:25

 $ qsub  -v DIR=130917_SN7001341_0044_AD26YCACXX bin/cp-to-archive.pbs
931109.hpc0.abudhabi.nyu.edu

PBS Job Id: 931109.hpc0.abudhabi.nyu.edu
Job Name:   archive-tar
Exec host:  compute-3-4/3
Execution terminated
Exit_status=0
resources_used.cput=00:05:48
resources_used.mem=5800kb
resources_used.vmem=330072kb
resources_used.walltime=02:08:57

gencore@butinah-0-4:~/bin $ qsub -v DIR=130917_SN7001341_0044_AD26YCACXX tar-md5.pbs 
934132.hpc0.abudhabi.nyu.edu

PBS Job Id: 934132.hpc0.abudhabi.nyu.edu
Job Name:   tar-md5
Exec host:  compute-1-31/0
Execution terminated
Exit_status=0
resources_used.cput=00:07:26
resources_used.mem=3568kb
resources_used.vmem=321560kb
resources_used.walltime=01:15:24

gencore@butinah-0-4:~/bin $ qsub -v DIR=130917_SN7001341_0045_BD2BBLACXX tar-md5.pbs 
934133.hpc0.abudhabi.nyu.edu

PBS Job Id: 934133.hpc0.abudhabi.nyu.edu
Job Name:   tar-md5
Exec host:  compute-1-31/9
Execution terminated
Exit_status=0
resources_used.cput=00:07:30
resources_used.mem=3572kb
resources_used.vmem=321560kb
resources_used.walltime=01:15:41
smondet commented 10 years ago

the checksums match!

gencore@butinah-0-2:/archive/gencore/raw/hiseq-7001341 $ cat 2013-10-07.18\:1*
03cc3c4cf77aae63ef24a29bed317ec3  /archive/gencore/raw/hiseq-7001341/130917_SN7001341_0044_AD26YCACXX.tar.gz
aad9089343fcc5ec21310b1ba3dbfbfa  /archive/gencore/raw/hiseq-7001341/130917_SN7001341_0045_BD2BBLACXX.tar.gz
smondet commented 10 years ago 
gencore@butinah-0-2:/data/cgsb/gencore/bcl_to_fastq/130917_SN7001341_0044_AD26YCACXX $ qsub pbs/script.pbs 
934614.hpc0.abudhabi.nyu.edu

gencore@butinah-0-2:/data/cgsb/gencore/bcl_to_fastq/130917_SN7001341_0045_BD2BBLACXX $ qsub pbs/script.pbs 
934615.hpc0.abudhabi.nyu.edu

both failed

Dear Sebastien Mondet:

Thank you for contacting NYUAD Technology Group. We have created a request regarding High Performance Computing (Abu Dhabi) with reference number #251356. Additional details can be found below.

Description: Install bcl2fastq on BuTinah

....

Reference Number: 251356

You wrote: Hi

The genomics core has been using a new option of the sequencer that requires an additional tool for postprocessing:

It is called 'bcl2fastq' (it's different from the `configureBclToFastq.pl` that we already have as part of Casava).

See here: http://support.illumina.com/sequencing/sequencing_instruments/hiseq_2500/questions.ilmn
(look for 'bcl2fastq' in the "analysis" section)

They say the dependencies are the same as for Casava so it should be easy to install I guess.

Tell me if you need anything or have questions.

thanks

Ref:MSG1165446

→ bcl2fastq got Installed

130917_SN7001341_0044_AD26YCACXX $ qsub pbs/script.pbs
941189.hpc0.abudhabi.nyu.edu

PBS Job Id: 941189.hpc0.abudhabi.nyu.edu
Job Name:   bcl_to_fastq-44
Exec host:  compute-3-14/11+compute-3-14/10+compute-3-14/9+compute-3-14/8+compute-3-14/7+compute-3-14/6+compute-3-14/5+compute-3-14/4+compute-3-14/3+compute-3-14/2+compute-3-14/1+compute-3-14/0
Execution terminated
Exit_status=0
resources_used.cput=11:01:08
resources_used.mem=400888kb
resources_used.vmem=9644988kb
resources_used.walltime=01:48:34

130917_SN7001341_0045_BD2BBLACXX $ qsub pbs/script.pbs 
941190.hpc0.abudhabi.nyu.edu

PBS Job Id: 941190.hpc0.abudhabi.nyu.edu
Job Name:   bcl_to_fastq-45
Exec host:  compute-3-14/11+compute-3-14/10+compute-3-14/9+compute-3-14/8+compute-3-14/7+compute-3-14/6+compute-3-14/5+compute-3-14/4+compute-3-14/3+compute-3-14/2+compute-3-14/1+compute-3-14/0
Execution terminated
Exit_status=2
resources_used.cput=06:46:05
resources_used.mem=404688kb
resources_used.vmem=4748684kb
resources_used.walltime=00:58:05

---- changed the script to do only the make
gencore@butinah-0-4:/scratch/cgsb/gencore/bcl_to_fastq/130917_SN7001341_0045_BD2BBLACXX $ qsub pbs/script.pbs 
942789.hpc0.abudhabi.nyu.edu

PBS Job Id: 942789.hpc0.abudhabi.nyu.edu
Job Name:   bcl_to_fastq-45
Exec host:  compute-1-10/11+compute-1-10/10+compute-1-10/9+compute-1-10/8+compute-1-10/7+compute-1-10/6+compute-1-10/5+compute-1-10/4+compute-1-10/3+compute-1-10/2+compute-1-10/1+compute-1-10/0
Execution terminated
Exit_status=2
resources_used.cput=00:00:28
resources_used.mem=9760kb
resources_used.vmem=328480kb
resources_used.walltime=00:05:13
smondet commented 10 years ago

Fixed the sample sheet

$ qsub pbs/script.pbs 
942826.hpc0.abudhabi.nyu.edu

PBS Job Id: 942826.hpc0.abudhabi.nyu.edu
Job Name:   bcl_to_fastq-45
Exec host:  compute-1-10/11+compute-1-10/10+compute-1-10/9+compute-1-10/8+compute-1-10/7+compute-1-10/6+compute-1-10/5+compute-1-10/4+compute-1-10/3+compute-1-10/2+compute-1-10/1+compute-1-10/0
Execution terminated
Exit_status=0
resources_used.cput=10:48:11
resources_used.mem=480100kb
resources_used.vmem=9728712kb
resources_used.walltime=01:45:28

gencore@butinah-0-4:/scratch/cgsb/gencore/bcl_to_fastq/130917_SN7001341_0045_BD2BBLACXX $ cat SampleSheet.csv 
FCID,Lane,SampleID,SampleRef,Index,Description,Control,Recipe,Operator,SampleProject
D2BBLACXX,1,Fly_H,,TAGGCATG-TAGATCGC,,N,,,Lane1
D2BBLACXX,1,Fly_J,,GGACTCCT-CTCTCTAT,,N,,,Lane1
D2BBLACXX,1,Fly_M,,AGGCAGAA-TATCCTCT,,N,,,Lane1
D2BBLACXX,2,Fly_K,,TAGGCATG-TAGATCGC,,N,,,Lane2
D2BBLACXX,2,Fly_N,,AGGCAGAA-TATCCTCT,,N,,,Lane2
D2BBLACXX,2,Fly_O,,GGACTCCT-AGAGTAGA,,N,,,Lane2
D2BBLACXX,3,Fly_P,,GGACTCCT-AGAGTAGA,,N,,,Lane3
D2BBLACXX,3,Fly_Q,,TCCTGAGC-AGAGTAGA,,N,,,Lane3
D2BBLACXX,3,Fly_R,,AGGCAGAA-TATCCTCT,,N,,,Lane3
D2BBLACXX,4,Phi_X_Control,,Undetermined,,N,,,Lane4
D2BBLACXX,5,Fly_T,,AGGCAGAA-TATCCTCT,,N,,,Lane5
D2BBLACXX,5,Fly_U,,TCCTGAGC-AGAGTAGA,,N,,,Lane5
D2BBLACXX,5,Fly_W,,GGACTCCT-AGAGTAGA,,N,,,Lane5
D2BBLACXX,6,Fly_X,,AGGCAGAA-TATCCTCT,,N,,,Lane6
D2BBLACXX,6,Fly_Y,,TCCTGAGC-AGAGTAGA,,N,,,Lane6
D2BBLACXX,6,Fly_Z,,GGACTCCT-AGAGTAGA,,N,,,Lane6
D2BBLACXX,7,Iranian_5,,TCCTGAGC-TATCCTCT,,N,,,Lane7
D2BBLACXX,7,DP1,,AGGCAGAA-AGAGTAGA,,N,,,Lane7
D2BBLACXX,8,DP14,,TAGGCATG-TAGATCGC,,N,,,Lane8
D2BBLACXX,8,DP15,,GGACTCCT-CTCTCTAT,,N,,,Lane8