search

Illumina / Isaac3

Aligner for sequencing data
Other
18 stars 2 forks source link

My job scheduler kills any job after 24H00. Is the any way to recover indexing ? #2

Closed lindenb closed 8 years ago

lindenb commented 8 years ago

Hi illumina, I'm trying to index a genome using isaac3.

/ccc/work/cont007/fg0019/lindenbp/packages/isaac/bin/isaac-sort-reference \
    -g  /ccc/work/cont007/fg/fg/biobank/by-taxonid/9606/38/grch38mt_all_chr.fasta \
    --quiet \
    --output-directory ${CCCSCRATCHDIR}/20160726-isaac-index-grch38mt

Unfortunately the job scheduler I'm using (a wrapper around slurm) kills any job after 24H00.

So my job was killed , here is the stderr:

(...)
+ gridShell=
+ QRSH_CMD=
+ make -j 1 -f /ccc/work/cont007/fg0019/lindenbp/packages/isaac/bin/../share/iSAAC-03.16.04.27/makefiles/reference/SortReference.mk -C /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt GENOME_FILE:=/ccc/work/cont007/fg/fg/biobank/by-taxonid/9606/38/grch38mt_all_chr.fasta ANNOTATION_MASK_WIDTH:=0 iSAAC_LOG_LEVEL:=1 NEIGHBORHOOD_DISTANCE:=2 'ANNOTATION_SEED_LENGTHS:=16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80' all
slurmd[airain6097]: error: *** JOB 4039578 CANCELLED AT 2016-07-27T16:37:30 DUE TO TIME LIMIT ***

and stdout:

(....)
/ccc/work/cont007/fg0019/lindenbp/packages/isaac/bin/../share/iSAAC-03.16.04.27/makefiles/reference/../../../../share/iSAAC-03.16.04.27/makefiles/common/../../../../libexec/iSAAC-03.16.04.27/findNeighbors -r /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbor-positions-20.xml \
    --seed-length 20 \
    --neighborhood-distance 2 \
    --mask-width 0 \
    --mask 0 \
    --output-file /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbors-2-20.16bpb.gz.tmp && mv /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbors-2-20.16bpb.gz.tmp /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbors-2-20.16bpb.gz

as this process seems to use make, is there a safe way to re-launch the process from the current state ?

rpetrovski commented 8 years ago

Just repeat the same command. If you did not touch any temp files or change command line it will resume and build missing files.

On 28 Jul 2016 10:39, "Pierre Lindenbaum" notifications@github.com wrote:

Hi illumina, I'm trying to index a genome using isaac3.

/ccc/work/cont007/fg0019/lindenbp/packages/isaac/bin/isaac-sort-reference \ -g /ccc/work/cont007/fg/fg/biobank/by-taxonid/9606/38/grch38mt_all_chr.fasta \ --quiet \ --output-directory ${CCCSCRATCHDIR}/20160726-isaac-index-grch38mt

Unfortunately the job scheduler I'm using (a wrapper around slurm) stops any job after 24H00.

So my job was canceled , here is the stderr:

(...)

  • gridShell=
  • QRSH_CMD=
  • make -j 1 -f /ccc/work/cont007/fg0019/lindenbp/packages/isaac/bin/../share/iSAAC-03.16.04.27/makefiles/reference/SortReference.mk -C /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt GENOME_FILE:=/ccc/work/cont007/fg/fg/biobank/by-taxonid/9606/38/grch38mt_all_chr.fasta ANNOTATION_MASK_WIDTH:=0 iSAAC_LOG_LEVEL:=1 NEIGHBORHOOD_DISTANCE:=2 'ANNOTATION_SEED_LENGTHS:=16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80' all slurmd[airain6097]: error: * JOB 4039578 CANCELLED AT 2016-07-27T16:37:30 DUE TO TIME LIMIT *

and stdout:

(....) /ccc/work/cont007/fg0019/lindenbp/packages/isaac/bin/../share/iSAAC-03.16.04.27/makefiles/reference/../../../../share/iSAAC-03.16.04.27/makefiles/common/../../../../libexec/iSAAC-03.16.04.27/findNeighbors -r /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbor-positions-20.xml \ --seed-length 20 \ --neighborhood-distance 2 \ --mask-width 0 \ --mask 0 \ --output-file /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbors-2-20.16bpb.gz.tmp && mv /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbors-2-20.16bpb.gz.tmp /ccc/scratch/cont007/fg0019/lindenbp/20160726-isaac-index-grch38mt/Temp/neighbors-2-20.16bpb.gz

as this process seems to use make, is there a safe way to re-launch the process from the current state ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Illumina/Isaac3/issues/2, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8scTg8rpqBPCD8Ca_7i-PVROSaYi7jks5qaFzJgaJpZM4JW8ZN .

lindenb commented 8 years ago

thanks !