search

TrinityCTAT / ctat-mutations

Mutation detection using GATK4 best practices and latest RNA editing filters resources. Works with both Hg38 and Hg19
https://github.com/TrinityCTAT/ctat-mutations
Other
71 stars 19 forks source link

AWS EC2 + docker and issue of cromwell or STAR #119

Open HyojeenLee opened 1 year ago

HyojeenLee commented 1 year ago

Hello, I am now using AWS EC2 server and docker as well, pull the image (+build dockerfile), and used the command like below.

docker run --rm -v /home/ec2-user/ctat/testing:/data -v /home/ec2-user/ctat/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro trinityctat/ctat_mutations /usr/local/src/ctat-mutations/ctat_mutations --left /home/ec2-user/ctat/test/reads_1.fastq.gz --right /home/ec2-user/ctat/test/reads_2.fastq.gz --sample_id test --output /data/ctat_mutations_outdir --cpu 10 --genome_lib_dir /ctat_genome_lib_dir --boosting_method none

and got an error like below. I captured for the first error which might seems to be related with cromwell or STAR.

[2022-11-30 11:51:11,00] [info] WorkflowManagerActor Workflow 91d240f2-ccf4-4547-afac-ceb05b5a819c failed (during ExecutingWorkflowState): cromwell.engine.workflow.lifecycle.execution.job.preparation.JobPreparationActor$$anonfun$1$$anon$1: Call input and runtime attributes evaluation failed for StarAlign: [Attempted 1 time(s)] - NoSuchFileException: /home/ec2-user/ctat/test/reads_1.fastq.gz

[2022-11-30 11:51:11,00] [info] WorkflowManagerActor WorkflowActor-91d240f2-ccf4-4547-afac-ceb05b5a819c is in a terminal state: WorkflowFailedState [2022-11-30 11:51:14,32] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2022-11-30 11:51:17,31] [info] SingleWorkflowRunnerActor writing metadata to /tmp/tmp9ujlu703.json [2022-11-30 11:51:17,33] [info] Workflow polling stopped

Did I missed something? Doesn't CTAT docker image contains STAR components?

Thank you.

brianjohnhaas commented 1 year ago

Hi,

Since you're mounting like so:

/home/ec2-user/ctat/testing:/data

It's going to be looking for your fastq files under /data instead of /home/ec2-user/ctat/testing, so you need to adjust your input parameters accordingly.

On Wed, Nov 30, 2022 at 7:25 AM HyojeenLee @.***> wrote:

Hello, I am now using AWS EC2 server and docker as well, pull the image (+build dockerfile), and used the command like below.

docker run --rm -v /home/ec2-user/ctat/testing:/data -v /home/ec2-user/ctat/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro trinityctat/ctat_mutations /usr/local/src/ctat-mutations/ctat_mutations --left /home/ec2-user/ctat/test/reads_1.fastq.gz --right /home/ec2-user/ctat/test/reads_2.fastq.gz --sample_id test --output /data/ctat_mutations_outdir --cpu 10 --genome_lib_dir /ctat_genome_lib_dir --boosting_method none

and got an error like below. I captured for the first error which might seems to be related with cromwell or STAR.

[2022-11-30 11:51:11,00] [info] WorkflowManagerActor Workflow 91d240f2-ccf4-4547-afac-ceb05b5a819c failed (during ExecutingWorkflowState): cromwell.engine.workflow.lifecycle.execution.job.preparation.JobPreparationActor$$anonfun$1$$anon$1: Call input and runtime attributes evaluation failed for StarAlign: [Attempted 1 time(s)] - NoSuchFileException: /home/ec2-user/ctat/test/reads_1.fastq.gz

[2022-11-30 11:51:11,00] [info] WorkflowManagerActor WorkflowActor-91d240f2-ccf4-4547-afac-ceb05b5a819c is in a terminal state: WorkflowFailedState [2022-11-30 11:51:14,32] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2022-11-30 11:51:17,31] [info] SingleWorkflowRunnerActor writing metadata to /tmp/tmp9ujlu703.json [2022-11-30 11:51:17,33] [info] Workflow polling stopped

Did I missed something? Doesn't CTAT docker image contains STAR components?

Thank you.

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/119, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX4WKFGO3PNB2MQRZM3WK5BULANCNFSM6AAAAAASPSKNQA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

HyojeenLee commented 1 year ago

Oh, I see. So I tried with new code indicating 'data':

docker run --rm -v /home/ec2-user/ctat/test:/data -v /home/ec2-user/ctat/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro trinityctat/ctat_mutations /usr/local/src/ctat-mutations/ctat_mutations --left data/reads_1.fastq.gz --right data/reads_2.fastq.gz --sample_id test --output /data/ctat_mutations_outdir --cpu 10 --genome_lib_dir /ctat_genome_lib_dir --boosting_method none

But, I got the same error :(

brianjohnhaas commented 1 year ago

needs to be /data rather than just data. :-)

hope it works

~b

On Wed, Nov 30, 2022 at 8:41 PM HJL @.***> wrote:

Oh, I see. So I tried with new code indicating 'data':

docker run --rm -v /home/ec2-user/ctat/test:/data -v /home/ec2-user/ctat/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro trinityctat/ctat_mutations /usr/local/src/ctat-mutations/ctat_mutations --left data/reads_1.fastq.gz --right data/reads_2.fastq.gz --sample_id test --output /data/ctat_mutations_outdir --cpu 10 --genome_lib_dir /ctat_genome_lib_dir --boosting_method none

But, I got the same error :(

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/119#issuecomment-1333028235, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX7GQRJYY3IDNSZNBPTWK765NANCNFSM6AAAAAASPSKNQA . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

HyojeenLee commented 1 year ago

Thank you. Hopefully, the error changed to..:)

[2022-12-01 02:12:59,40] [info] WorkflowManagerActor Workflow c77a0587-206f-4544-a755-ae567d8a20c5 failed (during ExecutingWorkflowState): Job ctat_mutations.StarAlign:NA:1 exited with return code 108 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details. Check the content of stderr for potential additional information: /data/ctat_mutations_outdir/cromwell-executions/ctat_mutations/c77a0587-206f-4544-a755-ae567d8a20c5/call-StarAlign/execution/stderr. [First 3000 bytes]:+ genomeDir=

EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc Possible cause 1: not enough RAM. Check if you have enough RAM 32053270194 bytes Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 32053270194

Dec 01 02:12:58 ...... FATAL ERROR, exiting

brianjohnhaas commented 1 year ago

Looks like it's running now. The problem here has to do with insufficient RAM. For running the STAR alignment step, it typically needs around 40G of RAM.

~b

On Wed, Nov 30, 2022 at 9:15 PM HJL @.***> wrote:

Thank you. Hopefully, the error changed to..:)

[2022-12-01 02:12:59,40] [info] WorkflowManagerActor Workflow c77a0587-206f-4544-a755-ae567d8a20c5 failed (during ExecutingWorkflowState): Job ctat_mutations.StarAlign:NA:1 exited with return code 108 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details. Check the content of stderr for potential additional information: /data/ctat_mutations_outdir/cromwell-executions/ctat_mutations/c77a0587-206f-4544-a755-ae567d8a20c5/call-StarAlign/execution/stderr. [First 3000 bytes]:+ genomeDir=

  • '[' '' == '' ']'
  • genomeDir=/ctat_genome_lib_dir/ref_genome.fa.star.idx
  • '[' -f /ctat_genome_lib_dir/ref_genome.fa.star.idx ']'
  • STAR --genomeDir /ctat_genome_lib_dir/ref_genome.fa.star.idx --runThreadN 10 --readFilesIn /data/ctat_mutations_outdir/cromwell-executions/ctat_mutations/c77a0587-206f-4544-a755-ae567d8a20c5/call-StarAlign/inputs/46481497/reads_1.fastq.gz /data/ctat_mutations_outdir/cromwell-executions/ctat_mutations/c77a0587-206f-4544-a755-ae567d8a20c5/call-StarAlign/inputs/46481497/reads_2.fastq.gz --readFilesCommand 'gunzip -c' --outSAMtype BAM SortedByCoordinate --twopassMode Basic --limitBAMsortRAM 30000000000 --outSAMmapqUnique 60 --outFileNamePrefix test.star.

EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc Possible cause 1: not enough RAM. Check if you have enough RAM 32053270194 bytes Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 32053270194

Dec 01 02:12:58 ...... FATAL ERROR, exiting

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/119#issuecomment-1333060622, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXYO5Q3CZWTETBGD3DLWLAC3PANCNFSM6AAAAAASPSKNQA . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas