Not able to run CTAT Mutations

[mariasolruiz]

Hello, I am trying to run CTAT mutations to find variants in my RNA-seq patient samples using singularity in a high performance cluster. I have not been able to run the test or any sample, could you help me find the problem?

The script I used for the test was:

srun singularity exec -e -B /home/jcotignola/Sol/CTAT-Mutations:/home/jcotignola/Sol/CTAT-Mutations -B /tmp:/tmp -B /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro ctat_mutations.v4.1.0.simg /usr/local/src/ctat-mutations/ctat_mutations --left /home/jcotignola/Sol/CTAT-Mutations/testing/reads_1.fastq.gz --right /home/jcotignola/Sol/CTAT-Mutations/testing/reads_2.fastq.gz --sample_id test --boosting_method none --output varcalling.outdir --genome_lib_dir /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir

The error archive says:

INFO: underlay of /etc/localtime required more than 50 (85) bind mounts 16:09:25 : INFO : CMD: java -Djava.io.tmpdir=varcalling.outdir/__tmpdir -Dconfig.file=/usr/local/src/ctat-mutations/WDL/local_provider_config.inc.conf -jar /usr/local/src/ctat-mutations/WDL/cromwell-58.jar run -i /tmp/tmp5s38qcxl.json -m /tmp/tmpvigxzb6q.json /usr/local/src/ctat-mutations/WDL/ctat_mutations.wdl Workflow a3a4b101-b409-4c63-ad65-a8dc4f8c6aee transitioned to state Failed

The out archive is attached. test_Sol-197666-out.txt

I would appreciate any suggestions to help me find the problem!

Best,

Sol

[brianjohnhaas]

Hi - looks like the target files might be on different mount points. Try putting everything needed into the same directory and see how that goes.

On Mon, May 13, 2024 at 1:00 PM mariasolruiz @.***> wrote:

Hello, I am trying to run CTAT mutations to find variants in my RNA-seq patient samples using singularity in a high performance cluster. I have not been able to run the test or any sample, could you help me find the problem?

The script I used for the test was:

srun singularity exec -e -B /home/jcotignola/Sol/CTAT-Mutations:/home/jcotignola/Sol/CTAT-Mutations -B /tmp:/tmp -B /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro ctat_mutations.v4.1.0.simg /usr/local/src/ctat-mutations/ctat_mutations --left /home/jcotignola/Sol/CTAT-Mutations/testing/reads_1.fastq.gz --right /home/jcotignola/Sol/CTAT-Mutations/testing/reads_2.fastq.gz --sample_id test --boosting_method none --output varcalling.outdir --genome_lib_dir /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir

The error archive says:

INFO: underlay of /etc/localtime required more than 50 (85) bind mounts 16:09:25 : INFO : CMD: java -Djava.io.tmpdir=varcalling.outdir/__tmpdir -Dconfig.file=/usr/local/src/ctat-mutations/WDL/local_provider_config.inc.conf -jar /usr/local/src/ctat-mutations/WDL/cromwell-58.jar run -i /tmp/tmp5s38qcxl.json -m /tmp/tmpvigxzb6q.json /usr/local/src/ctat-mutations/WDL/ctat_mutations.wdl Workflow a3a4b101-b409-4c63-ad65-a8dc4f8c6aee transitioned to state Failed

The out archive is attached. test_Sol-197666-out.txt https://github.com/NCIP/ctat-mutations/files/15297866/test_Sol-197666-out.txt

I would appreciate any suggestions to help me find the problem!

Best,

Sol

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/135, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX6PRYEVL4NKZVBMHNDZCDWTTAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4TGMZVGIYTCOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hello, Thank you for the suggestion. I moved the test files to the CTAT-Mutations folder and I obtained the same result. If it relevant for understanding the problem, both in the previous and current test, a varcalling.outdir folder is created.

Thank you,

[brianjohnhaas]

Very peculiar. Try mounting a home directory as a tmp directory - sometimes this can help.

eg.

singularity exec -H /tmp

and then the rest of it

On Mon, May 13, 2024 at 3:29 PM mariasolruiz @.***> wrote:

Hello, Thank you for the suggestion. I moved the test files to the CTAT-Mutations folder and I obtained the same result. If it relevant for understanding the problem, both in the previous and current test, a varcalling.outdir folder is created.

Thank you,

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/135#issuecomment-2108644675, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX3LDMRA55PJHHIMFQTZCEIA3AVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBYGY2DINRXGU . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hello, thank you for the suggestion. I modified the script as follows:

srun singularity exec -H /tmp -e -B /home/jcotignola/Sol/CTAT-Mutations:/home/jcotignola/Sol/CTAT-Mutations -B /tmp:/tmp -B /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro ctat_mutations.v4.1.0.simg /usr/local/src/ctat-mutations/ctat_mutations --left /home/jcotignola/Sol/CTAT-Mutations/reads_1.fastq.gz --right /home/jcotignola/Sol/CTAT-Mutations/reads_2.fastq.gz --sample_id test --boosting_method none --output varcalling.outdir --genome_lib_dir /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir

But I obtained the same result.

[brianjohnhaas]

I wonder if it's our singularity image that's a problem, or if it's singularity in general. Are you able to run another singularity image, like something simple that just prints 'hello world!' or similar?

On Mon, May 13, 2024 at 4:05 PM mariasolruiz @.***> wrote:

Hello, thank you for the suggestion. I modified the script as follows:

srun singularity exec -H /tmp -e -B /home/jcotignola/Sol/CTAT-Mutations:/home/jcotignola/Sol/CTAT-Mutations -B /tmp:/tmp -B /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro ctat_mutations.v4.1.0.simg /usr/local/src/ctat-mutations/ctat_mutations --left /home/jcotignola/Sol/CTAT-Mutations/reads_1.fastq.gz --right /home/jcotignola/Sol/CTAT-Mutations/reads_2.fastq.gz --sample_id test --boosting_method none --output varcalling.outdir --genome_lib_dir /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir

But I obtained the same result.

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/135#issuecomment-2108700380, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX4SRMJ4BV6CDNJOSBTZCEMJLAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBYG4YDAMZYGA . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hello, I don't think that it is a singularity problem because I can run STAR-Fusion using singularity. I repeated the STAR-Fusion yesterday just in case and it worked. I appreciate if you have other suggestions!

Thank you,

[brianjohnhaas]

So, what I thought was the problem before was likely not the problem. Instead, from looking through your earlier log file, I se this:

[2024-05-10 19:11:44,38] [ESC[38;5;220mwarnESC[0m] Localization via hard link has failed: /home/jcotignola/Sol/CTAT-Mutations/varcalling.outdir/cromwell-executions/ctat_mutations/a3a4b101-b409-4c63-ad65-a8dc4f8c6aee/call-SplitIntervals/inputs/-72417937/ref_genome.dict -> /home/jcotignola/Sol/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_genome.dict: Operation not permitted

and so it's not allowing hard links. It's supposed to try first hard links, then soft links, and if that fails, then to just copy the file over to the work area.

If you see an error like the above in the log, can you explore that output directory and see what ended up there? Is it missing altogether, or a broken link of some sort?

For example, in the above, you'd look for the file: /home/jcotignola/Sol/CTAT-Mutations/varcalling.outdir/cromwell-executions/ctat_mutations/a3a4b101-b409-4c63-ad65-a8dc4f8c6aee/call-SplitIntervals/inputs/-72417937/ref_genome.dict

best,

B

On Thu, May 16, 2024 at 8:02 AM mariasolruiz @.***> wrote:

Hello, I don't think that it is a singularity problem because I can run STAR-Fusion using singularity. I repeated the STAR-Fusion yesterday just in case and it worked. I appreciate if you have other suggestions!

Thank you,

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/135#issuecomment-2115057418, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX6DLUAFEFWN2KWLOJDZCSN7BAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJVGA2TONBRHA . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hello, Thank you for your help. In that directory there are three files:

ref_genome.dict ref_genome.fa ref_genome.fa.fai

Thank you,

[brianjohnhaas]

If you try to run 'less' or 'more' on the ref_genome.fa file there, does it show you the sequence data?

On Fri, May 17, 2024 at 9:20 AM mariasolruiz @.***> wrote:

Hello, Thank you for your help. In that directory there are three files:

ref_genome.dict ref_genome.fa ref_genome.fa.fai

Thank you,

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/135#issuecomment-2117591506, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX5IWQQLXDLWHQP424DZCX7Y7AVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGU4TCNJQGY . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Yes, it does

[brianjohnhaas]

OK, sorry, it's not clear to me what's happening here. I test this on ubuntu 20 on linux via a google VM, in case that's helpful.

On Fri, May 17, 2024 at 10:55 AM mariasolruiz @.***> wrote:

Yes, it does

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/135#issuecomment-2117784759, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX25T4G7FER6UHUJVP3ZCYK4VAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXG44DINZVHE . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hello, I will try to run it in a different HPC and will let you know if it works there.

Thank you,

[mariasolruiz]

Hello, I ran the CTAT Mutations test files through singularity in a different HPC and I obtained the expected output files (cvf, html, etc), with no variants in the test files (I don't know if this is ok), but with some variants in my samples, so now I believe it's working!

I still have to check further the results, to see if they make sense.

Thank you,

[brianjohnhaas]

Hi,

That is very peculiar. Can you send me your vcf and html files that were output? You can send to bhaas at broadinstitute dot org

thx,

Brian

On Fri, Jun 7, 2024 at 9:59 AM mariasolruiz @.***> wrote:

Hello, I ran the CTAT Mutations test files through singularity in a different HPC and I obtained the expected output files (cvf, html, etc) but with no variants.

The code I used was:

srun singularity exec -H /tmp -e -B pwd -B /tmp:/tmp -B /grupos/Marce/cancer/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir:/ctat_genome_lib_dir:ro ctat_mutations.v4.1.0.simg /usr/local/src/ctat-mutations/ctat_mutations --left /grupos/Marce/cancer/CTAT-Mutations/reads_1.fastq.gz --right /grupos/Marce/cancer/CTAT-Mutations/reads_2.fastq.gz --sample_id test --boosting_method none --output varcalling.outdir --genome_lib_dir /grupos/Marce/cancer/CTAT-Mutations/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir

Do you have any suggestions, o is there any output I could show you that could help identify the problem?

Thank you,

— Reply to this email directly, view it on GitHub https://github.com/TrinityCTAT/ctat-mutations/issues/135#issuecomment-2154905805, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX4DFOCUJ4EKQJ3IE5DZGG4ENAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJUHEYDKOBQGU . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Thank you, I sent them by email. I confirm that there is some problem because the variants found in my sample are all non-coding.

[brianjohnhaas]

I see. The cancer vcf won’t have results but the haplotypecaller vcf could

I’m away for the next week traveling and can look into this more as needed when I get back

Best

-- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

On Sat, Jun 8, 2024 at 11:13 AM mariasolruiz @.***> wrote:

Thank you, I sent them by email. I confirm that there is some problem because the variants found in my sample are all non-coding.

— Reply to this email directly, view it on GitHub https://github.com/TrinityCTAT/ctat-mutations/issues/135#issuecomment-2156071598, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXZO2CF7SBEJAFSHVULZGMNP7AVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJWGA3TCNJZHA . You are receiving this because you commented.Message ID: @.***>

[mariasolruiz]

Hello Brian,

I have tried several times but I cannot find the problem. I'd appreciate any suggestions to do some tests.

Thank you!

Maria Sol

[brianjohnhaas]

Hi,

Can you try running again in a clean working directory, capture the log from the run and also send me the outputs? I'll take another look. Also, if you're not using the sample reads that come as example data with the software, please give that a try:

https://github.com/TrinityCTAT/ctat-mutations/tree/master/testing

best,

Brian

On Mon, Jun 24, 2024 at 12:35 PM mariasolruiz @.***> wrote:

Hello Brian,

I have tried several times but I cannot find the problem. I'd appreciate any suggestions to do some tests.

Thank you!

Maria Sol

— Reply to this email directly, view it on GitHub https://github.com/TrinityCTAT/ctat-mutations/issues/135#issuecomment-2186976810, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2EYEJS4MWHJGQ4WVTZJBDEVAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWHE3TMOBRGA . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hello Brian, I have been using the sample reads in the provided folder. I ran the test again today and obtained no results, I am attaching a folder with the outputs, scripts and the log from the run.

Thank you,

Maria Sol test_24062024.zip

[brianjohnhaas]

It seems that the STAR aligner thinks there are zero reads to align (you can see that the star log file provided as one of the outputs says zero reads). The bam file has no alignments in it. Could it be that the file you're providing as input ( /grupos/Marce/cancer/CTAT-Mutations/reads_1.fastq.gz) is a symbolic link instead of the actual file? That might explain it.

On Mon, Jun 24, 2024 at 3:32 PM mariasolruiz @.***> wrote:

Hello Brian, I have been using the sample reads in the provided folder. I ran the test again today and obtained no results, I am attaching a folder with the outputs, scripts and the log from the run.

Thank you,

Maria Sol test_24062024.zip https://github.com/user-attachments/files/15960447/test_24062024.zip

— Reply to this email directly, view it on GitHub https://github.com/TrinityCTAT/ctat-mutations/issues/135#issuecomment-2187264131, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2PZUL35KPW35BJFUTZJBX3JAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGI3DIMJTGE . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[mariasolruiz]

Hi! I uploaded the test files again and now it worked perfectly, so thank you for the advice. Now I'm running my samples and I could obtain results for the first sample but not for other samples, and I cannot find the problem. I am suspecting of the processing capacity of the different nodes in the HPC. Is there a list of minimum requirements for the programme? I am attaching the log from the run in case it helps. Thank you for your advice! log final POS037 en nodo3.txt

[brianjohnhaas]

Great to hear that you were able to get it to work. The part where it failed in your log is at the STAR alignment step, which can require around 40GB to 50GB of RAM - which could limit the servers that you're able to use.

[mariasolruiz]

Thank you for your advice, I'm checking on this issue to check if this is the problem. I've found with this error message in the star temporal log file: EXITING because of FATAL ERROR in reads input: quality string length is not equal to sequence length @A00155:184:HKYNYDSXX:2:2468:12183:19930 GGGACATGGATFFFFFF68:12FFFFFFFFFFFFFFFCTTGGFFFFFCGAGGCCAAAGCCAAGG10972:TGGGATATAAGCTTCCTGAGGACAGGGATGTGATCGCCCAAGGFFFCTTCCATGTGCGGTTACGTGFFFFCTGCAATTTAATTCTATGGATTCCTTTCCACAATTCCFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFF,FFFFFF,F,F,FFFF:FFFFF:F GG7GATACAGGTAATTGTAAATGCTTACCTAAAGTACCGATATAAAATTFFFFTGAGGAGCGCCCAAGGAAACTGGFFFFCGTGTAAAACTCACTGGTAATTTGTTFFFFFFTCCCGTCTCACTCTCATAGCCAATGGAAATCAAAATGGAAATCFFFFFCCCATTGCAAGFFFFATGGACTGGTFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF19930 2:N:0:TCCAGCAA+CGGTTCTT SOLUTION: fix your fastq file

I don't know where this error comes from because I have used this fastq files for other analysis and had no problems. Do yo have any suggestions on this?

Thank you,

[brianjohnhaas]

Hmmm. Sounds like the fastq file is corrupt. Some tools are more sensitive to it than others

-- Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

On Wed, Jul 3, 2024 at 2:13 PM mariasolruiz @.***> wrote:

Thank you for your advice, I'm checking on this issue to check if this is the problem. I've found with this error message in the star temporal log file: EXITING because of FATAL ERROR in reads input: quality string length is not equal to sequence length @a00155 https://github.com/a00155:184:HKYNYDSXX:2:2468:12183:19930

GGGACATGGATFFFFFF68:12FFFFFFFFFFFFFFFCTTGGFFFFFCGAGGCCAAAGCCAAGG10972:TGGGATATAAGCTTCCTGAGGACAGGGATGTGATCGCCCAAGGFFFCTTCCATGTGCGGTTACGTGFFFFCTGCAATTTAATTCTATGGATTCCTTTCCACAATTCCFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFF,FFFFFF,F,F,FFFF:FFFFF:F GG7GATACAGGTAATTGTAAATGCTTACCTAAAGTACCGATATAAAATTFFFFTGAGGAGCGCCCAAGGAAACTGGFFFFCGTGTAAAACTCACTGGTAATTTGTTFFFFFFTCCCGTCTCACTCTCATAGCCAATGGAAATCAAAATGGAAATCFFFFFCCCATTGCAAGFFFFATGGACTGGTFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF19930 2:N:0:TCCAGCAA+CGGTTCTT SOLUTION: fix your fastq file

I don't know where this error comes from because I have used this fastq files for other analysis and had no problems. Do yo have any suggestions on this?

Thank you,

— Reply to this email directly, view it on GitHub https://github.com/TrinityCTAT/ctat-mutations/issues/135#issuecomment-2206931143, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX5XRBSGQ4XW4ZUYLYTZKQ5NZAVCNFSM6AAAAABHUSYMD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWHEZTCMJUGM . You are receiving this because you commented.Message ID: @.***>