search

TrinityCTAT / ctat-mutations

Mutation detection using GATK4 best practices and latest RNA editing filters resources. Works with both Hg38 and Hg19
https://github.com/TrinityCTAT/ctat-mutations
Other
71 stars 19 forks source link

pipeline extremely slow: stuck at "Annotate PASS Reads" #112

Closed gianfilippo closed 1 year ago

gianfilippo commented 1 year ago

Hi,

I am running the pipeline on my data and the jobs ends after 48 hours because of time limit. I set it to 48hrs, as it seems very reasonable for the task. I am running on a single node, setting 20 threads. Job scheduler is SLURM and I use singularity:

singularity exec -B pwd:/data -B ${genPATH}:/ctat_genome_lib_dir:ro $imPATH/ctat_mutations.v3.2.1.simg \ $runPATH/ctat-mutations-CTAT-Mutations-v3.2.0/ctat_mutations --left $fastqPATH/$fastq1 --right $fastqPATH/$fastq2 \ --sample_id $sampDIR -O $runPATH/$sampDIR --genome_lib_dir $genPATH --cpu 20

It seems that it gets stuck at "Annotate PASS Reads", where it reaches 99% and gets stuck.

I am rerunning this last step as individual command to see what happens.

Is there anything I can do to solve the problem ?

Also, if I restart, will it start from scratch ? is there a resume option ?

Thanks

brianjohnhaas commented 1 year ago

hi,

The annotation step can take a while, but it sounds like the multithreading might have gotten hung up. Try launching again. It should resume where it left off.

best,

~b

On Thu, Oct 13, 2022 at 1:07 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I am running the pipeline on my data and the jobs ends after 48 hours because of time limit. I set it to 48hrs, as it seems very reasonable for the task. I am running on a single node, setting 20 threads. Job scheduler is SLURM and I use singularity:

singularity exec -B pwd:/data -B ${genPATH}:/ctat_genome_lib_dir:ro $imPATH/ctat_mutations.v3.2.1.simg $runPATH/ctat-mutations-CTAT-Mutations-v3.2.0/ctat_mutations --left $fastqPATH/$fastq1 --right $fastqPATH/$fastq2 --sample_id $sampDIR -O $runPATH/$sampDIR --genome_lib_dir $genPATH --cpu 20

It seems that it gets stuck at "Annotate PASS Reads", where it reaches 99% and gets stuck.

I am rerunning this last step as individual command to see what happens.

Is there anything I can do to solve the problem ?

Also, if I restart, will it start from scratch ? is there a resume option ?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX77H574YLL3TMAHLZTWDA6W3ANCNFSM6AAAAAAREPN27Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

thanks. I restarted the analysis on a different node with much more memory, and it took only a few hours to get to the end. So, memory might have been the issue.

I do get another error, and I hope you can help me with it (see below). Thanks

The first occurrence of failure is here [2022-10-14 03:26:22,35] [info] WorkflowManagerActor Workflow fe31d339-9bd0-4a53-b7f8-358c085120fa failed (during ExecutingWorkflowState): Job ctat_mutations.VariantFiltration:NA:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details. Check the content of stderr for potential additional information: $PATH/cromwell-executions/ctat_mutations/fe31d339-9bd0-4a53-b7f8-358c085120fa/call-VariantFiltration/execution/stderr.

The rest is as follow: [First 3000 bytes]:+ boosting_method=XGBoost

gianfilippo commented 1 year ago

looking at the cromwell-executions/ctat_mutations/fe31d339-9bd0-4a53-b7f8-358c085120fa/call-VariantFiltration/execution/stderr file, it seems that then error is caused by a plot statement (see below). Is there a way around it ? Thanks

23:25:51 : INFO : Plotting Feature Importance Traceback (most recent call last): File "/scripts/ctat-mutations-CTAT-Mutations-v3.2.0/src/VariantBoosting/Apply_ML.py", line 735, in main() File "/scripts/ctat-mutations-CTAT-Mutations-v3.2.0/src/VariantBoosting/Apply_ML.py", line 710, in main feature_importance(boost_obj, args) File "/scripts/ctat-mutations-CTAT-Mutations-v3.2.0/src/VariantBoosting/Apply_ML.py", line 547, in feature_importance plt.figure(figsize=[10,10]) File "/$HOME/.local/lib/python3.7/site-packages/matplotlib/pyplot.py", line 546, in figure **kwargs) File "/$HOME/.local/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 3358, in new_figure_manager return cls.new_figure_manager_given_figure(num, fig) File "/$HOME/.local/lib/python3.7/site-packages/matplotlib/backends/_backend_tk.py", line 888, in new_figure_manager_given_figure window = tk.Tk(className="matplotlib") File "/opt/conda/lib/python3.7/tkinter/init.py", line 2023, in init self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use) _tkinter.TclError: couldn't connect to display "localhost:24.0"

gianfilippo commented 1 year ago

Final update. I modified a couple of lines in /scripts/ctat-mutations-CTAT-Mutations-v3.2.0/src/VariantBoosting/Apply_ML.py

right before the line import matplotlib.pyplot as plt

I added import matplotlib matplotlib.use('pdf')

I got this suggestion here https://stackoverflow.com/questions/19518352/tkinter-tclerror-couldnt-connect-to-display-localhost18-0

After resuming the job, the pipeline finished without errors!!

Thanks

brianjohnhaas commented 1 year ago

Interesting! I was going to say that it might have failed during the boosting step due to not having enough variants to train and boost on, but it's curious that it turned out to be a plotting issue. I'll look further into this and update the code as needed.

thanks!

On Fri, Oct 14, 2022 at 12:53 AM Gianfilippo Coppola < @.***> wrote:

Final update. I modified a couple of lines in

/scripts/ctat-mutations-CTAT-Mutations-v3.2.0/src/VariantBoosting/Apply_ML.py

right before the line import matplotlib.pyplot as plt

I added import matplotlib matplotlib.use('pdf')

I got this suggestion here

https://stackoverflow.com/questions/19518352/tkinter-tclerror-couldnt-connect-to-display-localhost18-0

After resuming the job, the pipeline finished without errors!!

Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1278481424, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2IAQGYX64APRDUA3DWDDRLPANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

I think it is both. In this case, I ended up with having a few variants n the CCA002MV.XGBoost-classifier.cancer.vcf file, but then I reran the pipeline with the RF model, and it crushed again, without a clear error I could see. I guess in this case this is because of lack or too small number of variants. I also did a run without model, to see results with hard filters, and the CCA002MV.cancer.vcf shows no variants.

Also, the CCA002MV.annot_pass_reads.vcf is empty in all cases.

What do you make out of it ?

Can you also explain me the difference between CCA002MV.annot_pass_reads.vcf and the .cancer.vcf ?

Also, I am not sure I understand the 'mode', classification vs regression. The default is classification. What so you use regression for ?

Thanks

brianjohnhaas commented 1 year ago

If everything is working correctly, you should be able to set the boosting parameter to 'none' and it should run w/o fatal errors, ideally. You can explore this with the small set of sample reads that we provide with the software for testing.

The annot_pass_reads.vcf shouldn't be empty once that process completes, so that would indicate a problem for sure. This step has a lot of multithreading built in, and while I remember experiencing some issues with early versions of that code, the current version should be far more robust. It does take a while to run, though, as it's fairly computationally intensive with a lot of I/O hitting the bam file across the multiple threads.

The .cancer.vcf is one of the final products of the pipeline, where found variants that have properties of pathogenic variants are extracted from the full set of variants. This is done based on the various variant annotations.

The boosting can be turned off altogether if it causes any trouble here. It tends to give a slight improvement in accuracy based on our benchmarking, but even without it, the results should be quite good. There's two methods to use for boosting, either regression or classifier based methods. The classifiers usually give better results.

hope this helps,

~b

On Fri, Oct 14, 2022 at 9:50 AM Gianfilippo Coppola < @.***> wrote:

Hi,

I think it is both. In this case, I ended up with having a few variants n the CCA002MV.XGBoost-classifier.cancer.vcf file, but then I reran the pipeline with the RF model, and it crushed again, without a clear error I could see. I guess in this case this is because of lack or too small number of variants. I also did a run without model, to see results with hard filters, and the CCA002MV.cancer.vcf shows no variants.

Also, the CCA002MV.annot_pass_reads.vcf is empty in all cases.

What do you make out of it ?

Can you also explain me the difference between CCA002MV.annot_pass_reads.vcf and the .cancer.vcf ?

Also, I am not sure I understand the 'mode', classification vs regression. The default is classification. What so you use regression for ?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1279037917, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXYKT3WJHQ6X5AT7XALWDFQKPANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

thanks for the reply.

I just tried the test data.

Without boosting it works. I do not get the .annot_pass_reads.vcf, though. Is this expected ?

Thanks

brianjohnhaas commented 1 year ago

That file should show up as one of the intermediates nested within an work directory that's within the output directory. Do you find other files there but not this one? I can dig into mine and see where it ends up exactly.

more soon,

~b

On Fri, Oct 14, 2022 at 11:58 AM Gianfilippo Coppola < @.***> wrote:

Hi,

thanks for the reply.

I just tried the test data.

Without boosting it works. I do not get the .annot_pass_reads.vcf, though. Is this expected ?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1279186208, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX66IXJBJYDTBDZ7ZWTWDF7IVANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

I asked about ti, because I found it in the main dir, not as one of the intermediates nested within, so I assumed it was a final file. I am not sure how that happened, since I do not see in the main dir in the other runs. I just looked into the intermediate dirs and found 3 copies of it, probably related to the 3 different runs on the same sample, with the XGBoost, RF and none setting. Anyway, I assume that file should include, at minimum, as may variants as the final .cancer file, correct ? Thanks

brianjohnhaas commented 1 year ago

I see. We now use cromwell as the execution engine, and so the intermediates show up in the cromwell-executions directory. If you rerun using different boosting options, it should reuse earlier intermediates, but cromwell copies them to new intermediate folders (pretty sure), which is why you might find multiple copies in there under different directory names.

Only the final output files are captured and symlinked in the output directory. 'ls -l' should show you the path to where these symlinks point to among the cromwell output folders.

The .cancer vcf file should be just a subset of the final 'boosted' (or non-boosted if turned off) annotated variants file.

best,

~b

On Fri, Oct 14, 2022 at 12:50 PM Gianfilippo Coppola < @.***> wrote:

I asked about ti, because I found it in the main dir, not as one of the intermediates nested within, so I assumed it was a final file. I am not sure how that happened, since I do not see in the main dir in the other runs. I just looked into the intermediate dirs and found 3 copies of it, probably related to the 3 different runs on the same sample, with the XGBoost, RF and none setting. Anyway, I assume that file should include, at minimum, as may variants as the final .cancer file, correct ? Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1279243275, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2DKG46XBNNEOP5AZLWDGFPBANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

thanks for the info and help! Best

brianjohnhaas commented 1 year ago

Hi,

It turns out there may still be a rare problem that shows up with annotate_PASS_reads.

@Maxwell Brown @.***> has been refactoring the code and is now testing his latest updates. Assuming it all works well, we'll aim to get a new release out asap.

best,

~brian

On Fri, Oct 14, 2022 at 3:01 PM Gianfilippo Coppola < @.***> wrote:

thanks for the info and help! Best

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1279356903, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXYGP5H6MLPFDHODE23WDGUWZANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Thanks. I look forward to the new release

brianjohnhaas commented 1 year ago

absolutely. @Maxwell Brown @.***> is still running some tests but it's looking good so far. We're hoping to get the new release out asap.

best,

~b

On Thu, Oct 27, 2022 at 1:27 PM Gianfilippo Coppola < @.***> wrote:

Thanks. I look forward to the new release

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1293845632, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX3FSMIT5S6IEMWH4LTWFK3PDANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

brianjohnhaas commented 1 year ago

We're still running some tests, but the latest code on the master branch should better deal with the sticking issue. Once we have our remaining tests complete, we'll work on getting the next release out.

On Thu, Oct 27, 2022 at 2:45 PM Brian Haas @.***> wrote:

absolutely. @Maxwell Brown is still running some tests but it's looking good so far. We're hoping to get the new release out asap.

best,

~b

On Thu, Oct 27, 2022 at 1:27 PM Gianfilippo Coppola @.***> wrote:

Thanks. I look forward to the new release

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas

gianfilippo commented 1 year ago

Hi,

I just noticed the new release. Thanks!

In my previous analysis I noticed that when I use hard filtering, I do not see any INDELS in any of the final .filtered.vcf.gz. This. is not an issue when I use the ML based filtering.

It seems strange to end up with no INDELS with hard filtering. Is it possible that you may be selecting for SNPS in the hard filtering branch ?

Tnanks

brianjohnhaas commented 1 year ago

It does look like it's only filtering on the snps and not returning any indels when the hard-filtering is in place.

I suppose we could just let them flow through as an alternative, unless there's some hard-filtering we can do on indels too. Let me know what you think. Also, let's leave this open and we can address it in the next release.

If you want to do hard filtering and still want the indels, I guess you could pull them from the unfiltered vcf for now and merge them in.

On Thu, Dec 8, 2022 at 1:09 PM Gianfilippo Coppola @.***> wrote:

Hi,

I just noticed the new release. Thanks!

In my previous analysis I noticed that when I use hard filtering, I do not see any INDELS in any of the final .filtered.vcf.gz. This. is not an issue when I use the ML based filtering.

It seems strange to end up with no INDELS with hard filtering. Is it possible that you may be selecting for SNPS in the hard filtering branch ?

Tnanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1343112644, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX3R6R3EWNDWAXFV3GDWMIP7DANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

thanks. I just wanted to make sure I was not missing anything.

If the new release works for me, I will be using the ML filter, otherwise I will take the raw vcf as you suggested, and be looking forward to the next release

Thanks

On Thu, Dec 8, 2022 at 1:55 PM Brian Haas @.***> wrote:

It does look like it's only filtering on the snps and not returning any indels when the hard-filtering is in place.

I suppose we could just let them flow through as an alternative, unless there's some hard-filtering we can do on indels too. Let me know what you think. Also, let's leave this open and we can address it in the next release.

If you want to do hard filtering and still want the indels, I guess you could pull them from the unfiltered vcf for now and merge them in.

On Thu, Dec 8, 2022 at 1:09 PM Gianfilippo Coppola @.***> wrote:

Hi,

I just noticed the new release. Thanks!

In my previous analysis I noticed that when I use hard filtering, I do not see any INDELS in any of the final .filtered.vcf.gz. This. is not an issue when I use the ML based filtering.

It seems strange to end up with no INDELS with hard filtering. Is it possible that you may be selecting for SNPS in the hard filtering branch ?

Tnanks

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1343112644 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABZRKX3R6R3EWNDWAXFV3GDWMIP7DANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1343188540, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPSFVGTUNKKINY6RYECXYDWMIVLNANCNFSM6AAAAAAREPN27Q . You are receiving this because you modified the open/close state.Message ID: @.***>

brianjohnhaas commented 1 year ago

Sound good. We find the xgboost / classifier works best.

Let me know if it gives you any trouble.

best,

~brian

On Thu, Dec 8, 2022 at 2:21 PM Gianfilippo Coppola @.***> wrote:

Hi,

thanks. I just wanted to make sure I was not missing anything.

If the new release works for me, I will be using the ML filter, otherwise I will take the raw vcf as you suggested, and be looking forward to the next release

Thanks

On Thu, Dec 8, 2022 at 1:55 PM Brian Haas @.***> wrote:

It does look like it's only filtering on the snps and not returning any indels when the hard-filtering is in place.

I suppose we could just let them flow through as an alternative, unless there's some hard-filtering we can do on indels too. Let me know what you think. Also, let's leave this open and we can address it in the next release.

If you want to do hard filtering and still want the indels, I guess you could pull them from the unfiltered vcf for now and merge them in.

On Thu, Dec 8, 2022 at 1:09 PM Gianfilippo Coppola @.***> wrote:

Hi,

I just noticed the new release. Thanks!

In my previous analysis I noticed that when I use hard filtering, I do not see any INDELS in any of the final .filtered.vcf.gz. This. is not an issue when I use the ML based filtering.

It seems strange to end up with no INDELS with hard filtering. Is it possible that you may be selecting for SNPS in the hard filtering branch ?

Tnanks

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1343112644

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABZRKX3R6R3EWNDWAXFV3GDWMIP7DANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1343188540 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACPSFVGTUNKKINY6RYECXYDWMIVLNANCNFSM6AAAAAAREPN27Q

. You are receiving this because you modified the open/close state.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1343237233, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX4K45OKBIRTNR3Q6HDWMIYLVANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

I think I am still having problems. All my runs get stuck at the "Annotate BLAT ED" step (see below an example). Eventually everything gets shut down because of time limit or preemption

What do you think ?

Thanks

/scripts/ctat-mutations-CTAT-Mutations-v3.3.0/src/annotate_ED.py \ --input_vcf /CTAT/HCC004T/cromwell-executions/ctat_mutations/a362788d-a9fa-4bb7-88ab-670f9f7de9bb/call-AnnotateVariants/annotate_variants_wf/aee41068-a2a1-43a6-86e5-91b4af65d180/call-annotate_blat_ED/inputs/22029523/HCC004T.splice_distance.vcf.gz \ --output_vcf HCC004T.blat_ED.vcf \ --reference /CTAT/HCC004T/cromwell-executions/ctat_mutations/a362788d-a9fa-4bb7-88ab-670f9f7de9bb/call-AnnotateVariants/annotate_variants_wf/aee41068-a2a1-43a6-86e5-91b4af65d180/call-annotate_blat_ED/inputs/2101008049/ref_genome.fa \ --temp_dir $TMPDIR \ --threads 20

bgzip -c HCC004T.blat_ED.vcf > HCC004T.blat_ED.vcf.gz
tabix HCC004T.blat_ED.vcf.gz

[2022-12-12 19:53:35,43] [info] BackgroundConfigAsyncJobExecutionActor [aee41068annotate_variants_wf.annotate_blat_ED:NA:1]: executing: /usr/bin/env bash /CTAT/HCC004T/cromwell-executions/ctat_mutations/a362788d-a9fa-4bb7-88ab-670f9f7de9bb/call-AnnotateVariants/annotate_variants_wf/aee41068-a2a1-43a6-86e5-91b4af65d180/call-annotate_blat_ED/execution/script [2022-12-12 19:53:39,65] [info] BackgroundConfigAsyncJobExecutionActor [aee41068annotate_variants_wf.annotate_blat_ED:NA:1]: job id: 223001 [2022-12-12 19:53:39,65] [info] BackgroundConfigAsyncJobExecutionActor [aee41068annotate_variants_wf.annotate_blat_ED:NA:1]: Status change from - to WaitingForReturnCode slurmstepd: error: JOB 19480554 ON p03r03u38n02 CANCELLED AT 2022-12-12T15:44:35 DUE TO PREEMPTION

brianjohnhaas commented 1 year ago

This step can take a while. I'll take a look and see if there's a way it can be sped up.

Do you have a sample you'd like to share with me privately to examine the runtime performance with?

You could make privately available to bhaas at broadinstitute dot org

best,

~brian

On Mon, Dec 12, 2022 at 3:55 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I think I am still having problems. All my runs get stuck at the "Annotate BLAT ED" step (see below an example). Eventually everything gets shut down because of time limit or preemption

What do you think ?

Thanks

/scripts/ctat-mutations-CTAT-Mutations-v3.3.0/src/annotate_ED.py --input_vcf /CTAT/HCC004T/cromwell-executions/ctat_mutations/a362788d-a9fa-4bb7-88ab-670f9f7de9bb/call-AnnotateVariants/annotate_variants_wf/aee41068-a2a1-43a6-86e5-91b4af65d180/call-annotate_blat_ED/inputs/22029523/HCC004T.splice_distance.vcf.gz

--output_vcf HCC004T.blat_ED.vcf --reference /CTAT/HCC004T/cromwell-executions/ctat_mutations/a362788d-a9fa-4bb7-88ab-670f9f7de9bb/call-AnnotateVariants/annotate_variants_wf/aee41068-a2a1-43a6-86e5-91b4af65d180/call-annotate_blat_ED/inputs/2101008049/ref_genome.fa

--temp_dir $TMPDIR --threads 20

bgzip -c HCC004T.blat_ED.vcf > HCC004T.blat_ED.vcf.gz tabix HCC004T.blat_ED.vcf.gz

[2022-12-12 19:53:35,43] [info] BackgroundConfigAsyncJobExecutionActor

/usr/bin/env bash /CTAT/HCC004T/cromwell-executions/ctat_mutations/a362788d-a9fa-4bb7-88ab-670f9f7de9bb/call-AnnotateVariants/annotate_variants_wf/aee41068-a2a1-43a6-86e5-91b4af65d180/call-annotate_blat_ED/execution/script [2022-12-12 19:53:39,65] [info] BackgroundConfigAsyncJobExecutionActor aee41068annotate_variants_wf.annotate_blat_ED:NA:1: job id: 223001 [2022-12-12 19:53:39,65] [info] BackgroundConfigAsyncJobExecutionActor aee41068annotate_variants_wf.annotate_blat_ED:NA:1: Status change from - to WaitingForReturnCode slurmstepd: error: JOB 19480554 ON p03r03u38n02 CANCELLED AT 2022-12-12T15:44:35 DUE TO PREEMPTION

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1347314830, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX57N47R3D5YGC2PTTTWM6GNRANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

brianjohnhaas commented 1 year ago

Hi,

Please try this dev singularity image: https://data.broadinstitute.org/Trinity/CTAT_SINGULARITY/CTAT_MUTATIONS/dev/

It should solve the problem with the extra long running blatED step and high mem usage there.

best,

~brian

gianfilippo commented 1 year ago

Hi,

I hope you had a happy Holiday.

I tried the new image, both continuing the previous run and rerunning the same sample from scratch. The result is still the same. It gets stuck at the "Annotate BLAT ED" step.

Is there anything I can try to get the pipeline to complete ?

Thanks

brianjohnhaas commented 1 year ago

Hi,

That is very curious. Is it a different sample from the one I tested? I'm wondering where it's getting stuck in that step. The blat search itself takes a while. The part I rectified earlier had to do with what was downstream from the blat search.

Do you know if it's timing out, or hitting a RAM limit?

If you run 'ls -ltr' in the cromwell execution directory for that step where it's failing, that could add some insights.

best,

~b

On Mon, Dec 26, 2022 at 2:14 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I hope you had a happy Holiday.

I tried the new image, both continuing the previous run and rerunning the same sample from scratch. The result is still the same. It gets stuck at the "Annotate BLAT ED" step.

Is there anything I can try to get the pipeline to complete ?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1365376494, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXY3PAOEBLNIQ75AFJLWPHVCJANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

thanks for the prompt reply!!

This is the CCA005P sample I believe I sent you.

My jobs end mostly because of the time limit. I allocate nodes with 118G of ram. In the past, probably I had some failure due to memory limits, perhaps. I can see that I have some rung on larger memory nodes

The execution directory for the blat step is below, along with the output from stdout and stderr files. At the very end is the end of the log file.

It looks like it got stuck on Dec 23. The job has been running now for 3-16:50:45

Thanks

ls -ltr

4623 Dec 22 22:26 script

289 Dec 22 22:26 script.submit

641 Dec 22 22:26 script.background

971 Dec 23 00:59 stdout.background

966 Dec 23 00:59 stdout

1383 Dec 23 01:05 stderr.background

1383 Dec 23 01:05 stderr

stdout

########### Annotate BLAT ED #############

Loaded 3099750718 letters in 194 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11344118 bases in 112318 sequences

Searched 11343512 bases in 112312 sequences

stderr

22:26:32 : INFO :

################################

Annotating VCF: Calculating ED

################################

22:26:32 : INFO : Processing VCF Positions

22:26:51 : INFO : Running samtools faidx

22:27:09 : INFO : Running Blat

00:59:57 : INFO : Processing Output

01:05:25 : INFO : Creating ED features

log file (last few lines)

[2022-12-23 03:26:29,59] [info] BackgroundConfigAsyncJobExecutionActor [ ea7b0c59annotate_variants_wf.annotate_blat_ED:NA:1]: executing: /usr/bin/env bash /CTAT/CCA005P/cromwell-executions/ctat_mutations/1da04eaa-f136-4cf5-9d82-823e352c2e19/call-AnnotateVariants/annotate_variants_wf/ea7b0c59-bfd4-425f-901f-50685d322945/call-annotate_blat_ED/execution/script

[2022-12-23 03:26:32,12] [info] BackgroundConfigAsyncJobExecutionActor [ ea7b0c59annotate_variants_wf.annotate_blat_ED:NA:1]: job id: 9351

[2022-12-23 03:26:32,12] [info] BackgroundConfigAsyncJobExecutionActor [ ea7b0c59annotate_variants_wf.annotate_blat_ED:NA:1]: Status change from - to WaitingForReturnCode

On Mon, Dec 26, 2022 at 2:39 PM Brian Haas @.***> wrote:

Hi,

That is very curious. Is it a different sample from the one I tested? I'm wondering where it's getting stuck in that step. The blat search itself takes a while. The part I rectified earlier had to do with what was downstream from the blat search.

Do you know if it's timing out, or hitting a RAM limit?

If you run 'ls -ltr' in the cromwell execution directory for that step where it's failing, that could add some insights.

best,

~b

On Mon, Dec 26, 2022 at 2:14 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I hope you had a happy Holiday.

I tried the new image, both continuing the previous run and rerunning the same sample from scratch. The result is still the same. It gets stuck at the "Annotate BLAT ED" step.

Is there anything I can try to get the pipeline to complete ?

Thanks

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1365376494 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABZRKXY3PAOEBLNIQ75AFJLWPHVCJANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1365402328, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPSFVGX6UIO2NGPVZLUE6DWPHYA7ANCNFSM6AAAAAAREPN27Q . You are receiving this because you modified the open/close state.Message ID: @.***>

brianjohnhaas commented 1 year ago

hmm....

Can you run

ls -ltr /CTAT/CCA005P/cromwell-executions/ctat_mutations/1da04eaa-f136-4cf5-9d82-823e352c2e19/call-AnnotateVariants/annotate_variants_wf/ea7b0c59-bfd4-425f-901f-50685d322945/call-annotate_blat_ED/

and send me the result?

gianfilippo commented 1 year ago

Hi,

here it is

CTAT/CCA005P/cromwell-executions/ctat_mutations/1da04eaa-f136-4cf5-9d82-823e352c2e19/call-AnnotateVariants/annotate_variants_wf/ea7b0c59-bfd4-425f-901f-50685d322945/call-annotate_blat_ED/inputs:

total 1

drwxr-xr-x 2 XXXXX coppola 2048 Dec 22 22:26 1352490180

drwxr-xr-x 2 XXXXX coppola 2048 Dec 22 22:26 2101008049

CTAT/CCA005P/cromwell-executions/ctat_mutations/1da04eaa-f136-4cf5-9d82-823e352c2e19/call-AnnotateVariants/annotate_variants_wf/ea7b0c59-bfd4-425f-901f-50685d322945/call-annotate_blat_ED/tmp.12f5884b:

total 12263552

prw-r--r-- 1 XXXXX coppola 0 Dec 22 22:26 out.9353

prw-r--r-- 1 XXXXX coppola 0 Dec 22 22:26 err.9353

-rw-r--r-- 1 XXXXX coppola 53636431 Dec 22 22:26 positions.fa

-rw-r--r-- 1 XXXXX coppola 287257248 Dec 22 22:27 faidx_output.fa

-rw-r--r-- 1 XXXXX coppola 12216706404 Dec 23 00:59 blat_output.psl

CTAT/CCA005P/cromwell-executions/ctat_mutations/1da04eaa-f136-4cf5-9d82-823e352c2e19/call-AnnotateVariants/annotate_variants_wf/ea7b0c59-bfd4-425f-901f-50685d322945/call-annotate_blat_ED/execution:

total 131

-rw-r--r-- 1 XXXXX coppola 4623 Dec 22 22:26 script

-rw-r--r-- 1 XXXXX coppola 289 Dec 22 22:26 script.submit

-rw-r--r-- 1 XXXXX coppola 641 Dec 22 22:26 script.background

-rw-r--r-- 1 XXXXX coppola 971 Dec 23 00:59 stdout.background

-rw-r--r-- 1 XXXXX coppola 966 Dec 23 00:59 stdout

-rw-r--r-- 1 XXXXX coppola 1383 Dec 23 01:05 stderr.background

-rw-r--r-- 1 XXXXX coppola 1383 Dec 23 01:05 stderr

On Mon, Dec 26, 2022 at 3:16 PM Brian Haas @.***> wrote:

hmm....

Can you run

ls -ltr

/CTAT/CCA005P/cromwell-executions/ctat_mutations/1da04eaa-f136-4cf5-9d82-823e352c2e19/call-AnnotateVariants/annotate_variants_wf/ea7b0c59-bfd4-425f-901f-50685d322945/call-annotate_blat_ED/

and send me the result?

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1365444460, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPSFVCCDLVCOCN3ZK2FUH3WPH4IVANCNFSM6AAAAAAREPN27Q . You are receiving this because you modified the open/close state.Message ID: @.***>

brianjohnhaas commented 1 year ago

That is very peculiar. So it looks like the blat output finished writing on Dec 23, and then it got hung up afterwards - but there are a couple of issues here that suggest the dev image I made for you maybe wasn't being used to generate those outputs. The latest singularity image I made:

https://data.broadinstitute.org/Trinity/CTAT_SINGULARITY/CTAT_MUTATIONS/dev/

( called ctat_mutations.v3.3.0-postdev2.simg )

should be doing a couple things differently. The temp dir that you find the blat output file in should be called 'tmpdir' now, and once that finishes, it should create an internal checkpoint 'blat_output.psl.ok' in the execution/ directory there.

How sure are you that the above image was being used instead of the current release one? At some point, I'll make a new official release, but for now, this dev version should work for you.

best,

~brian

brianjohnhaas commented 1 year ago

Hi Gianfilippo,

Just checking in. Any chance that latest dev singularity image worked for you here?

best,

~b

On Mon, Dec 26, 2022 at 4:09 PM Brian Haas @.***> wrote:

That is very peculiar. So it looks like the blat output finished writing on Dec 23, and then it got hung up afterwards - but there are a couple of issues here that suggest the dev image I made for you maybe wasn't being used to generate those outputs. The latest singularity image I made:

https://data.broadinstitute.org/Trinity/CTAT_SINGULARITY/CTAT_MUTATIONS/dev/

( called ctat_mutations.v3.3.0-postdev2.simg )

should be doing a couple things differently. The temp dir that you find the blat output file in should be called 'tmpdir' now, and once that finishes, it should create an internal checkpoint 'blat_output.psl.ok' in the execution/ directory there.

How sure are you that the above image was being used instead of the current release one? At some point, I'll make a new official release, but for now, this dev version should work for you.

best,

~brian

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi Brian,

happy New Year!

I am sorry, I forgot to email you about it.

The run using the latest dev singularity image you made for me ended again due to time limit, about 3 days.

It died still at the same "Annotate BLAT ED" step

Best

brianjohnhaas commented 1 year ago

oh no... still sounds like trouble. Are we sure based on the output directory contents that it looks like it's running this latest dev image?

Are you running it via wdl, or via the ctat_mutations driver python script?

best,

~b

On Tue, Jan 10, 2023 at 12:34 PM Gianfilippo Coppola < @.***> wrote:

Hi Brian,

happy New Year!

I am sorry, I forgot to email you about it.

The run using the latest dev singularity image you made for me ended again due to time limit, about 3 days.

It died still at the same "Annotate BLAT ED" step

Best

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377614967, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX3CIVQXBG64IQHBV6DWRWMRVANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

I issue the following command:

singularity exec -B pwd:/data -B ${genPATH}:/ctat_genome_lib_dir:ro $imPATH/$imFILE $ctatPATH/ctat_mutations --bam $bamPATH/$bamFILE \

       --sample_id $sampDIR -O $outPATH/$sampDIR --genome_lib_dir

$genPATH --cpu 20 --boosting_method none

where

imFILE=ctat_mutations.v3.3.0-postdev2.sing

Best

On Tue, Jan 10, 2023 at 1:03 PM Brian Haas @.***> wrote:

oh no... still sounds like trouble. Are we sure based on the output directory contents that it looks like it's running this latest dev image?

Are you running it via wdl, or via the ctat_mutations driver python script?

best,

~b

On Tue, Jan 10, 2023 at 12:34 PM Gianfilippo Coppola < @.***> wrote:

Hi Brian,

happy New Year!

I am sorry, I forgot to email you about it.

The run using the latest dev singularity image you made for me ended again due to time limit, about 3 days.

It died still at the same "Annotate BLAT ED" step

Best

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377614967 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABZRKX3CIVQXBG64IQHBV6DWRWMRVANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377649165, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPSFVELNC3L7ORNRMLV6MLWRWP5TANCNFSM6AAAAAAREPN27Q . You are receiving this because you modified the open/close state.Message ID: @.***>

brianjohnhaas commented 1 year ago

Can you tell me what $ctatPATH is set to?

On Tue, Jan 10, 2023 at 1:30 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I issue the following command:

singularity exec -B pwd:/data -B ${genPATH}:/ctat_genome_lib_dir:ro $imPATH/$imFILE $ctatPATH/ctat_mutations --bam $bamPATH/$bamFILE \

--sample_id $sampDIR -O $outPATH/$sampDIR --genome_lib_dir $genPATH --cpu 20 --boosting_method none

where

imFILE=ctat_mutations.v3.3.0-postdev2.sing

Best

On Tue, Jan 10, 2023 at 1:03 PM Brian Haas @.***> wrote:

oh no... still sounds like trouble. Are we sure based on the output directory contents that it looks like it's running this latest dev image?

Are you running it via wdl, or via the ctat_mutations driver python script?

best,

~b

On Tue, Jan 10, 2023 at 12:34 PM Gianfilippo Coppola < @.***> wrote:

Hi Brian,

happy New Year!

I am sorry, I forgot to email you about it.

The run using the latest dev singularity image you made for me ended again due to time limit, about 3 days.

It died still at the same "Annotate BLAT ED" step

Best

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377614967

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABZRKX3CIVQXBG64IQHBV6DWRWMRVANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377649165 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACPSFVELNC3L7ORNRMLV6MLWRWP5TANCNFSM6AAAAAAREPN27Q

. You are receiving this because you modified the open/close state.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377680411, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX7JMIKO64ZLQMWNQ2TWRWTGDANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

ctatPATH=$runPATH/ctat-mutations-CTAT-Mutations-v3.3.0

imPATH=$ctatPATH/images

Best

On Tue, Jan 10, 2023 at 1:39 PM Brian Haas @.***> wrote:

Can you tell me what $ctatPATH is set to?

On Tue, Jan 10, 2023 at 1:30 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I issue the following command:

singularity exec -B pwd:/data -B ${genPATH}:/ctat_genome_lib_dir:ro $imPATH/$imFILE $ctatPATH/ctat_mutations --bam $bamPATH/$bamFILE \

--sample_id $sampDIR -O $outPATH/$sampDIR --genome_lib_dir $genPATH --cpu 20 --boosting_method none

where

imFILE=ctat_mutations.v3.3.0-postdev2.sing

Best

On Tue, Jan 10, 2023 at 1:03 PM Brian Haas @.***> wrote:

oh no... still sounds like trouble. Are we sure based on the output directory contents that it looks like it's running this latest dev image?

Are you running it via wdl, or via the ctat_mutations driver python script?

best,

~b

On Tue, Jan 10, 2023 at 12:34 PM Gianfilippo Coppola < @.***> wrote:

Hi Brian,

happy New Year!

I am sorry, I forgot to email you about it.

The run using the latest dev singularity image you made for me ended again due to time limit, about 3 days.

It died still at the same "Annotate BLAT ED" step

Best

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377614967

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABZRKX3CIVQXBG64IQHBV6DWRWMRVANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377649165

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ACPSFVELNC3L7ORNRMLV6MLWRWP5TANCNFSM6AAAAAAREPN27Q

. You are receiving this because you modified the open/close state.Message ID: @.***>

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377680411 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABZRKX7JMIKO64ZLQMWNQ2TWRWTGDANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377690487, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPSFVFLG6HIASYC7QD5L5LWRWUGTANCNFSM6AAAAAAREPN27Q . You are receiving this because you modified the open/close state.Message ID: @.***>

brianjohnhaas commented 1 year ago

I think that ctatPATH=$runPATH/ctat-mutations-CTAT-Mutations-v3.3.0

means that you're still running the old code and not the new code that's in the singularity image. Could that be right?

Try setting ctatPATH=/usr/local/src/ctat-mutations

and then it should be using the code that's installed in the image.

best,

~b

On Tue, Jan 10, 2023 at 1:53 PM Gianfilippo Coppola < @.***> wrote:

Hi,

ctatPATH=$runPATH/ctat-mutations-CTAT-Mutations-v3.3.0

imPATH=$ctatPATH/images

Best

On Tue, Jan 10, 2023 at 1:39 PM Brian Haas @.***> wrote:

Can you tell me what $ctatPATH is set to?

On Tue, Jan 10, 2023 at 1:30 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I issue the following command:

singularity exec -B pwd:/data -B ${genPATH}:/ctat_genome_lib_dir:ro $imPATH/$imFILE $ctatPATH/ctat_mutations --bam $bamPATH/$bamFILE \

--sample_id $sampDIR -O $outPATH/$sampDIR --genome_lib_dir $genPATH --cpu 20 --boosting_method none

where

imFILE=ctat_mutations.v3.3.0-postdev2.sing

Best

On Tue, Jan 10, 2023 at 1:03 PM Brian Haas @.***> wrote:

oh no... still sounds like trouble. Are we sure based on the output directory contents that it looks like it's running this latest dev image?

Are you running it via wdl, or via the ctat_mutations driver python script?

best,

~b

On Tue, Jan 10, 2023 at 12:34 PM Gianfilippo Coppola < @.***> wrote:

Hi Brian,

happy New Year!

I am sorry, I forgot to email you about it.

The run using the latest dev singularity image you made for me ended again due to time limit, about 3 days.

It died still at the same "Annotate BLAT ED" step

Best

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377614967

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABZRKX3CIVQXBG64IQHBV6DWRWMRVANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377649165

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ACPSFVELNC3L7ORNRMLV6MLWRWP5TANCNFSM6AAAAAAREPN27Q

. You are receiving this because you modified the open/close state.Message ID: @.***>

— Reply to this email directly, view it on GitHub <

https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377680411

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABZRKX7JMIKO64ZLQMWNQ2TWRWTGDANCNFSM6AAAAAAREPN27Q

. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

— Reply to this email directly, view it on GitHub < https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377690487>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACPSFVFLG6HIASYC7QD5L5LWRWUGTANCNFSM6AAAAAAREPN27Q

. You are receiving this because you modified the open/close state.Message ID: @.***>

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas

gianfilippo commented 1 year ago

Hi,

I think you are right. I had the variable set to the wrong path. I will get back to you

Thanks

brianjohnhaas commented 1 year ago

Great, thanks!

On Tue, Jan 10, 2023 at 2:50 PM Gianfilippo Coppola < @.***> wrote:

Hi,

I think you are right. I had the variable set to the wrong path. I will get back to you

Thanks

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1377768830, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX62LN4EVQMWVXILWG3WRW4RVANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

gianfilippo commented 1 year ago

Hi,

it just completed. It seems ok now

Thanks a lot for your help!

Best

brianjohnhaas commented 1 year ago

Great to hear! I'll work on getting the new release out asap.

best,

~b

On Tue, Jan 10, 2023 at 9:47 PM Gianfilippo Coppola < @.***> wrote:

Hi,

it just completed. It seems ok now

Thanks a lot for your help!

Best

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/112#issuecomment-1378173707, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2JI27XWGZZQN4PW4DWRYNMHANCNFSM6AAAAAAREPN27Q . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas