search

TrinityCTAT / ctat-mutations

Mutation detection using GATK4 best practices and latest RNA editing filters resources. Works with both Hg38 and Hg19
https://github.com/TrinityCTAT/ctat-mutations
Other
71 stars 19 forks source link

ctat-mutations-WorkflowFailedState #104

Open taot2007 opened 2 years ago

taot2007 commented 2 years ago

Hi, I am running the CTAT-mutations (v3.2.0) pipeline, the library version is StarFv1.10. I used the commamd:

$ nohup time ~/src/CTAT-Mutations-v3.2.0/ctat_mutations --cpu 15 \ --genome_lib_dir ~/Share/CTAT_resource_lib/GRCh38_GENECODE_v38/build/ctat_genome_lib_build_dir/ \ --left ~/data/6015_R1_val_1.fq.gz \ --right ~/data/6015_R2_val_2.fq.gz \ --sample_id 6015 \ --output ~/data/CTATmutations/6015/ > ~/data/CTATmutations/ctat_mutations.nohup.out 2>&1 &

It stopped at this point, looks pretty end. Can you help to address this? Thanks so much!

########################## Running CTAT Boosting ########################## 07:49:32 : INFO : Preprocess Data ... 07:49:32 : INFO : Removing RNAediting sites ... 07:49:32 : INFO : Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable. 07:49:32 : INFO : Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 07:49:32 : INFO : NumExpr defaulting to 8 threads. 07:49:32 : INFO : Number of variants after removing RNAediting sites: 679499 07:49:32 : INFO : -examining AC 07:49:32 : INFO : -AC has 2 uniq entries 07:49:32 : INFO : -examining ALT 07:49:32 : INFO : -ALT has 3 uniq entries 07:49:32 : INFO : -examining BaseQRankSum 07:49:32 : INFO : -BaseQRankSum has 11417 uniq entries 07:49:32 : INFO : -examining DJ 07:49:32 : INFO : -DJ has 140547 uniq entries 07:49:32 : INFO : -examining DP 07:49:32 : INFO : -DP has 1202 uniq entries 07:49:32 : INFO : -examining ED 07:49:32 : INFO : -ED has 910 uniq entries 07:49:32 : INFO : -examining Entropy 07:49:32 : INFO : -Entropy has 11 uniq entries 07:49:32 : INFO : -examining ExcessHet 07:49:32 : INFO : -ExcessHet has 1 uniq entries 07:49:32 : INFO : -pruning feature column ExcessHet as theres no complexity 07:49:32 : INFO : -examining FS 07:49:32 : INFO : -FS has 9440 uniq entries 07:49:32 : INFO : -examining Homopolymer 07:49:32 : INFO : -Homopolymer has 2 uniq entries 07:4 [2022-01-09 07:50:14,24] [info] WorkflowManagerActor WorkflowActor-5da239af-2b16-4a5e-913a-f2324734f2a5 is in a terminal state: WorkflowFailedState [2022-01-09 07:50:23,59] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2022-01-09 07:50:27,05] [info] SingleWorkflowRunnerActor writing metadata to /tmp/tmptdvu1uzx.json [2022-01-09 07:50:27,08] [info] Workflow polling stopped [2022-01-09 07:50:27,10] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds [2022-01-09 07:50:27,10] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds [2022-01-09 07:50:27,10] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds [2022-01-09 07:50:27,10] [info] 0 workflows released by cromid-a657c44 [2022-01-09 07:50:27,10] [info] JobExecutionTokenDispenser stopped [2022-01-09 07:50:27,11] [info] Aborting all running workflows. [2022-01-09 07:50:27,11] [info] WorkflowStoreActor stopped [2022-01-09 07:50:27,11] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds [2022-01-09 07:50:27,11] [info] WorkflowLogCopyRouter stopped [2022-01-09 07:50:27,11] [info] WorkflowManagerActor All workflows finished [2022-01-09 07:50:27,11] [info] WorkflowManagerActor stopped [2022-01-09 07:50:27,32] [info] Connection pools shut down [2022-01-09 07:50:27,32] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down JobStoreActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down DockerHashActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down IoProxy - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] SubWorkflowStoreActor stopped [2022-01-09 07:50:27,33] [info] CallCacheWriteActor Shutting down: 0 queued messages to process [2022-01-09 07:50:27,33] [info] JobStoreActor stopped [2022-01-09 07:50:27,33] [info] KvWriteActor Shutting down: 0 queued messages to process [2022-01-09 07:50:27,33] [info] CallCacheWriteActor stopped [2022-01-09 07:50:27,33] [info] WriteMetadataActor Shutting down: 0 queued messages to process [2022-01-09 07:50:27,33] [info] IoProxy stopped [2022-01-09 07:50:27,33] [info] ServiceRegistryActor stopped [2022-01-09 07:50:27,34] [info] DockerHashActor stopped [2022-01-09 07:50:27,36] [info] Database closed [2022-01-09 07:50:27,36] [info] Stream materializer shut down [2022-01-09 07:50:27,37] [info] WDL HTTP import resolver closed Workflow 5da239af-2b16-4a5e-913a-f2324734f2a5 transitioned to state Failed 1408.72user 545.12system 21:31:31elapsed 2%CPU (0avgtext+0avgdata 1979140maxresident)k 166870624inputs+305136outputs (1major+9823645minor)pagefaults 0swaps

brianjohnhaas commented 2 years ago

hi,

Try running with --boosting_method none

and it should finish with the standard results.

I'm not seeing an error message that indicates why the process failed with the boosting set. Usually it'll indicate a stderr file from the workflow that indicates where it failed with the intermediate results and logs, which we'd use to troubleshoot further.

Boosting generally slightly improves the results, but you can do without it if it continues to crash. You can also try using our singularity or docker images that have everything preinstalled and tested.

best,

~brian

On Sun, Jan 9, 2022 at 10:50 AM taot2007 @.***> wrote:

Hi, I am running the CTAT-mutations (v3.2.0) pipeline, the library version is StarFv1.10. I used the commamd:

$ nohup time ~/src/CTAT-Mutations-v3.2.0/ctat_mutations --cpu 15 --genome_lib_dir ~/Share/CTAT_resource_lib/GRCh38_GENECODE_v38/build/ctat_genome_lib_build_dir/

--left ~/data/6015_R1_val_1.fq.gz --right ~/data/6015_R2_val_2.fq.gz --sample_id 6015 --output ~/data/CTATmutations/6015/ > ~/data/CTATmutations/ctat_mutations.nohup.out 2>&1 &

It stopped at this point, looks pretty end. Can you help to address this? Thanks so much!

########################## Running CTAT Boosting ########################## 07:49:32 : INFO : Preprocess Data ... 07:49:32 : INFO : Removing RNAediting sites ... 07:49:32 : INFO : Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable. 07:49:32 : INFO : Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 07:49:32 : INFO : NumExpr defaulting to 8 threads. 07:49:32 : INFO : Number of variants after removing RNAediting sites: 679499 07:49:32 : INFO : -examining AC 07:49:32 : INFO : -AC has 2 uniq entries 07:49:32 : INFO : -examining ALT 07:49:32 : INFO : -ALT has 3 uniq entries 07:49:32 : INFO : -examining BaseQRankSum 07:49:32 : INFO : -BaseQRankSum has 11417 uniq entries 07:49:32 : INFO : -examining DJ 07:49:32 : INFO : -DJ has 140547 uniq entries 07:49:32 : INFO : -examining DP 07:49:32 : INFO : -DP has 1202 uniq entries 07:49:32 : INFO : -examining ED 07:49:32 : INFO : -ED has 910 uniq entries 07:49:32 : INFO : -examining Entropy 07:49:32 : INFO : -Entropy has 11 uniq entries 07:49:32 : INFO : -examining ExcessHet 07:49:32 : INFO : -ExcessHet has 1 uniq entries 07:49:32 : INFO : -pruning feature column ExcessHet as theres no complexity 07:49:32 : INFO : -examining FS 07:49:32 : INFO : -FS has 9440 uniq entries 07:49:32 : INFO : -examining Homopolymer 07:49:32 : INFO : -Homopolymer has 2 uniq entries 07:4 [2022-01-09 07:50:14,24] [info] WorkflowManagerActor WorkflowActor-5da239af-2b16-4a5e-913a-f2324734f2a5 is in a terminal state: WorkflowFailedState [2022-01-09 07:50:23,59] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2022-01-09 07:50:27,05] [info] SingleWorkflowRunnerActor writing metadata to /tmp/tmptdvu1uzx.json [2022-01-09 07:50:27,08] [info] Workflow polling stopped [2022-01-09 07:50:27,10] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds [2022-01-09 07:50:27,10] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds [2022-01-09 07:50:27,10] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds [2022-01-09 07:50:27,10] [info] 0 workflows released by cromid-a657c44 [2022-01-09 07:50:27,10] [info] JobExecutionTokenDispenser stopped [2022-01-09 07:50:27,11] [info] Aborting all running workflows. [2022-01-09 07:50:27,11] [info] WorkflowStoreActor stopped [2022-01-09 07:50:27,11] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds [2022-01-09 07:50:27,11] [info] WorkflowLogCopyRouter stopped [2022-01-09 07:50:27,11] [info] WorkflowManagerActor All workflows finished [2022-01-09 07:50:27,11] [info] WorkflowManagerActor stopped [2022-01-09 07:50:27,32] [info] Connection pools shut down [2022-01-09 07:50:27,32] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down JobStoreActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down DockerHashActor - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] Shutting down IoProxy - Timeout = 1800 seconds [2022-01-09 07:50:27,32] [info] SubWorkflowStoreActor stopped [2022-01-09 07:50:27,33] [info] CallCacheWriteActor Shutting down: 0 queued messages to process [2022-01-09 07:50:27,33] [info] JobStoreActor stopped [2022-01-09 07:50:27,33] [info] KvWriteActor Shutting down: 0 queued messages to process [2022-01-09 07:50:27,33] [info] CallCacheWriteActor stopped [2022-01-09 07:50:27,33] [info] WriteMetadataActor Shutting down: 0 queued messages to process [2022-01-09 07:50:27,33] [info] IoProxy stopped [2022-01-09 07:50:27,33] [info] ServiceRegistryActor stopped [2022-01-09 07:50:27,34] [info] DockerHashActor stopped [2022-01-09 07:50:27,36] [info] Database closed [2022-01-09 07:50:27,36] [info] Stream materializer shut down [2022-01-09 07:50:27,37] [info] WDL HTTP import resolver closed Workflow 5da239af-2b16-4a5e-913a-f2324734f2a5 transitioned to state Failed 1408.72user 545.12system 21:31:31elapsed 2%CPU (0avgtext+0avgdata 1979140maxresident)k 166870624inputs+305136outputs (1major+9823645minor)pagefaults 0swaps

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/104, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXZTCPGCP3BAWMNQAH3UVGVC7ANCNFSM5LR7V3RQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

taot2007 commented 2 years ago

It worked! Thanks so much!

brianjohnhaas commented 2 years ago

Great, thanks for letting me know!

On Mon, Jan 10, 2022 at 10:42 PM taot2007 @.***> wrote:

It worked! Thanks so much!

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/104#issuecomment-1009571581, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXZ5WE37CEOXAWJBE5TUVORMBANCNFSM5LR7V3RQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

taot2007 commented 2 years ago

Hi Brian, I used the same command as above, most of the samples were processed successfully, but a few had an issue as below. How to get this solved? Thanks so much！

Traceback (most recent call last): File "/home/TT/src/CTAT-Mutations-v3.2.0/src/annotate_ED.py", line 165, in main() File "/home/TT/src/CTAT-Mutations-v3.2.0/src/annotate_ED.py", line 118, in main val1 = abs(int(names[i][1]) - psl_df["T start"][i]) ValueError: invalid literal for int() with base 10: ''

brianjohnhaas commented 2 years ago

hi,

would you be able to share one of the data sets with me that's causing this problem? It'll make it possible for me to debug it.

best,

~b

On Mon, Jan 24, 2022 at 10:20 PM taot2007 @.***> wrote:

Hi Brian, I used the same command as above, most of the samples were processed successfully, but a few had an issue as below. How to get this solved? Thanks so much！

Traceback (most recent call last): File "/home/TT/src/CTAT-Mutations-v3.2.0/src/annotate_ED.py", line 165, in main() File "/home/TT/src/CTAT-Mutations-v3.2.0/src/annotate_ED.py", line 118, in main val1 = abs(int(names[i][1]) - psl_df["T start"][i]) ValueError: invalid literal for int() with base 10: ''

— Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/104#issuecomment-1020771345, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX5VJW36JGLVBOFHEITUXYJI3ANCNFSM5LR7V3RQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas