how do I know the pipeline has successfully finished?

[tanpuekai]

Hi Jin,

Thanks a lot for your prev help. We finally finished one of our samples.

This is its final stdout output:

[2019-10-27 02:02:30,45] [info] JobExecutionTokenDispenser stopped
[2019-10-27 02:02:30,47] [info] Aborting all running workflows.
[2019-10-27 02:02:30,47] [info] WorkflowLogCopyRouter stopped
[2019-10-27 02:02:30,49] [info] WorkflowStoreActor stopped
[2019-10-27 02:02:30,49] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2019-10-27 02:02:30,49] [info] WorkflowManagerActor All workflows finished
[2019-10-27 02:02:30,49] [info] WorkflowManagerActor stopped
[2019-10-27 02:02:31,24] [info] Connection pools shut down
[2019-10-27 02:02:31,24] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2019-10-27 02:02:31,24] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2019-10-27 02:02:31,24] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2019-10-27 02:02:31,24] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2019-10-27 02:02:31,24] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2019-10-27 02:02:31,24] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2019-10-27 02:02:31,24] [info] SubWorkflowStoreActor stopped
[2019-10-27 02:02:31,26] [info] KvWriteActor Shutting down: 0 queued messages to process
[2019-10-27 02:02:31,26] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2019-10-27 02:02:31,26] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2019-10-27 02:02:31,26] [info] CallCacheWriteActor stopped
[2019-10-27 02:02:31,28] [info] IoProxy stopped
[2019-10-27 02:02:31,32] [info] JobStoreActor stopped
[2019-10-27 02:02:31,33] [info] ServiceRegistryActor stopped
[2019-10-27 02:02:31,38] [info] DockerHashActor stopped
[2019-10-27 02:02:31,42] [info] Database closed
[2019-10-27 02:02:31,42] [info] Stream materializer shut down
[2019-10-27 02:02:31,42] [info] WDL HTTP import resolver closed
[CaperURI] write to local, target: /project1/sample1/atac/7c08c359-0fc8-438e-a8ce-68cb0438ae4e/metadata.json, size: 310921
[Caper] troubleshooting 7c08c359-0fc8-438e-a8ce-68cb0438ae4e ...
This workflow ran successfully. There is nothing to troubleshoot
[Caper] run:  0 7c08c359-0fc8-438e-a8ce-68cb0438ae4e /project1/sample1/atac/7c08c359-0fc8-438e-a8ce-68cb0438ae4e/metadata.json

Is this considered a success?

My output structure looks like this:

sample1/
   |---caper_file_db.input.lobs
   |---caper_file_db.input.log
   |---caper_file_db.input.properties
   |---caper_file_db.input.script
   |---caper_file_db.input.tmp
   |---cromwell-workflow-logs/
   |---input.json
   |---input.local.json
   |---atac/
        |---7c08c359-0fc8-438e-a8ce-68cb0438ae4e/
              |---call-align/
              |---call-align_mito/
              |---call-annot_enrich/
              |---call-bam2ta/
              |---call-bam2ta_no_dedup/
              |---call-call_peak/
              |---call-call_peak_pr1/
              |---call-call_peak_pr2/
              |---call-filter/
              |---call-filter_no_dedup/
              |---call-frac_mito/
              |---call-fraglen_stat_pe/
              |---call-gc_bias/
              |---call-idr_pr/
              |---call-jsd/
              |---call-macs2_signal_track/
              |---call-overlap_pr/
              |---call-qc_report/
              |---call-read_genome_tsv/
              |---call-reproducibility_idr/
              |---call-reproducibility_overlap/
              |---call-spr/
              |---call-tss_enrich/
              |---call-xcor/
              |---metadata.json

Is this the right structure for a successful run of the pipeline?

Thanks

Chan

[leepc12]

Yes, it looks good.

You see the message in the bottom

This workflow ran successfully. There is nothing to troubleshoot

Or you can try to find qc.json (final output of the pipeline).

find call-qc_report/ -name qc.json

If you have used caper server then caper list shows status of each submitted workflow.

[tanpuekai]

thanks!