Mapping paired reads w/ giraffe, no EOF marker, job stalls, exit code 79

[AlphaJulietAlpha]

I am mapping paired end reads to HPRC-GrCH38 (indexed with vg autoindex)

vg autoindex --workflow giraffe -x grch38.xg -g grch38.gcsa -g /path/to/file/grch38/hprc-v1.0-minigraph-grch38.gfa -p GRCH38_pangenome --target-mem 80G -t 20 > grch38_pangenome.girrafe.gbz

I am using wdl/Cromwell to execute job.; quay.io/vgteam/vg:v1.56.0; The job is stalling in the VgGiraffe alignment step

`## task order: task convertCRAMtoFASTQ task VgGiraffe task VgPack task VgCall task VgStats

task VgGiraffe { input { String sampleId File r1Fastq File r2Fastq File min File dist Int threads = 30 Int memoryGb = 200 Int diskSize File giraffe_gbz File minigraph_grch38 String outGamPath = "~{sampleId}.VgGiraffe.gam" }

command {
    set -e -o pipefail
    vg \
    giraffe \
    -t ~{threads} \
    --read-group "ID:1 LB:lib1 SM:~{sampleId} PL:illumina PU:unit1" \
    --sample ~{sampleId} \
    --gbz-name ~{giraffe_gbz} \
    --minimizer-name ~{min}  \
    --dist-name ~{dist} \
    --fastq-in ~{r1Fastq} \
    --fastq-in ~{r2Fastq} \
    --ref-paths ~{minigraph_grch38} \
    --progress \
    --output-format gam \
    --show-work \
    > ~{outGamPath}

    ls -thl
}` 

This step produces a gam file, but the project fails with an exit code 79, so there is no EOF marker. Here are the stats from the gam file:

EOF marker is absent. The input is probably truncated Total alignments: 5118558 Total primary: 5118558 Total secondary: 0 Total aligned: 5079290 Total perfect: 3050515 Total gapless (softclips allowed): 4908914 Total paired: 5118558 Total properly paired: 5067712 Alignment score: mean 152.779, median 161, stdev 20.6774, max 161 (3050515 reads) Mapping quality: mean 54.2928, median 60, stdev 16.4384, max 60 (4427530 reads) Insertions: 272418 bp in 93519 read events Deletions: 303806 bp in 106883 read events Substitutions: 5294946 bp in 5294946 read events Softclips: 12320666 bp in 264475 read events Total time: 1.75599e+06 seconds Speed: 2.91492 reads/second

The docker was successfully pulled, and with these stats on the gam file, I don't understand why the job fails to go to the next task. I did a dry run on the wdl itself and that was successful, so no syntax errors.

[jeizenga]

Have you captured the logging output from one of the tasks that failed? There should be error output that could help identify the issue.

[AlphaJulietAlpha]

Yes I have the log from that run:

2024-05-15 16:05:30,947 INFO - DispatchedConfigAsyncJobExecutionActor [UUID(9963761b)Conversion_Alignment.VgGiraffe:NA:3]: executing: if [ $(sacct -j 41821072 -X -n -o state) = RUNNING ] || [ $(sacct -j 41821072 -X -n -o state) = PENDING ]; then exit 0; else exit 1; fi 2024-05-15 16:05:31,135 INFO - DispatchedConfigAsyncJobExecutionActor [UUID(9963761b)Conversion_Alignment.VgGiraffe:NA:3]: Status change from Running to WaitingForReturnCode 2024-05-15 16:08:46,237 ERROR - DispatchedConfigAsyncJobExecutionActor [UUID(9963761b)Conversion_Alignment.VgGiraffe:NA:3]: Return file not found after 120 seconds, assuming external kill 2024-05-15 16:08:46,244 INFO - DispatchedConfigAsyncJobExecutionActor [UUID(9963761b)Conversion_Alignment.VgGiraffe:NA:3]: Status change from WaitingForReturnCode to Failed 2024-05-15 16:35:43,929 INFO - $d [UUID(9963761b)]: Copying workflow logs

[jeizenga]

I'm not familiar with this WDL runner, but the line with the error suggests that the process was killed externally, possibly for using too many resources.

Do you also have the stderr output from the process itself? There will usually be something on stderr when the process is killed.

[AlphaJulietAlpha]

I do have the stderr output, but because I used the --show-work flag it prints the seeding process. I'll show you the first 100 lines. And the end of the file doesn't give any errors or warnings.

head -n 100 stderr warning:[vg giraffe] Reference path file (--ref-paths) is only used when output format (-o) is SAM, BAM, or CRAM. Preparing Indexes Loading Minimizer Index Loading GBZ Loading Distance Index v2 Paging in Distance Index v2 Initializing MinimizerMapper Loading and initialization: 149.584 seconds Of which Distance Index v2 paging: 81.7662 seconds Mapping reads to "-" (GAM) --watchdog-timeout 10 --match 1 --mismatch 4 --gap-open 6 --gap-extend 1 --full-l-bonus 5 --max-multimaps 1 --hit-cap 10 --hard-hit-cap 500 --score-fraction 0.9 --max-min 500 --num-bp-per-min 1000 --distance-limit 200 --max-extensions 800 --max-alignments 8 --cluster-score 50 --pad-cluster-score 20 --cluster-coverage 0.3 --extension-score 1 --extension-set 20 --rescue-attempts 15 --max-fragment-length 2000 --paired-distance-limit 2 --rescue-subgraph-size 4 --rescue-seed-limit 100 --chaining-cluster-distance 100 --precluster-connection-coverage-threshold 0.3 --min-precluster-connections 10 --max-precluster-connections 50 --max-lookback-bases 100 --min-lookback-items 1 --lookback-item-hard-cap 15 --chain-score-threshold 100 --min-chains 1 --chain-min-score 100 --max-chain-connection 100 --max-tail-length 100 --max-dp-cells 16777216 --show-work --rescue-algorithm dozeu T5: Read E00125:255:HJVCTCCXX:7:2115:10033:10328: ATTTATTGTACGTTTCTTCTGAAGAATTTTTGTGTGGTTTGTTTTTATAATTTCTCTTTCTAATTTTGTTCACAGATTGTATTCCTGATTTTCTTTATTCTTTGACCATATTTAGCTTTTGAGCATATTTAGAACACTTGCTTTAAAGTAT T5: All minimizers: T5: 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000111111111111111111111111111111111111111111111111111 T5: 0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999000000000011111111112222222222333333333344444444445 T5: 0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 T5: ATTTATTGTACGTTTCTTCTGAAGAATTTTTGTGTGGTTTGTTTTTATAATTTCTCTTTCTAATTTTGTTCACAGATTGTATTCCTGATTTTCTTTATTCTTTGACCATATTTAGCTTTTGAGCATATTTAGAACACTTGCTTTAAAGTAT T5: AAATTCTTCAGAAGAAACGTACAATAAAT---------- (#2, 1 hits) T5: -----ACACAAAAATTCTTCAGAAGAAACGTACA------ (#3, 1 hits) T5: ----------ACAAACCACACAAAAATTCTTCAGAAGAA------ (#5, 1 hits) T5: ----------GAAGAATTTTTGTGTGGTTTGTTTTTATA---------- (#0, 1 hits) T5: ----ATTTTTGTGTGGTTTGTTTTTATAATTTC---------- (#1, 1 hits) T5: -----GAAAGAGAAATTATAAAAACAAACCACAC---------- (#8, 1 hits) T5: ---GGTTTGTTTTTATAATTTCTCTTTCTAAT---------- (#4, 1 hits) T5: ----------ATAATTTCTCTTTCTAATTTTGTTCACAG--- (#6, 1 hits) T5: ----------TTTCTCTTTCTAATTTTGTTCACAGATTG---------- (#7, 1 hits) T5: -----GGAATACAATCTGTGAACAAAATTAGAAA-------- (#13, 1 hits) T5: ----------TTGTTCACAGATTGTATTCCTGATTTTCT--- (#9, 1 hits) T5: ----------TCACAGATTGTATTCCTGATTTTCTTTAT (#10, 1 hits) T5: ----------CACAGATTGTATTCCTGATTTTCTTTATT---------- (#11, 1 hits) T5: ----ATTGTATTCCTGATTTTCTTTATTCTTTG---------- (#12, 1 hits) T5: -------AATATGGTCAAAGAATAAAGAAAATCAGG----- (#16, 1 hits) T5: ----------AAGCTAAATATGGTCAAAGAATAAAGAAA---------- (#18, 1 hits) T5: ------ATTCTTTGACCATATTTAGCTTTTGAGCA---------- (#14, 1 hits) T5: ----------ATATTTAGCTTTTGAGCATATTTAGAACA--- (#15, 1 hits) T5: ----------TTAGCTTTTGAGCATATTTAGAACACTTG---------- (#17, 1 hits) T5: ------TTTAAAGCAAGTGTTCTAAATATGCTCAA---- (#19, 1 hits) T5: Found 20 seeds from 20 minimizers, rejected 0 T5: Minimizers in cluster 0 ⣿⣿⡇ T5: Found 1 clusters T5: Cluster 0 T5: Covers 0.97351/best-0.3 of read T5: Scores 144.292/94.2922 T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits T5: 20 seeds in read:20-146 with 1-1 hits

[jeizenga]

The --show-work option is intended for debugging purposes, and it will produce a way excessive amount of logging in an actual run. It's possible that you ran into a time limit or a disk limit from writing the logs. I would recommend trying these samples again without --show-work to see if they run successfully.