glennhickey
commented
2 years ago A few things
- You will note that GCSA indexing is not part of the Minigraph-Pangenome pipeline, nor is it mentioned as being supported anywhere in the documentation. So I don't think the title of this issue is fair (claiming the pipeline doesn't work).
- The steps you used to make a GBWT are wrong (you should use the one output by cactus-graphmap-join), but I don't think that'd make a difference for this crash.
- Your only chance to make a GCSA is probably to run an allele frequency filter as described here: https://github.com/ComparativeGenomicsToolkit/cactus/blob/master/doc/pangenome.md#hprc-graph-filtering-by-allele-frequency
Some relevant vg issues: https://github.com/vgteam/vg/issues/3339 https://github.com/vgteam/vg/issues/3411 https://github.com/vgteam/vg/issues/3547 https://github.com/vgteam/vg/issues/3485
Sunhh
I am working on a graph-based pan-genome construction for ten genomes from the same crop whose genome size is less than 500 Mb. When following the Minigraph-Cactus Pangenome Pipeline to construct the variation graph, I am able to generate the .vg format variation graph after finishing "cactus-graphmap-join" command, but I cannot finish the GCSA index from this graph. Initially, I met an error message saying the temporary space is over 2T limitation, which was solved by adding a pruning step with "vg prune". But now, the "vg index" continuously fails after throwing the log information "PathGraph::extend(): File 0: Created 8747360512 order-256 paths" "without coredump", dying with signal 9. May I ask what the problem could be or what I can do to fix this? Thank you and look forward to your help!
The genome I am working on is not very complicated, with 60% of it annotated as repeats which have been changed to lower cases in the input fasta files. The number seems high because it is assembled with HiFi reads. Besides, I can get the GCSA index if only six genomes are included, but I always fail when using seven genomes.
The commands I used to construct the pan-genome are listed below:
cactus-minigraph ./jobstore w-pg/w.pg.txt w-pg/w.gfa.gz --realTimeLogging --reference W97 --mapCores 80cactus-graphmap ./jobstore w-pg/w.pg.txt w-pg/w.gfa.gz w-pg/w.paf --realTimeLogging --reference W97 --outputFasta w-pg/w.gfa.fa.gz --mapCores 80cactus-align ./jobstore w-pg/w.pg.txt w-pg/w.paf w-pg/w.hal --pangenome --pafInput --outVG --reference W97 --realTimeLoggingcactus-graphmap-join ./jobstore --vg w-pg/w.vg --outDir ./w-pg --outName w-pg --reference W97 --vcf --giraffe --gfaffix --wlineSep "." --realTimeLoggingvg convert -v w-pg/w-pg.xg > w-pg/w-pg.vgmkdir w-pg/temp/vg index -x w-pg/new-w-pg.xg w-pg/w-pg.vg -t 60 -p --temp-dir w-pg/temp/vg gbwt -x w-pg/new-w-pg.xg -o w-pg/w-pg.gbwt -v w-pg/w-pg.vcf.gz -p --temp-dir w-pg/temp/vg prune -u -g w-pg/w-pg.gbwt -m w-pg/node_mapping w-pg/w-pg.vg -p -M 32 -t 100 > w-pg/w-pg.pruned.vgvg index -g w-pg/w-pg.gcsa -f w-pg/node_mapping w-pg/w-pg.pruned.vg -p --temp-dir w-pg/temp/ -t 60The logging message are below: GCSA::GCSA(): Step 3 (path length 64 -> 128) PathGraph::prune(): 988860002 -> 977822179 paths (794195377 ranges) PathGraph::prune(): 751730745 unique, 0 redundant, 210455666 unsorted, 15635768 nondeterministic paths PathGraph::prune(): 34.2304 GB in 1 file(s) PathGraph::read(): File 0: Read 977822179 order-64 paths PathGraph::extend(): File 0: Created 1214382892 order-128 paths PathGraph::read(): File 0: Read 1214382892 order-128 paths PathGraphBuilder::sort(): File 0: Sorted 1214382892 paths PathGraph::extend(): 977822179 -> 1214382892 paths (6180215938 ranks) PathGraph::extend(): 50.1667 GB in 1 file(s) GCSA::GCSA(): Step 4 (path length 128 -> 256) PathGraph::prune(): 1214382892 -> 1174714728 paths (935955192 ranges) PathGraph::prune(): 869797981 unique, 0 redundant, 266991520 unsorted, 37925227 nondeterministic paths PathGraph::prune(): 47.9335 GB in 1 file(s) PathGraph::read(): File 0: Read 1174714728 order-128 paths PathGraph::extend(): File 0: Created 8747360512 order-256 paths Child died with signal 9, without coredump.