Open brianabernathy opened 3 years ago
The error message indicates that the total size of temporary files exceeded the specified limit (default 2 TB). This is a safety feature, as GCSA construction could plausibly use petabytes of disk space before crashing. The limit can be adjusted in vg index
but apparently not in vg autoindex
.
You have a graph based on aligned sequences, but vg autoindex
seems to prune it with the "restore paths" option. That option is only intended for VCF-based graphs, as it will undo any pruning in alignment-based graphs if the aligned sequences are present as paths. As a result, the number of kmers grows far too quickly with k and you will eventually run out of disk space and/or memory.
Manual index construction should work here: https://github.com/vgteam/vg/wiki/Index-Construction. You first convert the GFA to another graph type (HashGraph?) with vg convert
and then follow the "small graph" / "complex graph" / "with many paths" branch.
Thanks for the detailed diagnosis, I was able to manually index the graph. It seems that vg autoindex
could benefit from an option to select sequence-based or vcf-based graph input. A temp disk limit similar to vg index -Z
would be useful too.
Thanks again for your work on this project.
I am trying to index a gfa constructed from 3 closely-related melon sequences (to use with
vg map
) usingvg autoindex
. The gfa is ~2G, not particularly large or complex. I'm running the job on a machine with 10+T free disk and 1.5T mem, no other jobs are running on the machine. relevant autoindex options are...The command eventually fails with the following output written to stderr.
I've used benchmarking tools to check memory usage and have found ~768G is being consumed before failure, despite
--target-mem 1500G
being specified. I noticed the doc states-M, --target-mem MEM target max memory usage (not exact, formatted INT[kMG]) (default: 1/2 of available)
So I tried running the command without the--target-mem
option and with--target-mem 3000G
. All attempts resulted in failure at the same processing step with similar run time and memory usage.Are there other autoindex options or any manual indexing steps you'd recommend trying?