search

vgteam / vg

tools for working with genome variation graphs
https://biostars.org/tag/vg/
Other
1.07k stars 191 forks source link

vg giraffe - Signal 6 occurred. VG has crashed #4246

Open jdamas13 opened 3 months ago

jdamas13 commented 3 months ago

1. What were you trying to do? Mapping Illumina WGS data to a cactus-minigraph pangenome. In case it is relevant, the pangenome was constructed with 5 marsupial genomes, which have a few long chromosomes (>512Mb) that often cause crashes because of hard-coded limits for some tools.

2. What did you want to happen? Obtain a sam file with the reads mapped to the pangenome.

3. What actually happened? vg crashed with the error posted below.

4. If you got a line like Stack trace path: /somewhere/on/your/computer/stacktrace.txt, please copy-paste the contents of that file here:

Crash report for vg v1.55.0 "Bernolda"
Stack trace (most recent call last) in thread 487:
#20   Object "", at 0xffffffffffffffff, in 
#19   Object "/usr/local/bin/vg", at 0x2160633, in __clone
#18   Object "/usr/local/bin/vg", at 0x20b9d4a, in start_thread
#17   Object "/usr/local/bin/vg", at 0x205c5dd, in gomp_thread_start
#16   Object "/usr/local/bin/vg", at 0x205ef27, in gomp_team_barrier_wait_end
#15   Object "/usr/local/bin/vg", at 0x205682a, in gomp_barrier_handle_tasks
#14   Object "/usr/local/bin/vg", at 0xeb9f2a, in unsigned long vg::io::paired_for_each_parallel_after_wait<vg::Alignment>(std::function<bool (vg::Alignment&, vg::Alignment&)>, std::function<void (vg::Alignment&, vg::Alignment&)>, std::function<bool ()>, unsigned long) [clone ._omp_fn.1]
#13   Object "/usr/local/bin/vg", at 0xd1bc71, in std::_Function_handler<void (vg::Alignment&, vg::Alignment&), main_giraffe(int, char**)::{lambda()#1}::operator()() const::{lambda(vg::Alignment&, vg::Alignment&)#6}>::_M_invoke(std::_Any_data const&, vg::Alignment&, vg::Alignment&)
#12   Object "/usr/local/bin/vg", at 0x117b146, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&, std::vector<std::pair<vg::Alignment, vg::Alignment>, std::allocator<std::pair<vg::Alignment, vg::Alignment> > >&)
#11   Object "/usr/local/bin/vg", at 0x1179f3c, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)
#10   Object "/usr/local/bin/vg", at 0x116d1ca, in void vg::MinimizerMapper::process_until_threshold_c<double>(unsigned long, std::function<double (unsigned long)> const&, std::function<bool (unsigned long, unsigned long)> const&, double, unsigned long, unsigned long, vg::LazyRNG&, std::function<bool (unsigned long)> const&, std::function<void (unsigned long)> const&, std::function<void (unsigned long)> const&) const [clone .constprop.0]
#9    Object "/usr/local/bin/vg", at 0x117decb, in vg::MinimizerMapper::map_paired(vg::Alignment&, vg::Alignment&)::{lambda(unsigned long)#16}::operator()(unsigned long) const
#8    Object "/usr/local/bin/vg", at 0x117ccda, in vg::MinimizerMapper::attempt_rescue(vg::Alignment const&, vg::Alignment&, vg::VectorView<vg::MinimizerMapper::Minimizer> const&, bool)
#7    Object "/usr/local/bin/vg", at 0xea8a42, in vg::Aligner::align_xdrop(vg::Alignment&, handlegraph::HandleGraph const&, std::vector<handlegraph::handle_t, std::allocator<handlegraph::handle_t> > const&, std::vector<vg::MaximalExactMatch, std::allocator<vg::MaximalExactMatch> > const&, bool, unsigned short) const
#6    Object "/usr/local/bin/vg", at 0xfbf2a1, in vg::DozeuInterface::align(vg::Alignment&, handlegraph::HandleGraph const&, std::vector<handlegraph::handle_t, std::allocator<handlegraph::handle_t> > const&, std::vector<vg::MaximalExactMatch, std::allocator<vg::MaximalExactMatch> > const&, bool, signed char, unsigned short)
#5    Object "/usr/local/bin/vg", at 0xfbcd7f, in vg::DozeuInterface::calculate_max_position(vg::DozeuInterface::OrderedGraph const&, vg::DozeuInterface::graph_pos_s const&, unsigned long, bool, std::vector<dz_forefront_s const*, std::allocator<dz_forefront_s const*> > const&)
#4    Object "/usr/local/bin/vg", at 0x2088545, in __assert_fail
#3    Object "/usr/local/bin/vg", at 0x5e6053, in __assert_fail_base.cold
#2    Object "/usr/local/bin/vg", at 0x5e612b, in abort
#1    Object "/usr/local/bin/vg", at 0x208eb55, in raise
#0    Object "/usr/local/bin/vg", at 0x20bb56c, in __pthread_kill
ERROR: Signal 6 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug.

Other warnings include:

warning[vg::giraffe]: Refusing to perform too-large rescue alignment of 101 bp against 15884 bp ordered subgraph for read A01204:224:HWHTYDRX3:2:2102:32814:27853 which would use more than 1572864 cells and might exhaust Dozeu's allocator; suppressing further warnings.
warning[vg::Watchdog]: Thread 194 has been checked in for 10 seconds processing: A01204:224:HWHTYDRX3:2:2147:4209:17832, A01204:224:HWHTYDRX3:2:2147:4209:17832
warning[vg::Watchdog]: Thread 194 finally checked out after 11 seconds and 0 kb memory growth processing: A01204:224:HWHTYDRX3:2:2147:4209:17832, A01204:224:HWHTYDRX3:2:2147:4209:17832
vg: src/dozeu_interface.cpp:126: vg::DozeuInterface::graph_pos_s vg::DozeuInterface::calculate_max_position(const vg::DozeuInterface::OrderedGraph&, const vg::DozeuInterface::graph_pos_s&, size_t, bool, const std::vector<const dz_forefront_s*>&): Assertion `forefronts.at(max_node_index)->mcap != nullptr' failed.

and multiple similar to this

error[MinimizerMapper::faster_cap]: Minimizers seem impossible to disrupt in region 39 40 0 1
T243:   00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
T243:   00000000001111111111222222222233333333334444444444555555555566666666667777777777888888888899999999990
T243:   01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
T243:   AAGTTGCTGAAAGTTATAGATGATTAATGATATCAATTTGCTTAGAAAAGGATTTTTTTCATTTGAATAAAATTCCACTTTTAAAAGATTCGTTACCAGAA
T243:                                          ---------AGGATTTTTTTCATTTGAA- (#0, 1 hits)
T243:                                           ----------TATTCAAATGAAAAAAATC-- (#2, 2 hits)
T243:                                              ----------TTTTATTCAAATGAAAAAA (#3, 4 hits)
T243:                                               ----------ATTTTATTCAAATGAAAAA---------- (#4, 5 hits)
T243:                                                          ---------GAATAAAATTCCACTTTTA---------- (#1, 1 hits)
Mininizer 0 agg start 39 length 29 core start 48 length 19
Mininizer 2 agg start 40 length 31 core start 68 length 19
Mininizer 3 agg start 43 length 29 core start 71 length 19
Mininizer 4 agg start 44 length 39 core start 72 length 19
Mininizer 1 agg start 55 length 38 core start 64 length 19
Read sequence: AAGTTGCTGAAAGTTATAGATGATTAATGATATCAATTTGCTTAGAAAAGGATTTTTTTCATTTGAATAAAATTCCACTTTTAAAAGATTCGTTACCAGAA
Read quality: ,,::F:,F,:FF,,F:FF:::,,FF,,,:FF,F,:,F:F::FF:F,F,FF,,F:F,,FFFF,FFF,F,:FFF:,,F,::FF,FF,,,,,F:,::,F,FF,,

5. What data and command can the vg dev team use to make the problem happen? Data is not public. Pangenome was built step-by-step with cactus-minigraph, followed by:

vg prune -P -k 19 -t ${threads} -p genome.vg > genome.prune.vg
vg mod -X 256 genome.prune.vg > genome.prunemod.vg
vg index -Z 8192 -k 16 -X 2 --temp-dir workDir -t $threads -x genome.xg -g genome.gcsa genome.prunemod.vg
vg minimizer -p -t $threads -k 19 -w 11 -d genome.dist -o genome.min genome.gbz

vg giraffe ran with:

vg giraffe -M 10 --fragment-mean 500 --fragment-stdev 250 -L 2000 --report-name aln.out.tsv -Z genome.gbz -m genome.min -d genome.dist -o SAM -t $threads -f C5752_R1.fastq.gz -f C5752_R2.fastq.gz > giraffe.out.sam

6. What does running vg version say?

vg version v1.55.0 "Bernolda"
Compiled with g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 on Linux
Linked against libstd++ 20230528
Built by jeizenga@mustard

Any assistance on how to troubleshoot this issue will be much appreciated!

adamnovak commented 2 months ago

@glennhickey The Cactus-Minigraph pipeline won't leave any absurdly long nodes, right?

I'm not sure how to connect either the error from faster_cap or the error from vg::DozeuInterface::calculate_max_position to the fact that some paths in the graph might be very long.

What looks most suspicious is your indexing. You shouldn't need to make genome.gcsa for Giraffe, and genome.xg here is not going to be usable with genome.gbz because they use different node IDs and have different nodes. But it's possible Giraffe is picking it up anyway; it automatically finds any inputs you don't give it based on filename patterns, so that you can just point it at a graph that has all the indexes next to it, and it can use a .xg file alongside a .gbz if one is available, to save on reconstructing some path position information.

I would try keeping all the files generated from the pruned/mod-ed graphs in a different directory. You can also run Giraffe with --progress and it should tell you what it is loading, which might help you figure out if it is loading anything it shouldn't be using.