search

vgteam / vg

tools for working with genome variation graphs
https://biostars.org/tag/vg/
Other
1.1k stars 194 forks source link

VG construct crashes #2490

Open prashantpandey opened 4 years ago

prashantpandey commented 4 years ago

Hi,

vg construct crashes while constructing vg representation. Can't share VCF files as they are protected. This is on GRCh38. Any help?

Crash report for vg v1.19.0 "Tramutola"
Stack trace (most recent call last):
#9    Object "/home/ppandey2/bin/vg", at 0x4d6a79, in _start
#8    Object "/home/ppandey2/bin/vg", at 0x1ac8318, in __libc_start_main
#7    Object "/home/ppandey2/bin/vg", at 0x40a7d2, in main
#6    Object "/home/ppandey2/bin/vg", at 0x9a94e7, in vg::subcommand::Subcommand::operator()(int, char**) const
#5    Object "/home/ppandey2/bin/vg", at 0x9e918d, in main_construct(int, char**)
#4    Object "/home/ppandey2/bin/vg", at 0xbb5167, in vg::Constructor::construct_graph(std::vector<FastaReference*, std::allocator<FastaReference*> > const&, std::vector<vcflib::VariantCallFile*, std::allocator<vcflib::VariantCallFile*> > const&, std::vector<FastaReference*, std::allocator<FastaReference*> > const&, std::function<void (vg::Graph&)>)
#3    Object "/home/ppandey2/bin/vg", at 0x1acc231, in __assert_fail
#2    Object "/home/ppandey2/bin/vg", at 0x1acc1bb, in __assert_fail_base
#1    Object "/home/ppandey2/bin/vg", at 0x1ad8be0, in abort
#0    Object "/home/ppandey2/bin/vg", at 0x10ab3d7, in raise
ekg commented 4 years ago

Which vg version or commit is this?

On Fri, Sep 27, 2019, 17:54 Prashant Pandey notifications@github.com wrote:

Hi,

vg construct crashes while constructing vg representation. Can't share VCF files as they are protected. This is on GRCh38. Any help?

Crash report for vg v1.19.0 "Tramutola" Stack trace (most recent call last):

9 Object "/home/ppandey2/bin/vg", at 0x4d6a79, in _start

8 Object "/home/ppandey2/bin/vg", at 0x1ac8318, in __libc_start_main

7 Object "/home/ppandey2/bin/vg", at 0x40a7d2, in main

6 Object "/home/ppandey2/bin/vg", at 0x9a94e7, in vg::subcommand::Subcommand::operator()(int, char**) const

5 Object "/home/ppandey2/bin/vg", at 0x9e918d, in main_construct(int, char**)

4 Object "/home/ppandey2/bin/vg", at 0xbb5167, in vg::Constructor::construct_graph(std::vector<FastaReference, std::allocator<FastaReference> > const&, std::vector<vcflib::VariantCallFile, std::allocator<vcflib::VariantCallFile> > const&, std::vector<FastaReference, std::allocator<FastaReference> > const&, std::function<void (vg::Graph&)>)

3 Object "/home/ppandey2/bin/vg", at 0x1acc231, in __assert_fail

2 Object "/home/ppandey2/bin/vg", at 0x1acc1bb, in __assert_fail_base

1 Object "/home/ppandey2/bin/vg", at 0x1ad8be0, in abort

0 Object "/home/ppandey2/bin/vg", at 0x10ab3d7, in raise

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AABDQEPEAB7NNNNUXIQE24DQLYUC5A5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HOGD4LQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AABDQEKY3Y5YPKVGTW3XUITQLYUC5ANCNFSM4I3ICBRA .

prashantpandey commented 4 years ago

vg: variation graph tool, version v1.19.0 "Tramutola"

glennhickey commented 4 years ago

These sorts of errors often come from the contig names in the VCF not matching those in the fasta.

On Sat, Sep 28, 2019 at 9:32 PM Prashant Pandey notifications@github.com wrote:

vg: variation graph tool, version v1.19.0 "Tramutola"

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AAG373SAEI2OTQNYA5E7SP3QMAARJA5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD73FUFI#issuecomment-536238613, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG373VSUBARKMMTHFA27DLQMAARJANCNFSM4I3ICBRA .

prashantpandey commented 4 years ago

Thanks, @glennhickey I was able to fix the contig name issue. Successfully constructed the vg graph. However, another crash occurred while building the XG and GBWT index.

Could you please help with this?

Crash report for vg v1.19.0 "Tramutola"
Stack trace (most recent call last):
#12   Object "/home/ppandey2/bin/vg", at 0x4d6a79, in _start
#11   Object "/home/ppandey2/bin/vg", at 0x1ac8318, in __libc_start_main
#10   Object "/home/ppandey2/bin/vg", at 0x40a7d2, in main
#9    Object "/home/ppandey2/bin/vg", at 0x9a94e7, in vg::subcommand::Subcommand::operator()(int, char**) const
#8    Object "/home/ppandey2/bin/vg", at 0x926936, in main_index(int, char**)
#7    Object "/home/ppandey2/bin/vg", at 0xc13ea4, in vg::VGset::to_xg(xg::XG&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, vg::Path, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, vg::Path> > >*)
#6    Object "/home/ppandey2/bin/vg", at 0x14c06ae, in xg::XG::from_enumerators(std::function<void (std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const&)> const&)> const&, std::function<void (std::function<void (long const&, bool const&, long const&, bool const&)> const&)> const&, std::function<void (std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const&, bool const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool const&, bool const&)> const&)> const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
#5    Object "/home/ppandey2/bin/vg", at 0x14aa49c, in sdsl::int_vector<(unsigned char)0>::operator[](unsigned long const&) const [clone .isra.330]
#4    Object "/home/ppandey2/bin/vg", at 0x407f48, in sdsl::int_vector<(unsigned char)0>::operator[](unsigned long const&) const [clone .isra.330] [clone .part.331]
#3    Object "/home/ppandey2/bin/vg", at 0x1acc231, in __assert_fail
#2    Object "/home/ppandey2/bin/vg", at 0x1acc1bb, in __assert_fail_base
#1    Object "/home/ppandey2/bin/vg", at 0x1ad8be0, in abort
#0    Object "/home/ppandey2/bin/vg", at 0x10ab3d7, in raise
glennhickey commented 4 years ago

This may be related to https://github.com/vgteam/vg/issues/2451, where xg indexing is running out of disk space. If that's the case, something to try would be to set the TMPDIR environment variable to point to somewhere with lots of free space before running vg index.

On Wed, Oct 2, 2019 at 9:49 AM Prashant Pandey notifications@github.com wrote:

Thanks, @glennhickey https://github.com/glennhickey I was able to fix the contig name issue. Successfully constructed the vg graph. However, another crash occurred while building the XG and GBWT index.

Could you please help with this?

Crash report for vg v1.19.0 "Tramutola" Stack trace (most recent call last):

12 Object "/home/ppandey2/bin/vg", at 0x4d6a79, in _start

11 Object "/home/ppandey2/bin/vg", at 0x1ac8318, in __libc_start_main

10 Object "/home/ppandey2/bin/vg", at 0x40a7d2, in main

9 Object "/home/ppandey2/bin/vg", at 0x9a94e7, in vg::subcommand::Subcommand::operator()(int, char**) const

8 Object "/home/ppandey2/bin/vg", at 0x926936, in main_index(int, char**)

7 Object "/home/ppandey2/bin/vg", at 0xc13ea4, in vg::VGset::to_xg(xg::XG&, std::function<bool (std::cxx11::basic_string<char, std::char_traits, std::allocator > const&)> const&, std::map<std::cxx11::basic_string<char, std::char_traits, std::allocator >, vg::Path, std::less<std::cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::cxx11::basic_string<char, std::char_traits, std::allocator > const, vg::Path> > >*)

6 Object "/home/ppandey2/bin/vg", at 0x14c06ae, in xg::XG::from_enumerators(std::function<void (std::function<void (std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, long const&)> const&)> const&, std::function<void (std::function<void (long const&, bool const&, long const&, bool const&)> const&)> const&, std::function<void (std::function<void (std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, long const&, bool const&, std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool const&, bool const&)> const&)> const&, bool, std::cxx11::basic_string<char, std::char_traits, std::allocator >)

5 Object "/home/ppandey2/bin/vg", at 0x14aa49c, in sdsl::int_vector<(unsigned char)0>::operator[](unsigned long const&) const [clone .isra.330]

4 Object "/home/ppandey2/bin/vg", at 0x407f48, in sdsl::int_vector<(unsigned char)0>::operator[](unsigned long const&) const [clone .isra.330] [clone .part.331]

3 Object "/home/ppandey2/bin/vg", at 0x1acc231, in __assert_fail

2 Object "/home/ppandey2/bin/vg", at 0x1acc1bb, in __assert_fail_base

1 Object "/home/ppandey2/bin/vg", at 0x1ad8be0, in abort

0 Object "/home/ppandey2/bin/vg", at 0x10ab3d7, in raise

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AAG373XXNPKP6T57VTTJA6DQMSRHRA5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAEZ3CY#issuecomment-537501067, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG373QXJOY2XT7NJSHKONDQMSRHRANCNFSM4I3ICBRA .

prashantpandey commented 4 years ago

I am trying with a different /tmp. Although, default /tmp had 1 GB free space. Does it need more?

glennhickey commented 4 years ago

I can't give you an exact number but in my limited experience, I've seen it need much more than 1G for a whole genome graph.

On Wed, Oct 2, 2019 at 10:02 AM Prashant Pandey notifications@github.com wrote:

I am trying with a with a different /tmp. Although, default /tmp had 1 GB free space. Does it need more?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AAG373UJHI3Q52UIX7QS433QMSSX7A5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAE3I5A#issuecomment-537506932, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG373REZGIJTBYNKTDOASTQMSSX7ANCNFSM4I3ICBRA .

prashantpandey commented 4 years ago

Hi @glennhickey and @ekg

I initially reported the contig name mismatch issue which I fixed by changing the contig name in fasta file. But it only works for chr1 and not for any other chromosomes.

Could you please help with this?

For example, for Chr22 it doesn't work. The fasta file contains this:

>chr22 dna:chromosome chromosome:GRCh38:22:1:50818468:1 REF

And the vcf file contains a contig 

...............
##contig=<ID=chr22,length=50818468>
...............
chr22   <position>        .       REF      ALT

Error:

vg: src/constructor.cpp:2095: void vg::Constructor::construct_graph(const std::vector<FastaReference*>&, const std::vector<vcflib::VariantCallFile*>&, const std::vector<FastaReference*>&, std::function<void(vg::Graph&)>): Assertion `reference_for.count(fasta_contig)' failed.
ERROR: Signal 6 occurred. VG has crashed. Run 'vg bugs --new' to report a bug.
Stack trace path: /tmp/vg_crash_1e953y/stacktrace.txt
glennhickey commented 4 years ago

Have you tried to change

chr22 dna:chromosome chromosome:GRCh38:22:1:50818468:1 REF

to just >chr22 in the fasta file?

I think there are two vg issues to be addressed here: We need to change that assert into something more verbose, and I think the fasta parser needs t obe more robust to special characters like tabs in the contig names.

On Sat, Oct 19, 2019 at 10:14 AM Prashant Pandey notifications@github.com wrote:

Hi @glennhickey https://github.com/glennhickey ,

I initially reported the contig name mismatch issue which I fixed by changing the contig name in fasta file. But it only works for chr1 and not for any other chromosomes.

Could you please help with this?

For example, for Chr22 it doesn't work. The fasta file contains this:

chr22 dna:chromosome chromosome:GRCh38:22:1:50818468:1 REF

And the vcf file contains a contig

...............

contig=

............... chr22 . REF ALT

Error:

vg: src/constructor.cpp:2095: void vg::Constructor::construct_graph(const std::vector<FastaReference>&, const std::vector<vcflib::VariantCallFile>&, const std::vector<FastaReference*>&, std::function<void(vg::Graph&)>): Assertion `reference_for.count(fasta_contig)' failed. ERROR: Signal 6 occurred. VG has crashed. Run 'vg bugs --new' to report a bug. Stack trace path: /tmp/vg_crash_1e953y/stacktrace.txt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AAG373S2PAD3ZKEVASRLGQDQPMI4HA5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBXRCNY#issuecomment-544149815, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG373XK2WQGLZOM5LKWV7TQPMI4HANCNFSM4I3ICBRA .

prashantpandey commented 4 years ago

Hi @glennhickey

I figured the issue was in the fai file. Those were generated on old fasta files.

I was able to construct the vg representation but the index construction failed.

Here's the log. Could you please tell me what's the problem? 

vg: /home/anovak/workspace/vg/include/sdsl/int_vector.hpp:1405: sdsl::int_vector<<anonymous> >::const_reference sdsl::int_vector<<anonymous> >::operator[](const size_type&) const [with unsigned char t_width = 0; sdsl::int_vector<<anonymous> >::const_reference = const long unsigned int; sdsl::int_vector<<anonymous> >::size_type = long unsigned int]: Assertion `idx < this->size()' failed.
ERROR: Signal 6 occurred. VG has crashed. Run 'vg bugs --new' to report a bug.
Stack trace path: /tmp/vg_crash_ZsjuJl/stacktrace.txt
#12   Object "/home/ppandey2/bin/vg", at 0x4d6a79, in _start
#11   Object "/home/ppandey2/bin/vg", at 0x1ac8318, in __libc_start_main
#10   Object "/home/ppandey2/bin/vg", at 0x40a7d2, in main
#9    Object "/home/ppandey2/bin/vg", at 0x9a94e7, in vg::subcommand::Subcommand::operator()(int, char**) const
#8    Object "/home/ppandey2/bin/vg", at 0x926936, in main_index(int, char**)
#7    Object "/home/ppandey2/bin/vg", at 0xc13ea4, in vg::VGset::to_xg(xg::XG&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, vg::Path, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, vg::Path> > >*)
#6    Object "/home/ppandey2/bin/vg", at 0x14c06ae, in xg::XG::from_enumerators(std::function<void (std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const&)> const&)> const&, std::function<void (std::function<void (long const&, bool const&, long const&, bool const&)> const&)> const&, std::function<void (std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long const&, bool const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool const&, bool const&)> const&)> const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
#5    Object "/home/ppandey2/bin/vg", at 0x14aa49c, in sdsl::int_vector<(unsigned char)0>::operator[](unsigned long const&) const [clone .isra.330]
#4    Object "/home/ppandey2/bin/vg", at 0x407f48, in sdsl::int_vector<(unsigned char)0>::operator[](unsigned long const&) const [clone .isra.330] [clone .part.331]
#3    Object "/home/ppandey2/bin/vg", at 0x1acc231, in __assert_fail
#2    Object "/home/ppandey2/bin/vg", at 0x1acc1bb, in __assert_fail_base
#1    Object "/home/ppandey2/bin/vg", at 0x1ad8be0, in abort
#0    Object "/home/ppandey2/bin/vg", at 0x10ab3d7, in raise
prashantpandey commented 4 years ago

Hi @glennhickey and @ekg ,

I wanted to add that I am trying to build XG and GBWT index on TCGA dataset. I can't share the VCF files because they are protected but if you have access to OV, LUAD and BRCA projects you can reproduce the bug locally.

Thanks, Prashant

glennhickey commented 4 years ago

I think that crash is from xg running out of disk. This is becoming a frequent issue since we changed to the xg that does disk-based sort.. There's an issue here #2451.

If this is indeed the problem, the work-around is to set the TMPDIR environment variable to point somewhere where's there's lots of free disk space.

ekg commented 4 years ago

Maybe we should default to using the output xg filename as the temporary file base name. Then we would be much less likely to have this problem.

On Tue, Oct 22, 2019, 18:41 Glenn Hickey notifications@github.com wrote:

I think that crash is from xg running out of disk. This is becoming a frequent issue since we changed to the xg that does disk-based sort.. There's an issue here #2451 https://github.com/vgteam/vg/issues/2451.

If this is indeed the problem, the work-around is to set the TMPDIR environment variable to point somewhere where's there's lots of free disk space.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AABDQEME6WGWDHVTB5JO4UTQP43LDA5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6SY7I#issuecomment-545074301, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEO2YAIKBF5IA5VU473QP43LDANCNFSM4I3ICBRA .

prashantpandey commented 4 years ago

Thanks! Let me try that. In any case, if the error message explains the problem that would be easier for the user to fix. As of now, it looks like a technical bug in how the SDSL vector is used or something.

glennhickey commented 4 years ago

@ekg I think that'd be an improvement. Pretty sure it's what samtools sort does as well.

prashantpandey commented 4 years ago

So, I am able to build the XG index OV dataset. But when I try to build both XG and GBWT index the process seems to go in an infinite loop. It just keeps running without any change to the MAX RSS or temp disk space.

Do you guys know if GBWT index can support ~5K samples in the OV dataset? It works fine for the 1000Genomes dataset.

glennhickey commented 4 years ago

There's some documentation on that here: https://github.com/vgteam/vg/wiki/Indexing-Huge-Datasets

On Tue, Oct 22, 2019 at 3:33 PM Prashant Pandey notifications@github.com wrote:

So, I am able to build the XG index OV dataset. But when I try to build both XG and GBWT index the process seems to go in an infinite loop. It just keeps running without any change to the MAX RSS or temp disk space.

Do you guys know if GBWT index can support ~5K samples in the OV dataset? It works fine for the 1000Genomes dataset.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2490?email_source=notifications&email_token=AAG373X7LWNELES4EETVTDTQP5IQLA5CNFSM4I3ICBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB65RZI#issuecomment-545118437, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG373XTSNHE2RAGRJJII5TQP5IQLANCNFSM4I3ICBRA .

Boer223 commented 4 years ago

Hi, @ekg and @glennhickey

I also have the similar error like above in vg construct -C -R chrA02 -r westar.fa -v jasmine.merge.25.sort.vcf.gz -t 1 -S -a >chrA02.vg, 

ERROR: Signal 11 occurred. VG has crashed. Run 'vg bugs --new' to report a bug.
Stack trace path: /tmp/vg_crash_zqVBqF/stacktrace.txt
Please include the stack trace file in your bug report!

My vg version

vg version v1.25.0 "Apice"
Compiled with g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 on Linux
Linked against libstd++ 20191114
Built by anovak@octagon

And I had also checked the reference genome chr name and CHROM tag in vcf file, they are same, but it still occurs the error.

So, how to solve the error?

Thanks in advance!