xxsds / DYNAMIC

Dynamic succinct/compressed data structures
MIT License
111 stars 21 forks source link

abort: packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() #24

Closed smoe closed 4 years ago

smoe commented 4 years ago

Hello, ran into

$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01
size = 1000000. P = 0.01
Benchmarking gap bitvector
insert ... done.
access ... done.
rank 0 ... done.
rank 1 ... done.
select 0 ... done.
select 1 ... done.
remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Aborted (core dumped)

And peeking a boo in gdb:

Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa19f15e55b in __GI_abort () at abort.c:79
#2  0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92
#3  0x00007fa19f16d092 in __GI___assert_fail (
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101
#4  0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981
#5  0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591
#6  0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276
#7  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253
#8  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197
#9  benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122
#10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165

but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1. Anything you want me to do about it?

Cheers, Steffen

smoe commented 4 years ago

First invocation:

$ ./obj-x86_64-linux-gnu/wm_string 
SPEED
access:252ms
rank:331ms
wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)

and in gdb:

Core was generated by `./obj-x86_64-linux-gnu/wm_string'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f08b36ef55b in __GI_abort () at abort.c:79
#2  0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0", 
    file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92
#3  0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, 
    function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]")
    at assert.c:101
#4  0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=<optimized out>, rank=0, c=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:1058
#5  0x000056266528c474 in speed_select (num=100000, num_of_alphabet=<optimized out>) at ./wm_string.cpp:74
#6  0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335
#7  0x000056266528aa2e in main () at ./wm_string.cpp:394

This is very nicely reproducible.

ekg commented 4 years ago

I wonder if the recent merge of stuff from my repo caused a regression.

On Sat, Jul 18, 2020 at 7:29 PM Steffen Möller notifications@github.com wrote:

First invocation:

$ ./obj-x86_64-linux-gnu/wm_string SPEED access:252ms rank:331ms wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed. Aborted (core dumped)

and in gdb:

Core was generated by `./obj-x86_64-linux-gnu/wm_string'. Program terminated with signal SIGABRT, Aborted.

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

1 0x00007f08b36ef55b in __GI_abort () at abort.c:79

2 0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0",

file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92

3 0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200,

function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]")
at assert.c:101

4 0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=, rank=0, c=) at /usr/include/c++/9/bits/stl_vector.h:1058

5 0x000056266528c474 in speed_select (num=100000, num_of_alphabet=) at ./wm_string.cpp:74

6 0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335

7 0x000056266528aa2e in main () at ./wm_string.cpp:394

This is very nicely reproducible.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-660514898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEMRB4UO6XYOJ4AIWDDR4HLW5ANCNFSM4PAEUNXA .

ekg commented 4 years ago

Actually, it was from vgteam, correct?

On Sat, Jul 18, 2020 at 7:47 PM Erik Garrison erik.garrison@gmail.com wrote:

I wonder if the recent merge of stuff from my repo caused a regression.

On Sat, Jul 18, 2020 at 7:29 PM Steffen Möller notifications@github.com wrote:

First invocation:

$ ./obj-x86_64-linux-gnu/wm_string SPEED access:252ms rank:331ms wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed. Aborted (core dumped)

and in gdb:

Core was generated by `./obj-x86_64-linux-gnu/wm_string'. Program terminated with signal SIGABRT, Aborted.

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

1 0x00007f08b36ef55b in __GI_abort () at abort.c:79

2 0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0",

file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92

3 0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200,

function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]")
at assert.c:101

4 0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=, rank=0, c=) at /usr/include/c++/9/bits/stl_vector.h:1058

5 0x000056266528c474 in speed_select (num=100000, num_of_alphabet=) at ./wm_string.cpp:74

6 0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335

7 0x000056266528aa2e in main () at ./wm_string.cpp:394

This is very nicely reproducible.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-660514898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEMRB4UO6XYOJ4AIWDDR4HLW5ANCNFSM4PAEUNXA .

smoe commented 4 years ago

That is what I presume since I requested a pull from vgteam/DYNAMIC.

nicolaprezza commented 4 years ago

Hello, ran into

$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01
size = 1000000. P = 0.01
Benchmarking gap bitvector
insert ... done.
access ... done.
rank 0 ... done.
rank 1 ... done.
select 0 ... done.
select 1 ... done.
remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Aborted (core dumped)

And peeking a boo in gdb:

Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa19f15e55b in __GI_abort () at abort.c:79
#2  0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92
#3  0x00007fa19f16d092 in __GI___assert_fail (
    assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", 
    file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101
#4  0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981
#5  0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591
#6  0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276
#7  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253
#8  dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197
#9  benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122
#10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165

but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1. Anything you want me to do about it?

Cheers, Steffen

I thought I had solved this bug but apparently it's alive and well. I'll look into it next week.

nicolaprezza commented 4 years ago

Hi, I solved the first part of the issue:

Assertion `(size_ / int_perword == words.size() || !(words[size_ / int_perword] >> ((size_ % int_perword) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.

Actually, the code was correct and only the assert had a bug : -)

@ekg : the second issue happens in wm_string.cpp. The failed assert happens because the code calls select(rank, c) at wm_string.hpp:200 with rank == 0. Could it be simply a bug of the checker wm_string.cpp? select is not supposed to be called with rank == 0.

ekg commented 4 years ago

That does look like a bug in the check in wm_string. I'll take a look.

On Wed, Jul 22, 2020 at 12:44 PM Nicola Prezza notifications@github.com wrote:

Hi, I solved the first part of the issue:

Assertion `(size_ / int_perword == words.size() || !(words[size_ / int_perword] >> ((size_ % int_perword) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.

Actually, the code was correct and only the assert had a bug : -)

@ekg https://github.com/ekg : the second issue happens in wm_string.cpp. The failed assert happens because the code calls select(rank, c) at wm_string.hpp:200 with rank == 0. Could it be simply a bug of the checker wm_string.cpp? select is not supposed to be called with rank == 0.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-662382890, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEJZJAEW4QBBYLQRSZLR427KNANCNFSM4PAEUNXA .

smoe commented 4 years ago

There these automated build tests spreading throughout github that are executed for every commit. Is this something that should be introduced to DYNAMIC, too?

ekg commented 4 years ago

Yes, it should be done here too.

On Wed, Jul 22, 2020, 13:09 Steffen Möller notifications@github.com wrote:

There these automated build tests spreading throughout github that are executed for every commit. Is this something that should be introduced to DYNAMIC, too?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-662392691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEOXOKEIIXHGAX7L23LR43CGNANCNFSM4PAEUNXA .

smoe commented 4 years ago

All dependencies of DYNAMIC are in Debian/Ubuntu already - I guess this would be rather straight-forward to implement. Ping me if you need help.

ekg commented 4 years ago

@nicolaprezza I'm not able to reproduce the problem with wm_string. It's at least not happened in the 20-30 runs that I've done (running it in a loop and watching...)

smoe commented 4 years ago

Latest checkout:

$ ./obj-x86_64-linux-gnu/wm_string 
SPEED
access:229ms
rank:293ms
wm_string: /home/moeller/git/med-team/dynamic/a/DYNAMIC/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)

Maybe because of other compiler settings?

/usr/bin/cmake -E cmake_link_script CMakeFiles/wm_string.dir/link.txt --verbose=1
/usr/local/bin/c++  -g -O2 -fdebug-prefix-map=/home/moeller/git/med-team/dynamic/a/DYNAMIC=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --std=c++11  -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -rdynamic CMakeFiles/wm_string.dir/wm_string.cpp.o  -o wm_string 

The benchmark seems stable now. Would you have other ideas what I should test over here?

./obj-x86_64-linux-gnu/cw-bwt /etc/passwd /tmp/bla.txt worked - but not on /proc/cpuinfo ./obj-x86_64-linux-gnu/h0_lz77 /etc/passwd /tmp/bla.txt worked ./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txt worked ./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txtworked

I can get a VM online somewhere for a common playground to investigate the wm_string issue.

nicolaprezza commented 4 years ago

I can reproduce it. Have you enabled asserts? with cmake, it's simply cmake .. -DCMAKE_BUILD_TYPE=Debug

ekg commented 4 years ago

Thanks :) That's my mistake. I can reproduce it by enabling the assert!

ekg commented 4 years ago

It does look like a possible problem with the input. There is a quirky PRNG being used. I might switch to modern c++ random distributions.

nicolaprezza commented 4 years ago

Thanks @ekg! now it works.