Closed smoe closed 4 years ago
First invocation:
$ ./obj-x86_64-linux-gnu/wm_string
SPEED
access:252ms
rank:331ms
wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)
and in gdb:
Core was generated by `./obj-x86_64-linux-gnu/wm_string'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f08b36ef55b in __GI_abort () at abort.c:79
#2 0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0",
file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92
#3 0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200,
function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]")
at assert.c:101
#4 0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=<optimized out>, rank=0, c=<optimized out>) at /usr/include/c++/9/bits/stl_vector.h:1058
#5 0x000056266528c474 in speed_select (num=100000, num_of_alphabet=<optimized out>) at ./wm_string.cpp:74
#6 0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335
#7 0x000056266528aa2e in main () at ./wm_string.cpp:394
This is very nicely reproducible.
I wonder if the recent merge of stuff from my repo caused a regression.
On Sat, Jul 18, 2020 at 7:29 PM Steffen Möller notifications@github.com wrote:
First invocation:
$ ./obj-x86_64-linux-gnu/wm_string SPEED access:252ms rank:331ms wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string
::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed. Aborted (core dumped) and in gdb:
Core was generated by `./obj-x86_64-linux-gnu/wm_string'. Program terminated with signal SIGABRT, Aborted.
0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt
0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
1 0x00007f08b36ef55b in __GI_abort () at abort.c:79
2 0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0",
file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92
3 0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200,
function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]") at assert.c:101
4 0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=
, rank=0, c= ) at /usr/include/c++/9/bits/stl_vector.h:1058 5 0x000056266528c474 in speed_select (num=100000, num_of_alphabet=
) at ./wm_string.cpp:74 6 0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335
7 0x000056266528aa2e in main () at ./wm_string.cpp:394
This is very nicely reproducible.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-660514898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEMRB4UO6XYOJ4AIWDDR4HLW5ANCNFSM4PAEUNXA .
Actually, it was from vgteam, correct?
On Sat, Jul 18, 2020 at 7:47 PM Erik Garrison erik.garrison@gmail.com wrote:
I wonder if the recent merge of stuff from my repo caused a regression.
On Sat, Jul 18, 2020 at 7:29 PM Steffen Möller notifications@github.com wrote:
First invocation:
$ ./obj-x86_64-linux-gnu/wm_string SPEED access:252ms rank:331ms wm_string: /home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string
::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed. Aborted (core dumped) and in gdb:
Core was generated by `./obj-x86_64-linux-gnu/wm_string'. Program terminated with signal SIGABRT, Aborted.
0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt
0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
1 0x00007f08b36ef55b in __GI_abort () at abort.c:79
2 0x00007f08b36ef42f in __assert_fail_base (fmt=0x7f08b3855b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56266529ce16 "rank > 0",
file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200, function=<optimized out>) at assert.c:92
3 0x00007f08b36fe092 in __GI___assert_fail (assertion=0x56266529ce16 "rank > 0", file=0x56266529bb40 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/wm_string.hpp", line=200,
function=0x56266529c628 "ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]") at assert.c:101
4 0x0000562665294d2f in dyn::wm_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256u, 16u> > >::select (this=
, rank=0, c= ) at /usr/include/c++/9/bits/stl_vector.h:1058 5 0x000056266528c474 in speed_select (num=100000, num_of_alphabet=
) at ./wm_string.cpp:74 6 0x0000562665290ada in speed_test (num=100000, num_of_alphabet=10000000) at /usr/include/c++/9/bits/char_traits.h:335
7 0x000056266528aa2e in main () at ./wm_string.cpp:394
This is very nicely reproducible.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-660514898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEMRB4UO6XYOJ4AIWDDR4HLW5ANCNFSM4PAEUNXA .
That is what I presume since I requested a pull from vgteam/DYNAMIC.
Hello, ran into
$ obj-$(dpkg-architecture -qDEB_TARGET_GNU_TYPE)/benchmark -g 1000000 0.01 size = 1000000. P = 0.01 Benchmarking gap bitvector insert ... done. access ... done. rank 0 ... done. rank 1 ... done. select 0 ... done. select 1 ... done. remove ... benchmark: /home/moeller/git/med-team/dynamic/dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp:465: void dyn::packed_vector::remove(uint64_t): Assertion `(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && "uninitialized non-zero values in the end of the vector"' failed. Aborted (core dumped)
And peeking a boo in gdb:
Core was generated by `obj-x86_64-linux-gnu/benchmark -g 1000000 0.01'. Program terminated with signal SIGABRT, Aborted. #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007fa19f15e55b in __GI_abort () at abort.c:79 #2 0x00007fa19f15e42f in __assert_fail_base (fmt=0x7fa19f2c4b48 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=<optimized out>) at assert.c:92 #3 0x00007fa19f16d092 in __GI___assert_fail ( assertion=0x557aacd476a0 "(size_ / int_per_word_ == words.size() || !(words[size_ / int_per_word_] >> ((size_ % int_per_word_) * width_))) && \"uninitialized non-zero values in the end of the vector\"", file=0x557aacd47058 "/home/moeller/git/med-team/dynamic/xxsds-dynamic-1.0~alpha.1/include/dynamic/internal/packed_vector.hpp", line=465, function=0x557aacd477e0 "void dyn::packed_vector::remove(uint64_t)") at assert.c:101 #4 0x0000557aacd3867c in dyn::packed_vector::remove (this=0x557aae434630, i=<optimized out>) at ./include/dynamic/internal/packed_vector.hpp:981 #5 0x0000557aacd4373a in dyn::spsi<dyn::packed_vector, 256u, 16u>::node::remove (this=0x557aae4232c0, i=3) at ./include/dynamic/internal/packed_vector.hpp:591 #6 0x0000557aacd46a16 in dyn::spsi<dyn::packed_vector, 256u, 16u>::remove (this=0x7ffd24959ba0, i=3) at ./include/dynamic/internal/spsi.hpp:276 #7 dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::delete1 (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:253 #8 dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> >::remove (i=123, this=0x7ffd24959ba0) at ./include/dynamic/internal/gap_bitvector.hpp:197 #9 benchmark_bv<dyn::gap_bitvector<dyn::spsi<dyn::packed_vector, 256u, 16u> > > (size=1000000, p=<optimized out>) at ./benchmark.cpp:122 #10 0x0000557aacd373f5 in main (argc=<optimized out>, argv=0x7ffd24959d48) at ./benchmark.cpp:165
but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1. Anything you want me to do about it?
Cheers, Steffen
I thought I had solved this bug but apparently it's alive and well. I'll look into it next week.
Hi, I solved the first part of the issue:
Assertion `(size_ / int_perword == words.size() || !(words[size_ / int_perword] >> ((size_ % int_perword) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Actually, the code was correct and only the assert had a bug : -)
@ekg : the second issue happens in wm_string.cpp. The failed assert happens because the code calls select(rank, c) at wm_string.hpp:200 with rank == 0. Could it be simply a bug of the checker wm_string.cpp? select is not supposed to be called with rank == 0.
That does look like a bug in the check in wm_string. I'll take a look.
On Wed, Jul 22, 2020 at 12:44 PM Nicola Prezza notifications@github.com wrote:
Hi, I solved the first part of the issue:
Assertion `(size_ / int_perword == words.size() || !(words[size_ / int_perword] >> ((size_ % int_perword) * width_))) && "uninitialized non-zero values in the end of the vector"' failed.
Actually, the code was correct and only the assert had a bug : -)
@ekg https://github.com/ekg : the second issue happens in wm_string.cpp. The failed assert happens because the code calls select(rank, c) at wm_string.hpp:200 with rank == 0. Could it be simply a bug of the checker wm_string.cpp? select is not supposed to be called with rank == 0.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-662382890, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEJZJAEW4QBBYLQRSZLR427KNANCNFSM4PAEUNXA .
There these automated build tests spreading throughout github that are executed for every commit. Is this something that should be introduced to DYNAMIC, too?
Yes, it should be done here too.
On Wed, Jul 22, 2020, 13:09 Steffen Möller notifications@github.com wrote:
There these automated build tests spreading throughout github that are executed for every commit. Is this something that should be introduced to DYNAMIC, too?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xxsds/DYNAMIC/issues/24#issuecomment-662392691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDQEOXOKEIIXHGAX7L23LR43CGNANCNFSM4PAEUNXA .
All dependencies of DYNAMIC are in Debian/Ubuntu already - I guess this would be rather straight-forward to implement. Ping me if you need help.
@nicolaprezza I'm not able to reproduce the problem with wm_string. It's at least not happened in the 20-30 runs that I've done (running it in a loop and watching...)
Latest checkout:
$ ./obj-x86_64-linux-gnu/wm_string
SPEED
access:229ms
rank:293ms
wm_string: /home/moeller/git/med-team/dynamic/a/DYNAMIC/include/dynamic/internal/wm_string.hpp:200: ulint dyn::wm_string<dynamic_bitvector_t>::select(ulint, ulint) const [with dynamic_bitvector_t = dyn::succinct_bitvector<dyn::spsi<dyn::packed_bit_vector, 256, 16> >; ulint = long unsigned int]: Assertion `rank > 0' failed.
Aborted (core dumped)
Maybe because of other compiler settings?
/usr/bin/cmake -E cmake_link_script CMakeFiles/wm_string.dir/link.txt --verbose=1
/usr/local/bin/c++ -g -O2 -fdebug-prefix-map=/home/moeller/git/med-team/dynamic/a/DYNAMIC=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --std=c++11 -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -rdynamic CMakeFiles/wm_string.dir/wm_string.cpp.o -o wm_string
The benchmark seems stable now. Would you have other ideas what I should test over here?
./obj-x86_64-linux-gnu/cw-bwt /etc/passwd /tmp/bla.txt
worked - but not on /proc/cpuinfo
./obj-x86_64-linux-gnu/h0_lz77 /etc/passwd /tmp/bla.txt
worked
./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txt
worked
./obj-x86_64-linux-gnu/rle_lz77_v2 /etc/passwd /tmp/bla.txt
worked
I can get a VM online somewhere for a common playground to investigate the wm_string issue.
I can reproduce it. Have you enabled asserts? with cmake, it's simply cmake .. -DCMAKE_BUILD_TYPE=Debug
Thanks :) That's my mistake. I can reproduce it by enabling the assert!
It does look like a possible problem with the input. There is a quirky PRNG being used. I might switch to modern c++ random distributions.
Thanks @ekg! now it works.
Hello, ran into
And peeking a boo in gdb:
but found it to reproduce only every 2nd to 5th invocation. Spotted with version 1.0-alpha.1. Anything you want me to do about it?
Cheers, Steffen