lemire
commented
6 years ago Thank you for raising the issue.
Open ghost opened 6 years ago
lemire
commented
6 years ago Thank you for raising the issue.
lemire
commented
6 years ago What happens if you type make && make test?
I expect that the script will run through... reporting a few harmless core dumps. If you are not interested in benchmarking memory usage, you can simply ignore these failed tests.
I would expect that if type make test, the script will happily run through, despite the specific failures. You will simply not get memory-usage reports. If that's not the case, please report it... It would then be scripting bug.
In order to get things to compile (old gcc/g++ stack) I changed (...)-m64
Why would -m64 be necessary?
On another note, should we actually doing benchmarks while recording allocated memory anyway by default ?
We want to be benchmarking memory usage. I feel that it is important to try to compare fairly memory usage, as part of the benchmark. As I stated above, if you don't care about getting these numbers, just ignore this particular part of the benchmark.
(...) usage perhaps this is not entirely safe
It is not part of the C standard, for sure. But we need some way to track memory usage and I don't know a better way. As far as I can tell, what we do is the standard approach. I'd love to be corrected though!
Your stack trace seems to indicate that the problem occurs with the malloc function. Our malloc function is this (to track memory usage, we overload malloc):
void* malloc(size_t sz) {
void *(*libc_malloc)(size_t) = dlsym(RTLD_NEXT, "malloc");
void * answerplus = libc_malloc(sz + sizeof(size_t) + sizeof(myalloc_cookie) );
if(answerplus == NULL) return answerplus;// nothing can be done
malloced_memory_usage += sz;
memcpy(answerplus ,&myalloc_cookie,sizeof(myalloc_cookie));
memcpy((char *) answerplus + sizeof(myalloc_cookie),&sz,sizeof(sz));
return ((char *) answerplus) + sizeof(size_t) + sizeof(myalloc_cookie);
}So what could go wrong?
Formally speaking, we should check that libc_malloc is not-null. But, really, if you have no malloc, life is not good for you. And that's not what seems to be happening.
The fact that you need -m64 suggests you have both 32-bit and 64-bit software, so this can create issues. Could you simplify this part?
It is possible that libc_malloc is not, actually, the libc malloc we expect. Maybe typing ldd ./malloced_roaring_benchmarks could help. You should get something like this...
$ ldd ./malloced_roaring_benchmarks
linux-vdso.so.1 => (0x00007ffed835e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4c37635000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4c3726c000)
/lib64/ld-linux-x86-64.so.2 (0x00005588d2b92000)
ghost
commented
6 years ago What happens if you type make && make test?
Exactly as you say, I can of course ignore theses core dumps (or better yet modify all.sh).
Why would -m64 be necessary?
without -m64 the macro RDTSC_START in roaring_benchmarks.c causes gcc errors - most likely due to my older compiler (I will try a more recent one soon):
In file included from src/roaring_benchmarks.c:11:0:
src/roaring.c: In function 'bitset_set_list_withcard':
src/roaring.c:10009:5: error: can't find a register in class 'GENERAL_REGS' while reloading 'asm'
src/roaring.c:10009:5: error: 'asm' operand has impossible constraints
src/roaring_benchmarks.c: In function 'main':
src/roaring_benchmarks.c:130:5: error: PIC register clobbered by '%rbx' in 'asm'It is not part of the C standard, for sure. But we need some way to track memory usage and I don't know a better way.
I have had good luck with dmalloc from http://dmalloc.com/ granted it really slows things down but it does a lot more than just "statistics". If you want I could remove the DRECORD_MALLOCS and try things with the dmalloc library I mention.
It is possible that libc_malloc is not, actually, the libc malloc we expect.
My ldd output
# ldd malloced_roaring_benchmarks
libdl.so.1 => /lib/64/libdl.so.1
libc.so.1 => /lib/64/libc.so.1
libm.so.2 => /lib/64/libm.so.2I have had good luck with dmalloc from http://dmalloc.com/ granted it really slows things down but it does a lot more than just "statistics". If you want I could remove the DRECORD_MALLOCS and try things with the dmalloc library I mention.
Yes! Pull Request invited!
Disclaimer I run an "old" gcc compiler so maybe that's the problem, but I still thought I would share.
Note, it seems like -DRECORD_MALLOCS causes a failure scenario on my system with the "malloced_roaring_benchmarks" if it is compiled as per the default in the Makefile.
If I remove the flag -DRECORD_MALLOCS from the Makefile the make file everything works.
The following is what happens if I leave -DRECORD_MALLOCS active, e.g. the failure mode. On another note, should we actually doing benchmarks while recording allocated memory anyway by default ?
We experience a "Segmentation Fault" on the call to libc_malloc(...) on line 15 of this header file "src/cmemcounter.h" which is used to track memory - usage perhaps this is not entirely safe
My Server config (recent SmartOS VM with a zone limited to just 4GB, gcc 4.7.4):
In order to get things to compile (old gcc/g++ stack) I changed the following three (3) files:
It seems all the runs of ./malloced_roaring_benchmarks fail with a Segmentation Fault except one which hangs (or seems to hang) this exception is census1881_srt. I will only show what happens for the more typical Segmentation Fault case.
Running the the same command above under truss
looking at the generated core file in this case "core.malloced_roaring.89085"