pelotoncycle / bsort

Extremely fast inplace radix sort
Other
42 stars 17 forks source link

Can't reproduce performance #2

Open danielpol opened 7 years ago

danielpol commented 7 years ago

I'm trying to reproduce the gensort performance you mention in the Readme. By they way, it would be great to have this running multi-threaded to take advantage of multiple cores. If I use j1e8.c I get a "Bus error" when trying to sort. If I use bsort.c or qsort.c I run for >24 hours without finishing. Worth to mention that when compiling bsort.c I get these warnings: bsort.c: In function ‘shellsort’: bsort.c:36:7: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default] memcpy(a+j_record_size, a+(j-3)_record_size, recordsize); ^ In file included from bsort.c:4:0: /usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char ’ extern void memcpy (void restrict dest, const void restrict src, ^ bsort.c:38:5: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default] memcpy(a+j_record_size, &temp, recordsize); ^ In file included from bsort.c:4:0: /usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char ’ extern void memcpy (void restrict dest, const void restrict src, ^ bsort.c:44:7: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default] memcpy(a+j_record_size, a+(j-1)_record_size, record_size); ^ In file included from bsort.c:4:0: /usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char ’ extern void memcpy (void restrict dest, const void restrict src, ^ bsort.c:46:5: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default] memcpy(a+j_record_size, &temp, record_size); ^ In file included from bsort.c:4:0: /usr/include/string.h:42:14: note: expected ‘void * restrict’ but argument is of type ‘const unsigned char ’ extern void memcpy (void restrict __dest, const void restrict src, ^ bsort.c: In function ‘radixify’: bsort.c:128:11: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default] memcpy(&buffer[stack[stack_pointer] * record_size], &buffer[stack[stack_pointer-1] * record_size], record_size); ^ In file included from bsort.c:4:0: /usr/include/string.h:42:14: note: expected ‘void * __restrict’ but argument is of type ‘const unsigned char ’ extern void memcpy (void restrict __dest, const void restrict src, ^ bsort.c:131:9: warning: passing argument 1 of ‘memcpy’ discards ‘const’ qualifier from pointer target type [enabled by default] memcpy(&buffer[stack[0] * record_size], &temp, record_size); ^ In file included from bsort.c:4:0: /usr/include/string.h:42:14: note: expected ‘void * __restrict’ but argument is of type ‘const unsigned char ’ extern void memcpy (void restrict dest, const void restrict src,

adamdeprince commented 7 years ago

Yeah, I have to clean up the warnings. What are your source files? How large are they? How much ram do you have? What is your underlying file system and I/O device? bsort is designed to perform very very well when swapping heavily, but qsort, a direct application of linux's underlying qsort function, performs very poorly when paging. j11e8 is designed for a very specific input, and deviation from that is "undefined" :-)

I may at some point make this code multithreaded - bsort is really I/O, not CPU, bound, and would benefit tremendously from having as many inflight threads as possible.

danielpol commented 7 years ago

I run this on output generated by gensort (100 bytes fixed binary records with 10 fixed binary key). ~10GB source files. I ended up using nsort trial version to do my testing.

harveywi commented 5 years ago

@danielpol, try turning on the "-v" verbose flag and seeing what happens with different values for the "-c" parameter. In my case, sorting a very small 10MB file of 128-bit numbers took over a minute, but I increased "-c" and achieved much better performance.

Some discussion about the effect of the various command line parameters would be a nice addition to the readme.

adamdeprince commented 5 years ago

Hi,

Improvements and bug fixes for this are long overdue. Gimmie a few days and I'll get this together.

On Jan 22, 2019, at 3:39 PM, William Harvey notifications@github.com wrote:

@danielpol https://github.com/danielpol, try turning on the "-v" verbose flag and seeing what happens with different values for the "-c" parameter. In my case, sorting a very small 10MB file of 128-bit numbers took over a minute, but I increased "-c" and achieved much better performance.

Some discussion about the effect of the various command line parameters would be a nice addition to the readme.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pelotoncycle/bsort/issues/2#issuecomment-456591028, or mute the thread https://github.com/notifications/unsubscribe-auth/AAC0ONfL_iLe41YY_4dNLEem_hGeGPv2ks5vF5MOgaJpZM4JmoKM.