search

mrboojum / CRoaring4VS

C++ Roaring bitmaps for microsoft Visual Studio 2015 compiled with clang.
Apache License 2.0
8 stars 4 forks source link

Add benchmark #2

Open lemire opened 8 years ago

lemire commented 8 years ago

It would be interesting to have some benchmark so that we could check for possible performance issues.

mrboojum commented 8 years ago

Indeed that would be interesting. However: 1) I see that within the CRoaring benchmark cycles are counted using the macros RDTSC_START and RDTSC_FINAL. I've no experience with measuring benchmarks in cycles (usually I just measure time and/or use the visual studio performance analyzer to detect bottlenecks). I'll try to compile those headers. 2) There would be nothing to compare the performance except a compilation with mingw64 on the same machine. Do you have a suggestion on which test in the benchmark directory would be the most interesting to convert to c++? Personally I would be more interested in a comparison with fastbit (on the data for my project). Do you know if there is previous work done on this?

lemire commented 8 years ago

@mrboojum

1) I see that within the CRoaring benchmark cycles are counted using the macros RDTSC_START and RDTSC_FINAL. I've no experience with measuring benchmarks in cycles (usually I just measure time and/or use the visual studio performance analyzer to detect bottlenecks). I'll try to compile those headers.

RDTSC_START and RDTSC_FINAL implement the recommended way to benchmark as per Intel's documentation. Unfortunately, it is unlikely to work with Visual Studio as I think it won't allow inline assembly.

Benchmarks serve to produce a reference. So you know how long it takes to execute some task given a compiler, its settings and a given machine. You can publish these benchmarks. They are reproducible. Performance analysis is what you do afterward to either better understand or improve these results.

Do you have a suggestion on which test in the benchmark directory would be the most interesting to convert to c++?

Yes: https://github.com/RoaringBitmap/CBitmapCompetition

The README provides sample results that I got on a Linux box. As you can see, CRoaring has excellent performance and compression.

I think BitMagic is probably the most interesting target to compare against. Note however that CRoaring will be at a disadvantage with the intrinsics disabled. Still, we should document any differences.

I think that this handicap will go away "soon" because I expect Microsoft to add support for Intel intrinsics at some point. Another source of worry is that Microsoft CodeGen could generate poor code if the clang/CodeGen combo is too immature.

Note that you can try with the Intel compiler (you can get a version for free). The Intel compiler would allow you to build CRoaring without compromise...

Personally I would be more interested in a comparison with fastbit (on the data for my project). Do you know if there is previous work done on this?

The CBitmapCompetition package I link to above compares against WAH, which is used by fastbit.

A direct comparison with fastbit is more difficult because fastbit is lot more than just just compressed bitmaps... it is a whole framework. Still, speaking for myself, I'd encourage you to pursue any comparison you can come up with. But keep in mind that the results are more difficult to interpret.

mrboojum commented 8 years ago

Perhaps I'll look into this in the future but for now I'll first focus on bridging the functionality gap between ''fastbit'' and croaring. Despite that only a small part of the ''fastbit'' framework is used within the application this will still be quite a challenge. If this is successful it makes sense for me to look into the performance.