ECP-VeloC / redset

Low-level distributed erasure coding library to protect datasets of MPI applications
MIT License
2 stars 3 forks source link

Optimize RS encode/decode with Intel ISA-L #18

Open adammoody opened 4 years ago

adammoody commented 4 years ago

Intel has an open-source library supporting vectorized functions to compute RS GF(2^8) math:

https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version https://github.com/intel/isa-l

This is licensed under BSD-3. It looks like it may also support processors other than just Intel, including PowerPC. This could offer a performance boost for our internal RS encoding.

For example, the gf_vect_mul function here:

https://github.com/intel/isa-l/blob/master/include/gf_vect_mul.h

might plug in for:

https://github.com/ECP-VeloC/redset/blob/638c6501a6ce02c6fdf95e72c92b1c8a40e5cacb/src/redset_reedsolomon_common.c#L768

This will take some developer time to make the switch, and it adds a dependency. Also, if we are bottlenecked by storage or network bandwidth, reducing encode/decode time may not improve overall performance by much. So some work should be done to investigate before taking this on.