search

celestiaorg / go-leopard

Go wrapper for Leopard Reed-Solomon erasure code library.
5 stars 6 forks source link

Decide on future of (go-)leopard #8

Open liamsi opened 3 years ago

liamsi commented 3 years ago

leopard is unmaintained and we did some (minor) modifications to the C++ code already.

There are several paths we could go with this library:

  1. maintain and improve the C++ library and continue using it via cgo (like this repository does)
  2. re-implement the algorithm in golang
  3. re-implement the performance-critical parts in assembly and package it directly with the go-code (e.g. like On the long run, we should consider to implement parts of leopard in assembly (seriously) and integrate it directly in go-code (like e.g. sha2 in golang's crypto lib: https://golang.org/src/crypto/sha256/)

I think 3. would be the cleanest and probably also the most performant solution (but it also is the most time-consuming). Some tools might help us generate the assembly from the c code, e.g. https://github.com/minio/c2goasm#a-simple-example but they require more investigation: we'd first need to clarify which functions from leopard e.g. using AVX2 intrinsic we'd really need and also if there are still up to date (@adlerjohn reported benchmarks seem slower on a bleeding-edge AMD machine 🤔 ), then we could generate assembly for those few functions and write the remaining code plain golang.

adlerjohn commented 3 years ago

Thoughts:

  1. This is not-ideal because 1) C++ and 2) it would require both cgo (no cross-compile, performance overhead) and requiring installing a C++ library.
  2. Not performant without intrinsics.
  3. This is the good one.

https://github.com/mmcloughlin/avo allows generating go-flavored assembly from go code (so you can leverage the go compiler for type safety) and is an alternative to c2goasm.

liamsi commented 3 years ago

avo looks really cool and interesting. Not sure what would be the best approach here as "generating go-flavored assembly from go code" sounds cool but it is also (slightly) more work than locating the few functions in the C code in the original leopard library and using a tool to generate go-flavoured assembly from these. It sounds like the result could be more or less the same with either approach and with avo you can leverage the go compiler additionally. So it's probably a good idea.

While this sounds like a temptingly cool weekend project, it might end up eating much more time than anticipated.