search

BLAKE3-team / BLAKE3

the official Rust and C implementations of the BLAKE3 cryptographic hash function
Apache License 2.0
5.23k stars 353 forks source link

Expose Platform under a feature flag #351

Open rklaehn opened 1 year ago

rklaehn commented 1 year ago

There are multiple projects that make use of some blake3 internals. Most notably bao, but also my bao-tree for n0/iroh and the fleek project.

There is a guts api, but that is pretty anemic. The unfortunate end result is that there are now several forks of blake3 on crates.io.

https://crates.io/crates/fleek-blake3 https://crates.io/crates/iroh-blake3

Until a better guts api is available, would it be an option to expose Platform under a feature flag? That would be a very low level guts API that requires a lot of knowledge about internals, but it would expose the entire SIMD optimized engine of blake3 to people that want to do something more complex. Basically something like blake3-sys.

oconnor663 commented 1 year ago

I just wanted to let you know that I haven't forgotten about this! I have a lot of things working on my guts_api branch, and my short-term goal is to expose this as its own crate:

https://github.com/BLAKE3-team/BLAKE3/tree/guts_api/rust/guts

What I need to do is stabilize the API a bit and then release it with just the portable implementation. That'll let people start trying it out, without blocking on the large amount of work that needs to be done to port all the existing assembly code.

rklaehn commented 1 year ago

I just wanted to let you know that I haven't forgotten about this! I have a lot of things working on my guts_api branch, and my short-term goal is to expose this as its own crate:

https://github.com/BLAKE3-team/BLAKE3/tree/guts_api/rust/guts

What I need to do is stabilize the API a bit and then release it with just the portable implementation. That'll let people start trying it out, without blocking on the large amount of work that needs to be done to port all the existing assembly code.

Great. The fork works fine for now, but it is already stale, and it is causing all kinds of problems such as the C exports colliding when somebody uses the real blake3 crate. See https://github.com/n0-computer/iroh-blake3/pull/2.

It would make me quite happy to get rid of it eventually.

The reason we are using the fork: iroh is using bao-tree with a chunk group size of 16 chunks. Without the fork, we don't get any parallelism when hashing chunk groups. With it - it is absolutely flying. Outboard creation is at > 1 GB/s on my machine. It is probably IO bound...

devinrsmith commented 1 year ago

Noting interest here too; there are some guts that I'd liked exposed that make streaming / bao-like consumption more possible.